[antlr-interest] Trying to convert a file-oriented lexer to a line-oriented one

[Gavin Lambert]

At 09:49 26/06/2008, siva.kumar at loglogic.com wrote:
>I've looked for examples of using a lexer in a line-oriented 
>fashion but can't seem to find one.
[...]
>But I'm having some trouble figuring out how to call the lexer on 
>each line that I've read from the file.

ANTLR isn't really designed to operate in that manner; it wants to 
suck up all the input and tokenise it before parsing begins.

There was some discussion on the list a while back (and this wiki 
page: 
http://www.antlr.org/wiki/pages/viewpage.action?pageId=7929859) 
about making it lazy-load input for cases where the input stream 
is long-running (eg. interactive input), but I'm not sure how far 
it got.

>What I am essentially trying to do is:
>
>             Open a file
>             While lines exist
>                         Get a line
>                         Call the Parser

Why do you need to do this (especially if it's already in a file, 
and not using a slow/incomplete connection)?  Why can't you parse 
the entire file in one go?  Even if each line represents a 
separate expression or command or something, you can just make the 
top-level rule accept an arbitrary number of these separated by 
newlines.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080626/2f88bcf8/attachment-0001.html