[antlr-interest] Is there an ANTLR trick/hack to specify "NEWLINE or EOF" in Lexer

Terence Parr parrt at jguru.com
Tue Feb 4 09:39:46 PST 2003


Hi Michael,

What if you just made NEWLINE normal and then subclass your lexer and 
override nextToken so that you detect the EOF_TYPE coming in the token 
and replace it with a NEWLINE token?

Ter

On Tuesday, February 4, 2003, at 03:31 AM, micheal_jor 
<open.zone at virgin.net> wrote:

> In an extension of the "single line comment can be followed by
> NEWLINE or end-of-file" scenario, I need to return a NEWLINE token
> when either an actual end-of-line or the end-of-file condition is
> encountered.
>
> [How] can this be specified in ANTLR?.
>
> 1. Subclassing uponEOF() to somehow persuade nextToken() to fudge and
> return one final NEWLINE token seems to be the "wrong" approach. Will
> it work?
>
> 2. Perhaps ANTLR should support a "virtual EOF char" that can be
> matched like any other char in Lexer rules in addition to the current
> [upon]EOF end-of-file condition mechanism?
>
> 3. Leaving the decision to a Parser generated errors at end-of-
> file "expecting "NEWLINE", found ''":
>
> (a) Matching against the EOF "token"
> newlineOrEOF
> :   NEWLINE
> |!  EOF
> ;
>
> (b) Checking if LA(1) against EOF_TYPE
> newlineOrEOF
> { bool atEndOfFile = true; }
> :   ( NEWLINE { atEndOfFile = false; } )?
>     { if ( atEndOfFile && (LA(1) != Token.EOF_TYPE) )
>          throw new RecognitionException(...);
>     }
> ;
>
> As a side-effect, I have pages full of non-determinism errors with
> method (b)  ;-)
>
>
> Related Question:
> -----------------
> Is there a standard inbuilt mechanism for stuffing arbitary Token
> into the Lexer's output TokenStream?
>
> Cheers!,
>
> Micheal
>
>
>
>
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
>
>
--
Co-founder, http://www.jguru.com
Creator, ANTLR Parser Generator: http://www.antlr.org
Lecturer in Comp. Sci., University of San Francisco


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list