[antlr-interest] Is there an ANTLR trick/hack to specify "NEWLINE or EOF" in Lexer
Terence Parr
parrt at jguru.com
Tue Feb 4 09:39:46 PST 2003
Hi Michael,
What if you just made NEWLINE normal and then subclass your lexer and
override nextToken so that you detect the EOF_TYPE coming in the token
and replace it with a NEWLINE token?
Ter
On Tuesday, February 4, 2003, at 03:31 AM, micheal_jor
<open.zone at virgin.net> wrote:
> In an extension of the "single line comment can be followed by
> NEWLINE or end-of-file" scenario, I need to return a NEWLINE token
> when either an actual end-of-line or the end-of-file condition is
> encountered.
>
> [How] can this be specified in ANTLR?.
>
> 1. Subclassing uponEOF() to somehow persuade nextToken() to fudge and
> return one final NEWLINE token seems to be the "wrong" approach. Will
> it work?
>
> 2. Perhaps ANTLR should support a "virtual EOF char" that can be
> matched like any other char in Lexer rules in addition to the current
> [upon]EOF end-of-file condition mechanism?
>
> 3. Leaving the decision to a Parser generated errors at end-of-
> file "expecting "NEWLINE", found ''":
>
> (a) Matching against the EOF "token"
> newlineOrEOF
> : NEWLINE
> |! EOF
> ;
>
> (b) Checking if LA(1) against EOF_TYPE
> newlineOrEOF
> { bool atEndOfFile = true; }
> : ( NEWLINE { atEndOfFile = false; } )?
> { if ( atEndOfFile && (LA(1) != Token.EOF_TYPE) )
> throw new RecognitionException(...);
> }
> ;
>
> As a side-effect, I have pages full of non-determinism errors with
> method (b) ;-)
>
>
> Related Question:
> -----------------
> Is there a standard inbuilt mechanism for stuffing arbitary Token
> into the Lexer's output TokenStream?
>
> Cheers!,
>
> Micheal
>
>
>
>
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/
>
>
--
Co-founder, http://www.jguru.com
Creator, ANTLR Parser Generator: http://www.antlr.org
Lecturer in Comp. Sci., University of San Francisco
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list