[antlr-interest] Is there an ANTLR trick/hack to specify "NEWLINE or EOF" in Lexer

[micheal_jor <open.zone at virgin.net>]

In an extension of the "single line comment can be followed by 
NEWLINE or end-of-file" scenario, I need to return a NEWLINE token 
when either an actual end-of-line or the end-of-file condition is 
encountered.

[How] can this be specified in ANTLR?. 

1. Subclassing uponEOF() to somehow persuade nextToken() to fudge and 
return one final NEWLINE token seems to be the "wrong" approach. Will 
it work?

2. Perhaps ANTLR should support a "virtual EOF char" that can be 
matched like any other char in Lexer rules in addition to the current 
[upon]EOF end-of-file condition mechanism?

3. Leaving the decision to a Parser generated errors at end-of-
file "expecting "NEWLINE", found ''":

(a) Matching against the EOF "token"
newlineOrEOF
:   NEWLINE
|!  EOF
;

(b) Checking if LA(1) against EOF_TYPE
newlineOrEOF
{ bool atEndOfFile = true; }
:   ( NEWLINE { atEndOfFile = false; } )?
    { if ( atEndOfFile && (LA(1) != Token.EOF_TYPE) )
         throw new RecognitionException(...);
    }
;

As a side-effect, I have pages full of non-determinism errors with 
method (b)  ;-)


Related Question:
-----------------
Is there a standard inbuilt mechanism for stuffing arbitary Token 
into the Lexer's output TokenStream?

Cheers!,

Micheal


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/