[antlr-interest] RE: Combination of, TokenStreamHiddenTokenFilter and TokenStreamSelector

[Bill Canfield]

>> The problem:
>> 
>> On a change of lexers (forced by our PascalLexer uponEOF()) the
>> TokenStreamHiddenTokenFilter method IToken nextToken()
>> 
>> skips the last token of the file currently being parsed/lexed and
>> returns as the next token the first token of next $include (pascal
>> interface) statement.

Hi Arni:

I don't use TokenStreamHiddenToken, but it sounds like uponEOF() is
getting called when lookahead LA(1) sees EOF.

Don't know about C#, but in C++, TokenStreamSelector has a couple
defects:  1. there is a memory leak unless you delete pop()ped lexers,
and 2. retry() called by the default uponEOF() throws an exception,
which is a performance loser.

You might try deriving your own TSS subclass which deals with
include/EOF on its own.  I gave a sketch of this at ANTLR 2005:

http://www.antlr.org/workshop/ANTLR2005/presentations/Verilog.ppt

-- 
Bill Canfield
Lead Software Developer
Zocalo Tech, Inc.
http://www.zocalo-tech.com/