[antlr-interest] RE: Combination of,
TokenStreamHiddenTokenFilter and TokenStreamSelector
Bill Canfield
canfield at zocalo-tech.com
Sat May 6 09:35:09 PDT 2006
>> The problem:
>>
>> On a change of lexers (forced by our PascalLexer uponEOF()) the
>> TokenStreamHiddenTokenFilter method IToken nextToken()
>>
>> skips the last token of the file currently being parsed/lexed and
>> returns as the next token the first token of next $include (pascal
>> interface) statement.
Hi Arni:
I don't use TokenStreamHiddenToken, but it sounds like uponEOF() is
getting called when lookahead LA(1) sees EOF.
Don't know about C#, but in C++, TokenStreamSelector has a couple
defects: 1. there is a memory leak unless you delete pop()ped lexers,
and 2. retry() called by the default uponEOF() throws an exception,
which is a performance loser.
You might try deriving your own TSS subclass which deals with
include/EOF on its own. I gave a sketch of this at ANTLR 2005:
http://www.antlr.org/workshop/ANTLR2005/presentations/Verilog.ppt
--
Bill Canfield
Lead Software Developer
Zocalo Tech, Inc.
http://www.zocalo-tech.com/
More information about the antlr-interest
mailing list