[antlr-interest] Re: circular token buffer and overriding nextToken()

[micheal_jor]

--- In antlr-interest at yahoogroups.com, "netminka" <netminka at n...> wrote:
> hi all,
> I have tried overriding nextToken() because I need my lexer to return
> dummy tokens at certain points and what happens is that the 
> dummy token writes over a valid token in the token buffer, and my input
> program is made incorrect. 
> 
> the k for my lexer is four. This problem happens when only one 
> dummy token is inserted. It is erratic. 
> the token buffer is circular and it's not clear to me where / how
> I can safely insert a dummy token.
> 
> Can anyone who really understands this interface give me a clue?
> 
> Thanks VERY much.
> megan

Megan,

The TokenStream interface itself has a single method - nextToken(). It
is called repeatedly until it returns an EOF token. Simple enough. The
implementation of nextToken() in generated lexers is non-virtual for
C++ & C# (also for Java I think). Which suggests it isn't [really]
intended/expected to be overridden.

Now on to the issue you are having:

The implementation of nextToken() uses the method makeToken()
to...well...make the tokens. This is an extension point and you could
try overriding this method and plugging in custom code to make dummy
tokens here. Alternativey, you may prefer to use a separate
TokenStream object that is logically sited between your Lexer and
Parser - you can derived something from TokenStreamBasicFilter or
write your own.

So, you should either:
a. Override makeToken()
b. Implement a TokenStream filter

You may want to look at Monty's website http://www.codetransform.com/
for ideas on building and using TokenStream filters.

Just out of interest, what are you using ANTLR/C# for?.

Cheers,

Micheal
ANTLR/C#



[*]  Incidentally, I should have asked you WHY you wanted to override
nextToken() before, rather than just suggesting a possible workaround
for the default non-virtual intent.



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/