[antlr-interest] Short circuit of the lexer

Xavier Decoret decoret at graphics.lcs.mit.edu
Fri Jan 17 14:24:03 PST 2003


I have a grammar with arrays of int defined as follow:

[ 12, 12, 4, 5, 6 ]

The number of numbers in an array can be quite large (several thousands) 
and I found that antlr (though it is *really* a cool stuff that 
simplified my life so much) is pretty slow at this. What I would like to 
do is entering a special loop when the parser figures out it has started 
a list. It would then consume by itself the int to read using fast code 
reading directly the flow of char until it finds a closing bracket which 
has to be put back in the flow.

My idea is to overide the lexer::nextToken class and use a state 
variable to decide in that function wether to use the nexttoken build by 
ANTLR or use a local loop that directly reads the input stream until it 
finds a closing bracket in which case it returns that Token, and 
everything continues as if nothing happens.

But I am not sure how this would fit with the LA() functions if I remove 
things of the input stream of the lexer without telling him.
Is what I want to do something known?
Are they any caveheat I will run into?
Is it the correct way to speed up this part of the parsing?

Thanks for any hint.


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list