[antlr-interest] How to set the token buffer length?

chain one chainone at gmail.com
Wed Dec 3 19:01:53 PST 2008


I just found that ANTLR V3 would firstly turn input file stream into tokens,
then the parser started to work.Is there anyway to set the token buffer
length?
Check the following Lexer rule:

IDENT

: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
{
if (!parser.isFirst) $type=globalSearchId($text);
}
;

Function globalSearchId depends on what the current scope is to search the
ID. Current scope is already recorded in the first pass and is updated when
parser moving forward.
So in the second pass if the token stream buffer is very long, before
feeding the tokens to Parser, all tokens have been recognized and
globalSearchId has been called N times. However, because Parser hasn't
started to work, the current scope couldn't be updated. So
calling globalSearchId always returns null.

I checked the Java target Runtime DOC , found no such interfaces to modify
the token stream buffer length.


Best Regards,
chainone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20081204/507c3652/attachment.html 


More information about the antlr-interest mailing list