[antlr-interest] tokenVocab-limit of 127?

Arne Schröder arne.schroeder at gmail.com
Wed Jun 30 05:41:59 PDT 2010

Hi again,

sorry for the disruption. Deleting the tokens-files and re-generating has
solved the issue.


On Wed, Jun 30, 2010 at 12:22, Arne Schröder <arne.schroeder at gmail.com>wrote:

> Hi everyone,
> I am working on a parser to collect information about my source-code.
> Recently I found it quite handy to split my parsing-efforts into a "normal"
> parser and a tree-parser. To simplify things I told the parser to leave
> tokens in the AST, indicating a certain construction, so the tree-parser
> does not need to double-check that.
> To let the tree-parser know about the tokens, I used the option
> "tokenVocab=<NameOfParser>". So far this has worked out very nicely but now
> I seem to have reached a limit at the token-number 127. That is, tokenVocab
> does not copy any token with a number exceeding 128 (128 is interestingly
> mapped to 127), so these tokens are unknown in my tree-parser.
> Looks to me as if this is a bug in ANTLR ;-P. I am using "Version 3.2 Sep
> 23, 2009 12:02:23" to generate my parsers.
> Has anyone come accross this phenomenon before and/or does know a
> work-around (besides the obvious of reducing the number of tokens ;-)).
> Many thanks for any help in advance,
> Arne

More information about the antlr-interest mailing list