[antlr-interest] tokenVocab-limit of 127?

[Arne Schröder]

Hi everyone,

I am working on a parser to collect information about my source-code.

Recently I found it quite handy to split my parsing-efforts into a "normal"
parser and a tree-parser. To simplify things I told the parser to leave
tokens in the AST, indicating a certain construction, so the tree-parser
does not need to double-check that.

To let the tree-parser know about the tokens, I used the option
"tokenVocab=<NameOfParser>". So far this has worked out very nicely but now
I seem to have reached a limit at the token-number 127. That is, tokenVocab
does not copy any token with a number exceeding 128 (128 is interestingly
mapped to 127), so these tokens are unknown in my tree-parser.

Looks to me as if this is a bug in ANTLR ;-P. I am using "Version 3.2 Sep
23, 2009 12:02:23" to generate my parsers.
Has anyone come accross this phenomenon before and/or does know a
work-around (besides the obvious of reducing the number of tokens ;-)).


Many thanks for any help in advance,

Arne