[antlr-interest] tokenVocab-limit of 127?

[Arne Schröder]

Hi again,

sorry for the disruption. Deleting the tokens-files and re-generating has
solved the issue.


Arne

On Wed, Jun 30, 2010 at 12:22, Arne Schröder <arne.schroeder at gmail.com>wrote:

> Hi everyone,
>
> I am working on a parser to collect information about my source-code.
>
> Recently I found it quite handy to split my parsing-efforts into a "normal"
> parser and a tree-parser. To simplify things I told the parser to leave
> tokens in the AST, indicating a certain construction, so the tree-parser
> does not need to double-check that.
>
> To let the tree-parser know about the tokens, I used the option
> "tokenVocab=<NameOfParser>". So far this has worked out very nicely but now
> I seem to have reached a limit at the token-number 127. That is, tokenVocab
> does not copy any token with a number exceeding 128 (128 is interestingly
> mapped to 127), so these tokens are unknown in my tree-parser.
>
> Looks to me as if this is a bug in ANTLR ;-P. I am using "Version 3.2 Sep
> 23, 2009 12:02:23" to generate my parsers.
> Has anyone come accross this phenomenon before and/or does know a
> work-around (besides the obvious of reducing the number of tokens ;-)).
>
>
> Many thanks for any help in advance,
>
> Arne
>