[antlr-interest] unexpected char error

Martin Probst mail at martin-probst.com
Sat Mar 24 02:06:37 PDT 2007


> But, its supposed to implicitly define the token as a literal when  
> I use
> it.  That's the whole point of allowing me to use the string in the
> parser.  It should then appear in the literal table.  IN the second  
> case
> above, I see:
>
> "="=11 in my myLexerTokenTypes.txt file.

It does. But the thing is that this "test literals" feature only  
tests lexed tokens for literals. Meaning, if the lexer reads  
something in, and creates a token from it, it then looks up in a  
table if it's one of the literals (you can switch this behaviour on/ 
off on a per-token type basis). If yes, it sets the type of the token  
to the corresponding type.

Of course this only works if the input can be matched by any lexer  
rule, and if this rule has the test literals thing turned on. This is  
typically done to filter tokens like "if" from the identifier rule of  
the language. I guess in your case, because identifiers are not  
allowed to contain "=" tokens, it won't work.

So generally speaking this test literals feature is nice to easily  
pick keywords that would also be matched by the identifier rule (or  
similar rules), but it's not useful for tokens that do not match the  
identifier production.

Regards,
Martin


More information about the antlr-interest mailing list