[antlr-interest] ENHANCEMENT - Have "lexer grammar" generate recognition for string literals in tokenVocab

Gavin Lambert antlr at mirality.co.nz
Tue Oct 9 13:24:04 PDT 2007


At 02:31 10/10/2007, Austin Hastings wrote:
 >
 >When more than one parser is in use, they need to share a common 

 >(initial) vocabulary. This can come from the source lexer via 
the
 >tokenVocab option.
[...]
 >I would like the "lexer grammar" mode of antlr to be extended to 

 >accept a tokenVocab (it currently does this) and further to
 >generate the appropriate tokens for string literals it finds in
 >the tokenVocab.

I can't see how that could possibly work -- you've set up a cyclic 
dependency.  You can't build the parser until you've built the 
lexer (since the parser uses the lexer's tokens) and you can't 
build the lexer until you've built the parser (since the lexer 
uses the parser's extra tokens).  So you can't build anything 
unless you've already built it, which doesn't work.

(The problem is even worse when you have multiple parsers and a 
common lexer, which is what you appear to be describing -- the 
lexer has to support the union of string literals from all 
potential parsers, or they won't work.)

Just define your string literals in a "tokens" block at the top of 
the lexer.  That's what it's for.



More information about the antlr-interest mailing list