[antlr-interest] ENHANCEMENT - Have "lexer grammar" generate recognition for string literals in tokenVocab

[Austin Hastings]

Below...

Gavin Lambert wrote:
> At 02:31 10/10/2007, Austin Hastings wrote:
> >
> >When more than one parser is in use, they need to share a common
> >(initial) vocabulary. This can come from the source lexer via the
> >tokenVocab option.
> [...]
> >I would like the "lexer grammar" mode of antlr to be extended to
> >accept a tokenVocab (it currently does this) and further to
> >generate the appropriate tokens for string literals it finds in
> >the tokenVocab.
>
> I can't see how that could possibly work -- you've set up a cyclic 
> dependency.  You can't build the parser until you've built the lexer 
> (since the parser uses the lexer's tokens) and you can't build the 
> lexer until you've built the parser (since the lexer uses the parser's 
> extra tokens).  So you can't build anything unless you've already 
> built it, which doesn't work.
>
> (The problem is even worse when you have multiple parsers and a common 
> lexer, which is what you appear to be describing -- the lexer has to 
> support the union of string literals from all potential parsers, or 
> they won't work.)
>
> Just define your string literals in a "tokens" block at the top of the 
> lexer.  That's what it's for.
>
>
>
Gavin,

There's no loop. My current system has the parser using the lexer's 
tokens. What I would like is for the lexer to use the parser's tokens. 
That would let the parser generate "anonymous" tokens like 'while'=91 
and then have the lexer go "oh, a literal string. I should generate an 
imaginary token for it with number = 91!"

The multiple parsers problem just doesn't happen - at least for me - 
because there's one syntax parser and the rest are tree parsers. The 
tree parsers depend on the output (AST) of the syntax parser, so 
basically everybody want's to use the tokenVocab from the syntax parser, 
instead of the lexer.

=Austin