[antlr-interest] Bug report: Composite grammar with all tokens defined in Lexer

[Gavin Lambert]

At 06:52 23/12/2008, George S. Cowan wrote:
>Using ANTLR 3.1.1 on Windows XP, I was unable to split Yang 
>Jiang's java.g 
>(http://openjdk.java.net/projects/compiler-grammar/antlrworks/Java.g) 
>into separate parser and lexer grammars for a composite grammar. 
>The following grammar distilled from that one works, but not if 
>the PUBLIC rule is uncommented. The file Debug_DebugLex.java is 
>not generated; however, there is no warning or error message.
[...]
>------------ begin DebugParse.g -----------
>parser grammar DebugParse;
>
>modifiers
>     :
>     (   'public'
>     )*
>     ;
>
>interfaceHeader
>     :   modifiers 'interface' IDENTIFIER
>     ;
>------------- end DebugParse.g ------------

It's illegal to use quoted strings in a parser-only grammar.  (And 
it ought to produce an error, but it doesn't at the 
moment.)  It'll most definitely produce an unparsable grammar if 
you do so when your lexer defines proper rules for these strings, 
as the token type produced by the lexer will be different than the 
token type the parser is using to match.

Also, if this were a standalone grammar then you'd need to use the 
tokenVocab option in the parser.  I'm not sure if this is 
necessary or not when you're importing it into another grammar, 
though.