[antlr-interest] Conversion of Antlr v2 grammars to v3: tokens section for lexers.

[Kenneth Domino]

I'm trying to convert some Antlr version 2 grammars to work under Antlr 
version 3, using the "v2me" tool, the information in 
http://www.antlr.org/wiki/display/ANTLR3/Migrating+from+ANTLR+2+to+ANTLR+3#MigratingfromANTLR2toANTLR3-NewinANTLR3, 
and the old and new documentations.  But, I'm having some problems.

The following grammar worked under v2:

class test extends Lexer;

tokens {
 FOO = "foo";
 BAR = "bar";
}

VERB : "is";

This is converted by v3me to:

lexer grammar test;

tokens {
 FOO = 'foo';
 BAR = 'bar';
}

VERB : 'is';

Unfortunately, it looks like this is now illegal because I get Antlr v3 
error messages:

error(108): test.g:4:8: literals are illegal in lexer tokens{} section: 
'foo'
error(108): test.g:5:8: literals are illegal in lexer tokens{} section: 
'bar'

I can change the grammar by adding explicit productions for the literals:

lexer grammar test;

tokens {
 FOO;
 BAR;
}

VERB : 'is';
FOO : 'foo';
BAR : 'bar';

But, is this the only work around?  The only other possibility seems that I 
specify a combined lexer and parser as in:

grammar test;

tokens {
 FOO = 'foo';
 BAR = 'bar';
 VERB;
}

VERB : 'is';
f : FOO BAR VERB;