[antlr-interest] Island Grammars with identical tokens

[Mike J. Bell]

I have a main grammar that passes control to two different island
grammars to parse a language.  One island grammar is an island because
it's so big and can be used in a context outside the main language,
and the other island has vastly different lexer rules (whitespace
isn't ignored, etc.).

All three generate ASTs for processing by a follow-on tree parser.  As
such, I need the lexer token numbers to match for all three lexers.
It seems like this is problematic; I thought if I modified the .tokens
files they would be used as input files to the ANTLR generator; this
appears to be false...i.e. they are generated from the .g file each
time.

Does anybody have any advice?  It's like I need to specify a global
tokens {...} and block of lexer rule names, so that the generated
files all use the same set of numbers.  Maybe it's possible for me to
make a "base" lexer class that mentions all the token names and has
overridable lexer rules.  Not sure if that can even happen.

Thanks in advance for any tips...

-- 
Mike J. Bell on gmail