[antlr-interest] Island Grammars with identical tokens

[Thomas Brandon]

On Thu, May 21, 2009 at 11:42 PM, Mike J. Bell <ckimyt at gmail.com> wrote:
> I have a main grammar that passes control to two different island
> grammars to parse a language.  One island grammar is an island because
> it's so big and can be used in a context outside the main language,
> and the other island has vastly different lexer rules (whitespace
> isn't ignored, etc.).
>
> All three generate ASTs for processing by a follow-on tree parser.  As
> such, I need the lexer token numbers to match for all three lexers.
> It seems like this is problematic; I thought if I modified the .tokens
> files they would be used as input files to the ANTLR generator; this
> appears to be false...i.e. they are generated from the .g file each
> time.
>
> Does anybody have any advice?  It's like I need to specify a global
> tokens {...} and block of lexer rule names, so that the generated
> files all use the same set of numbers.  Maybe it's possible for me to
> make a "base" lexer class that mentions all the token names and has
> overridable lexer rules.  Not sure if that can even happen.
>
> Thanks in advance for any tips...
You should be able to use the importVocab option to specify a common
.tokens file for all lexers to import. Or you can chain the lexers by
having one island grammar import the vocab of the other and then have
the main grammar import that island grammars vocab. That will avoid
the need to keep the manually generated common token file in sync.
Though then you will need to check if ANTLR's build ordering can
handle this dependency or else make sure you enforce the build order
manually or tokens will get out of sync.

Tom.
>
> --
> Mike J. Bell on gmail
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>