[antlr-interest] Common token vocabulary

[Jim Idle]

It's easier than that.

List your tokens in a file (mytokens.tokens), copying the trivial output
style of one of your lexers. Add all the tokens for both lexers. Then use
the vocab=mytokens; option in both of your lexers. Just make sure that the
token numbers are contiguous, and that they are not duplicate. Also, make
sure you add to the list if you add new rules.

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Richard Knox
> Sent: Wednesday, August 22, 2012 2:53 PM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Common token vocabulary
>
> I have an ANTLR based application that parses either of two variations
> of a target language. The two variations have a mostly common
> vocabulary. There are around 200 tokens in common and less than 10
> specific to one variations or the other. I want to token types for the
> common vocabulary tokens to be the same for both language variations so
> that I can reference token types in custom Java code that is used by
> both variations.
>
> I had a solution working with ANTLR 3.1 that put the common tokens in a
> base vocabulary lexer grammar. The base vocabulary grammar was imported
> into combined lexer/parse grammars, one for each language variation.
> ANTLR 3.1 generated token types in the order the token appeared, so the
> common vocabulary tokens had the same types in both language
> variations. The variation specific tokens were then assigned larger
> numbers.
>
> I'm currently attempting to upgrade this to ANTLR 3.4, and have run
> into a problem. ANTLR 3.4 seems to alpha sort the tokens by name before
> assigning token types. This mixes variation specific tokens with common
> vocabulary tokens, and the type numbers no longer match for the common
> tokens.
>
> Is there a better way to accomplish this in ANTLR 3.4? Thanks.
>
> -rich
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address