[antlr-interest] Common token vocabulary

[Kirby Bohling]

Isn't there an option to specify the tokens property file.  You'd just have
to enumerate the tokens once by hand (which would be annoying).

Here this guy talks about the option and a problem he's had with it:
http://qscribble.blogspot.com/2007/07/im-not-sure-if-lexers-support.html

The option is called "tokenVocab".

Kirby


On Wed, Aug 22, 2012 at 4:53 PM, Richard Knox <rich at dreambox.com> wrote:

> I have an ANTLR based application that parses either of two variations of
> a target language. The two variations have a mostly common vocabulary.
> There are around 200 tokens in common and less than 10 specific to one
> variations or the other. I want to token types for the common vocabulary
> tokens to be the same for both language variations so that I can reference
> token types in custom Java code that is used by both variations.
>
> I had a solution working with ANTLR 3.1 that put the common tokens in a
> base vocabulary lexer grammar. The base vocabulary grammar was imported
> into combined lexer/parse grammars, one for each language variation. ANTLR
> 3.1 generated token types in the order the token appeared, so the common
> vocabulary tokens had the same types in both language variations. The
> variation specific tokens were then assigned larger numbers.
>
> I'm currently attempting to upgrade this to ANTLR 3.4, and have run into a
> problem. ANTLR 3.4 seems to alpha sort the tokens by name before assigning
> token types. This mixes variation specific tokens with common vocabulary
> tokens, and the type numbers no longer match for the common tokens.
>
> Is there a better way to accomplish this in ANTLR 3.4? Thanks.
>
> -rich
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>