[antlr-interest] Import problems when splitting out tokens

[Roger Lefebvre]

Doing upgrades and want to split the current composite grammar to serve as a base for two new grammars that will share approx 90% rules. After going through the process of splitting out tokens from rules, I'm left with:

parser RootParser;
value_expr:
..
;
...


lexer RootLexer;

LeftParen: '(';
...


grammar Timer;
options{
tokenVocab = RootLexer;
}
import RootParser
@header{package com.timer.parser;}

start:    value_expr;
...


grammar Calculator;
options{
tokenVocab = RootLexer;
}
import RootParser
@header{package com.timer.parser;}

start:    value_expr | control | txt_alias;

control_spec:
...


Although the grammar in the above checks out, the problem surfaces when generating source code - specifically the token files. Unlike with composite grammar where the lexer (RootLexer.java) is generated when generating the parser, I manually generate the lexer and subsequently generate the parser. The generated files are: RootLexer.java, RootLexer.tokens, TimerParser.java, Timer_RootParser.java, Timer.tokens, CalculatorParser.java, Calculator_RootParser.java, Calculator.tokens.

On inspecting the token files, I noticed the token values in the lexer file did not match up with the equivalents in the two parser token files. From what I understand in all that I read, this cannot be as it will cause incorrect switch jumps, amongst other errors. 

As alternatives, I tried:

- remove the tokenVocab options and import both RootParser and RootLexer
- import the RootLexer into RootParser instead. This generates an error "parser rule value_expr not allowed in lexer" which seems to have something to do with the "parser grammar RootParser" definition on the opening line
- removed "parser" qualifier from grammar definition. Generated error due to importing combined grammar Timer/Calculator.
- tried import instead of tokenVocab on RootLexer. It too generated combined grammar error.

Is there some way to specify the created lexer when generating the parser? 

I dislike the idea of having to resort to two grammars that are essentially the same. Any help would be much appreciated.

Roger



      __________________________________________________________________
Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail.  Click on Options in Mail and switch to New Mail today or register for free at http://mail.yahoo.ca