[antlr-interest] common tokens

[Robert Soule]

Hi,

I just found this web page:

http://www.antlr.org/wiki/display/ANTLR3/Composite+Grammars

and I think this is close to what I want. I can create a lexer grammar
that is imported by my two language parsers/lexers, so they can share the
common lexer rule :

SEMI : ';' ;

However, I would prefer to share the literals. Is this possible? This
is preferable, for example, in 'for' loops, where I have something like:

forStmt : 'for' condition '{' body '}' ;

I would like to share a common definition of the 'for' literal, without
having to create a lexer rule:

FOR : 'for' ;
forStmt : FOR condition '{' body '}' ;

Is this possible with composite grammars?

thanks,
Robert

On Thu, Feb 26, 2009 at 11:51 AM, Robert Soule <robert.soule at gmail.com> wrote:
> Hi,
>
> I am converting from one language to another using tree re-write rules.
> I am getting an error when both languages define the same token. For
> example:
>
> From A.g:
> grammar A;
> options { output = AST; }
> tokens { PROGRAM; A; }
> accept : program EOF      ->   ^(PROGRAM program);
> program : ',' 'a'         -> ^(A ',' 'a');
> WHITE_SPACE      : (' '|'\t'|'\r'|'\n')+                     {
> $channel = HIDDEN; };
>
> From B.g:
> grammar B;
> options { output = AST; }
> tokens { BPROGRAM; B; }
> accept : program EOF      ->   ^(BPROGRAM program);
> program : 'b' ','             -> ^(B 'b' ',');
> WHITE_SPACE      : (' '|'\t'|'\r'|'\n')+                     {
> $channel = HIDDEN; };
>
> From AtoB.g:
> tree grammar AtoB;
> options { tokenVocab=A; tokenVocab=B;  ASTLabelType=CommonTree; output=AST; }
> accept : ^(PROGRAM program)  -> ^(PROGRAM program);
> program : ^(A ',' 'a')           -> ^(B 'b' ',');
>
>
> The error seems to occur because both A.g and B.g use a ',' literal, assigned
> different values in their respective .tokens files. When I reference
> the ',' literal
> in AtoB.g, I'm guessing that there is no way to tell if it is A's token or B's.
>
> Is there either a way to scope tokens (A::';') or is there way to
> define a common
> set of tokens that are shared by both grammars, and then import them?
>
> thank you,
> Robert
>