[antlr-interest] Reuse tokens from multiple grammars in tree grammar

[Jim Idle]

First create the lexer for say the Java language and generate it. You will
see you get a .tokens file that looks like:

CLASS=5
IF=6

and so on. You don't need to do this bit, but it shows what a .tokens file
should look like.

Now, take control of this file away from your lexer by renaming it to
commontokens.tokens or something similar.

Next, add the token names of all the different lexers and all the
imaginary tokens you need (you can add to this as you go of course) making
sure that the numbers you assign are contiguous.

Now, all your grammars share this with:

options {

tokenVocab=commontokens;
 ...

}

And now you have a common set of tokens and any parser producing an AST
with such tokens produces a generic AST that you can walk with a single
tree parser/walker, so long as that walker encompasses all the constructs
that each individual language might need.

Jim



> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Robin
> Sent: Monday, November 07, 2011 4:58 AM
> To: antlr-interest at antlr.org interest
> Subject: [antlr-interest] Reuse tokens from multiple grammars in tree
> grammar
>
> Hi all,
>
> I'm curently working on a thesis project and I need to write tree
> grammars that translate ASTs produced by several parsers (Java, C, etc)
> into "generic" ASTs. These "generic" ASTs should only contain basic
> information about the source code being parsed such as function
> signatures, class names, etc.
>
> I of course thought about ANTLR for this purpose but I'm facing some
> problems:
>
> * How can I define a set of imaginary tokens for this "generic" AST so
> that they can be reused in tree grammars? (such as JavaToGeneric.g,
> CToGeneric.g, etc)
> * If I only want a portion of an AST to be translated, can I use option
> "filter = true"?
>
> I don't know if I've been clear, I could give examples of what I am
> trying to accomplish if you need. If I'm going the wrong way, please
> tell me so.
>
> Thanks in advance for your help :)
>
> Robin
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address