[antlr-interest] Fragments for rewrite rules

John B. Brodie jbb at acm.org
Mon May 16 12:20:18 PDT 2011


Greetings!

On Mon, 2011-05-16 at 09:43 +0200, Ben Corne wrote:
> Hi
> 
> Below an example of me trying to use fragment lexer rules to use in rewrite
> rules. But when I try this out in the interpreter:
> 
> - 'test bar', with a fragment rule AGTEST conflicting with reading 'test',
> doesn't parse
> - 'foo bar', where no fragmented rule conflicts with reading input, parses
> 
> I thought fragment rules only tokenize input when used by other lexer rules?
> Besides using literals as tree roots, what other alternatives do I have for
> rewriting to abstract grammar elements?

I do not use ANTLRWorks so can not verify, but I think you have tripped
upon yet another quirk of the interpreter.  I suspect your example
grammar will work if you either use the debugger or use the command line
to tool, compile, and run your test rig.


You might also use a tokens {} section meta-syntax in your grammar to
define Imaginary Tokens. The tokens section goes after the options
section but before your first rule. It contains the names of the
Imaginary Tokens you want to define for use in tree construction.

So in your example, remove the fragment lexer rule and place this

tokens { AGTEST; }

after your options {}.


Also when you use an Imaginary Token in a tree rewrite expression, you
really should supply location information for that token so that
generation of meaningful error messages in later tree processing phases
becomes easier. Specify the location information as parameters to the
Imaginary Token in the rewrite expression. Something like:

program : x='test' t=test -> ^(AGTEST[$x,"AGTEST"] $t) ...........

the first parameter (e.g. the $x in this example) is a token from witch
the location information is extracted. the second parameter is a string
that is used when the tree is printed using the toString methods. the
second parameter is optional, when missing the $text field is extracted
from the first parameter.

Hope this helps...
   -jbb

> 
> =============================
> grammar Foo;
> 
> options {
>   language = Java;
>   output = AST;
>   ASTLabelType=CommonTree;
> }
> 
> program
> : 'test' t=test
> -> ^(AGTEST $t)
>  | 'foo' t=test
> -> ^(AGTEST $t)
> ;
> 
> test
> : NAM
> ;
> 
> NAM
> : LETTER ( LETTER | DIGIT )*
>  ;
> 
> fragment AGTEST : 'test';
> 
> fragment LETTER
> : 'a'..'z' | 'A'..'Z'
>  ;
> fragment DIGIT
> : '0'..'9'
>  ;
> =============================
> 
> Regards,
> Ben
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address





More information about the antlr-interest mailing list