[antlr-interest] Combining Tokens with rewrite rules

Ben Corne ben.corne at gmail.com
Thu Jun 16 05:54:34 PDT 2011


I've read some other threads about combining tokens in rewrite rules but I
don't think it's what I need (
http://www.antlr.org/pipermail/antlr-interest/2007-November/024612.html and
the one referenced in this thread).

---abstract tokens---
@tokens {
NAME : 'a'..'z'+ ;
KEY    : NAME ':' ;

I need to parse keyworded messages and want to combine KEY tokens into
AGKEYSTRING or not depending on which context the keywords are given in:

1) &foo:bar: // references to the keyword function definition 'foo:bar:' =>
2) foo: 1 bar: 2 // application of  'foo:bar:' => don't combine
3) foo:bar:(1,2) // canonical application of the 'foo:bar:' function =>
4) foo:bar:2 // application of 'foo:' on argument "application of 'bar: 1' "
=> don't combine

I believe in my grammar file I cannot succeed in doing this with fragment
tokens & regular tokens.
Combining the tokens with Java code is my current solution but this is bad
for using the same grammar for other bindings, which is one of the reasons
I'm making the parser with antlr.

Any ideas?

Ben C;

More information about the antlr-interest mailing list