[antlr-interest] Combining Tokens with rewrite rules

[Ben Corne]

Hi

I've read some other threads about combining tokens in rewrite rules but I
don't think it's what I need (
http://www.antlr.org/pipermail/antlr-interest/2007-November/024612.html and
the one referenced in this thread).

---abstract tokens---
@tokens {
  AGKEYSTRING;
}
----tokens--------------
NAME : 'a'..'z'+ ;
KEY    : NAME ':' ;
----------------------------

I need to parse keyworded messages and want to combine KEY tokens into
AGKEYSTRING or not depending on which context the keywords are given in:

1) &foo:bar: // references to the keyword function definition 'foo:bar:' =>
combine
2) foo: 1 bar: 2 // application of  'foo:bar:' => don't combine
3) foo:bar:(1,2) // canonical application of the 'foo:bar:' function =>
combine
4) foo:bar:2 // application of 'foo:' on argument "application of 'bar: 1' "
=> don't combine

I believe in my grammar file I cannot succeed in doing this with fragment
tokens & regular tokens.
Combining the tokens with Java code is my current solution but this is bad
for using the same grammar for other bindings, which is one of the reasons
I'm making the parser with antlr.

Any ideas?

Regards
Ben C;