[antlr-interest] combine tokens in rewrite rule

[Jim Idle]

But that is a lexical solution. When '.' is used in many places it is 
quite often a better bet to have the parser determine the pieces of a 
valid reference and in many cases you need the individual components 
because the meanings change according to context.

For instance x.y could be an enumeration, or a property reference or 
something else.

All that needs to be done is to take the .text of each element of the ID 
and concatenate them. To be honest, I would probably not even do that in 
the parser, but in the tree parser, where you probably have the 
contextual information available (and may well not have in the parser). 
Then the write would be something like ->^(IDEXPR $ids+ ) or some such.

Jim

-----Original Message-----
From: Curtis Clauson [mailto:NOSPAM at TheSnakePitDev.com] 
Sent: Friday, November 09, 2007 1:56 PM
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] combine tokens in rewrite rule

I must admit, I am somewhat confused by the answer given in the 
referenced thread. Doesn't the use of fragment lexer rules solve this?

For example, the grammar below will parse this input
<<
name
name.subName1
name.subName1.subName2.subName3
 >>
into a tree that has three ID nodes under the root nil node, each with 
the complete qualified ID text as a single token. Is this what you mean, 

or have I missed something?

(tested with AntLR v3.0.1 and ANTLRWorks v1.1.4)
----------
grammar ABer1;

options {
     output = AST;
}


unit: ID+;


ID: UnqualifiedID ('.' UnqualifiedID)*;
WS: (' ' | '\t' | '\r' | '\n' | '\f')+ {$channel = HIDDEN;};


fragment UnqualifiedID     : UnqualifiedIDFirst (UnqualifiedIDRest)*;
fragment UnqualifiedIDFirst: 'a'..'z' | 'A'..'Z' | '_';
fragment UnqualifiedIDRest : 'a'..'z' | 'A'..'Z' | '_' | '0'..'9';
----------

I hope that helps.
-- Curtis


Adrian Ber wrote:
> Hi all,
> 
> I want to find a way to combine multiple tokens in a single one.
> I've searched the archive and found this thread: 
http://www.antlr.org/pipermail/antlr-interest/2007-January/019161.html.
> Does any of you have a short sample code on how to do it?