[antlr-interest] combine tokens in rewrite rule

Curtis Clauson NOSPAM at TheSnakePitDev.com
Fri Nov 9 13:56:23 PST 2007


I must admit, I am somewhat confused by the answer given in the 
referenced thread. Doesn't the use of fragment lexer rules solve this?

For example, the grammar below will parse this input
<<
name
name.subName1
name.subName1.subName2.subName3
 >>
into a tree that has three ID nodes under the root nil node, each with 
the complete qualified ID text as a single token. Is this what you mean, 
or have I missed something?

(tested with AntLR v3.0.1 and ANTLRWorks v1.1.4)
----------
grammar ABer1;

options {
     output = AST;
}


unit: ID+;


ID: UnqualifiedID ('.' UnqualifiedID)*;
WS: (' ' | '\t' | '\r' | '\n' | '\f')+ {$channel = HIDDEN;};


fragment UnqualifiedID     : UnqualifiedIDFirst (UnqualifiedIDRest)*;
fragment UnqualifiedIDFirst: 'a'..'z' | 'A'..'Z' | '_';
fragment UnqualifiedIDRest : 'a'..'z' | 'A'..'Z' | '_' | '0'..'9';
----------

I hope that helps.
-- Curtis


Adrian Ber wrote:
> Hi all,
> 
> I want to find a way to combine multiple tokens in a single one.
> I've searched the archive and found this thread: http://www.antlr.org/pipermail/antlr-interest/2007-January/019161.html.
> Does any of you have a short sample code on how to do it?



More information about the antlr-interest mailing list