[antlr-interest] combine tokens in rewrite rule
Jim Idle
jimi at temporal-wave.com
Fri Nov 9 14:08:11 PST 2007
But that is a lexical solution. When '.' is used in many places it is
quite often a better bet to have the parser determine the pieces of a
valid reference and in many cases you need the individual components
because the meanings change according to context.
For instance x.y could be an enumeration, or a property reference or
something else.
All that needs to be done is to take the .text of each element of the ID
and concatenate them. To be honest, I would probably not even do that in
the parser, but in the tree parser, where you probably have the
contextual information available (and may well not have in the parser).
Then the write would be something like ->^(IDEXPR $ids+ ) or some such.
Jim
-----Original Message-----
From: Curtis Clauson [mailto:NOSPAM at TheSnakePitDev.com]
Sent: Friday, November 09, 2007 1:56 PM
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] combine tokens in rewrite rule
I must admit, I am somewhat confused by the answer given in the
referenced thread. Doesn't the use of fragment lexer rules solve this?
For example, the grammar below will parse this input
<<
name
name.subName1
name.subName1.subName2.subName3
>>
into a tree that has three ID nodes under the root nil node, each with
the complete qualified ID text as a single token. Is this what you mean,
or have I missed something?
(tested with AntLR v3.0.1 and ANTLRWorks v1.1.4)
----------
grammar ABer1;
options {
output = AST;
}
unit: ID+;
ID: UnqualifiedID ('.' UnqualifiedID)*;
WS: (' ' | '\t' | '\r' | '\n' | '\f')+ {$channel = HIDDEN;};
fragment UnqualifiedID : UnqualifiedIDFirst (UnqualifiedIDRest)*;
fragment UnqualifiedIDFirst: 'a'..'z' | 'A'..'Z' | '_';
fragment UnqualifiedIDRest : 'a'..'z' | 'A'..'Z' | '_' | '0'..'9';
----------
I hope that helps.
-- Curtis
Adrian Ber wrote:
> Hi all,
>
> I want to find a way to combine multiple tokens in a single one.
> I've searched the archive and found this thread:
http://www.antlr.org/pipermail/antlr-interest/2007-January/019161.html.
> Does any of you have a short sample code on how to do it?
More information about the antlr-interest
mailing list