[antlr-interest] How to combine tokens around comments without using AST
Jim Idle
jimi at temporal-wave.com
Wed Apr 25 09:06:25 PDT 2012
First note the given example will not work as your ID token is allowed to
be empty, which means it will match the empty string infinitely, you
needed a + not a *.
However, I would generally make a pre-pass on the text and just remove the
comments as this is usually the easiest. You can possible even do this
with regexp based replacement function (you do not say what your
target/development language is). But if you cannot do this:
ID
@init {
StringBuilder sb = new StringBuilder();
}
: ('a'..'z'|'A'..'Z'|'_') { sb.add(input.LA(-1); }
(
('a'..'z'|'A'..'Z'|'_') { sb.add(input.LA(-1); }
| COMMENT
)*
{ setText(sb.toString();); }
;
If you do not need the COMMENT tokens, then use skip() rather than
$channel=HIDDEN.
Jim
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Jamal Haider
> Sent: Wednesday, April 25, 2012 3:34 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] How to combine tokens around comments without
> using AST
>
> I am a newbie to ANTLR and using it to develop a parser for an
> ambiguous language. What I want to do is to some how combine the tokens
> around the "comments" into one token without using AST.
>
> I am using this simple grammar to illustrate the problem
>
> grammar test;
>
> query
> : expression+
> ;
>
> expression
> : alpha
> ;
>
>
> alpha
> : ID
> ;
>
>
> ID : ('a'..'z'|'A'..'Z'|'_')*
> ;
>
>
> COMMENT
> : '{' ( options {greedy=false;} : . )* '}' {$channel=HIDDEN;}
> ;
>
> Now if we execute it with a simple text "Test{Comments}er" two separate
> tokens are generated i.e. "Test" and "er". while I want to create a
> single token out of it. Any help will be much appreciated.
>
> Thanks in advance
>
> Jim
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
More information about the antlr-interest
mailing list