[antlr-interest] How to combine tokens around comments without using AST

Jim Idle jimi at temporal-wave.com
Wed Apr 25 09:06:25 PDT 2012


First note the given example will not work as your ID token is allowed to
be empty, which means it will match the empty string infinitely, you
needed a + not a *.

However, I would generally make a pre-pass on the text and just remove the
comments as this is usually the easiest. You can possible even do this
with regexp based replacement function (you do not say what your
target/development language is). But if you cannot do this:


ID
@init {

  StringBuilder sb = new StringBuilder();
}
:  ('a'..'z'|'A'..'Z'|'_') { sb.add(input.LA(-1); }
     (
       ('a'..'z'|'A'..'Z'|'_') { sb.add(input.LA(-1); }
       | COMMENT
     )*
     { setText(sb.toString();); }
;

If you do not need the COMMENT tokens, then use skip() rather than
$channel=HIDDEN.

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Jamal Haider
> Sent: Wednesday, April 25, 2012 3:34 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] How to combine tokens around comments without
> using AST
>
> I am a newbie to ANTLR and using it to develop a parser for an
> ambiguous language. What I want to do is to some how combine the tokens
> around the "comments" into one token without using AST.
>
> I am using this simple grammar to illustrate the problem
>
> grammar test;
>
> query
>     :     expression+
>     ;
>
> expression
>     :   alpha
>     ;
>
>
> alpha
>     : ID
>     ;
>
>
> ID  :   ('a'..'z'|'A'..'Z'|'_')*
>     ;
>
>
> COMMENT
>     :   '{' ( options {greedy=false;} : . )* '}' {$channel=HIDDEN;}
>     ;
>
> Now if we execute it with a simple text "Test{Comments}er" two separate
> tokens are generated i.e. "Test" and "er". while I want to create a
> single token out of it. Any help will be much appreciated.
>
> Thanks in advance
>
> Jim
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address


More information about the antlr-interest mailing list