[antlr-interest] Is '!' operator working with ANTLR 3.1.2

[Colin Macdonald]

OK, checked the reference last night (page 98) and ! is definitely parser 
only

Jim's got a good point about white space, so you need to go back to your 
original lexer rule and hack the text.

The easiest option would be to always trim out the "EXT." by calling 
something like

        $EXTERNAL_CALL.text.substring(4)

I don't suggest messing around with start & stop.  You don't have access 
to the Token in the lexer actions, besides it would be fragile unless you 
extend CommonToken to track if you've made the change already.

However, you could add a lexer action to alter the state property to start 
after the prefix.  This will should cause the Token created to only 
represent ID.

        EXTERNAL_CALL: 'EXT.' ID {state.tokenStartCharIndex+=4;};

Have you linked the ANTLR source to the jar in your environment?  In 
Eclipse, F3 is your friend, but you need the source linked.

Colin



From:   "Philippe Frankson" <Philippe.Frankson at Frsglobal.com>
To:     <antlr-interest at antlr.org>
Date:   11/18/2010 06:40 AM
Subject:        Re: [antlr-interest] Is '!' operator working with ANTLR 
3.1.2
Sent by:        antlr-interest-bounces at antlr.org



I'm sorry but I'm not sure to understand what you mean by 'change the
start and end points of the token'.
Do you mean I should split my lexer this way:

fragment INT             : ('0'..'9');
fragment ALPHA    : ('a'..'z'|'A'..'Z'|'_');

ID               : ALPHA (ALPHA | INT)*;
EXT              : 'EXT.';

rule: EXT ID '(' { stack.push(new FuncName($ID.getText())); }
(expressList { ... } )? ')'              expressList: ...;


In fact I cannot do that because I have the following lexer rule (I
didn't put it in my previous mail to make it lighter):

IDENTIFIER                               : ID (('.' ID)=> '.' ID { $type = 
CELLNAME; })*;

Because of that rule, 'EXT.Myfunc' would return the token CELLNAME
instead of EXT + ID;

Now, of course, like you said, I can strip the characters out but I
thought it was possible to do that at lexer side and that would have
been better.

Thank you for your feedback.
Philippe Frankson

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Jim Idle
Sent: 17 November 2010 23:24
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Is '!' operator working with ANTLR 3.1.2

In the lexer, but the context of the question tells you this. The !
operator
in the parser is used to prevent being auto written in to the tree on an
AST
producing parser. That is not the same thing.

Ignoring the token in the parser is not the same thing as it allows
whitespace and any other characters that are not passed to the parser to
be
placed in between.

All you need do is change the start and end points of the token or just
strip the characters out when you come to need the string value of the
token.

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Colin Macdonald
> Sent: Wednesday, November 17, 2010 12:43 PM
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Is '!' operator working with ANTLR 3.1.2
> 
> Jim,
> 
> Don't you mean that the ! operator is not supported in the Lexer?
> 
> Phil would need a parser rule to discard the token.  Something like:
> 
> //Convert ID from a fragment to a token
> ID: ALPHA (ALPHA | INT)*;
> EXT: 'EXT.';
> external_call: EXT! ID;
> 
> rule: call=external_call '(' { stack.push(new
>         FuncName($call.text)); } (expressList { ... } )? ')'
>         expressList: ...;
> 
> 
> 
> Colin Macdonald
> Senior Consultant
> _____________________________________
> 
> GROUP Business Software
> Phone: 770 720 1300 ext: 6132
> 
> http://www.gbs.com/
> 
> 
> 
> 
> From:   "Jim Idle" <jimi at temporal-wave.com>
> To:     <antlr-interest at antlr.org>
> Date:   2010-11-17 11:19 AM
> Subject:        Re: [antlr-interest] Is '!' operator working with
ANTLR
> 3.1.2
> Sent by:        antlr-interest-bounces at antlr.org
> 
> 
> 
> Remember antlr.markmail.org
> 
> This operator is not supported in ANTLR3 for performance reasons. But
> if the pieces you don't want are at the start or end, then you can
just
> change the start and/or end points of the token.
> 
> Jim
> 
> > -----Original Message-----
> > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > bounces at antlr.org] On Behalf Of Philippe Frankson
> > Sent: Wednesday, November 17, 2010 4:51 AM
> > To: antlr-interest at antlr.org
> > Subject: [antlr-interest] Is '!' operator working with ANTLR 3.1.2
> >
> >
> > When I call my parser with -> EXT.MyFunc() The value returned by
> > $EXTERNAL_CALL.getText() is 'EXT.MyFunc' where I would expect to
have
> > only 'MyFunc' (because I'm using the '!' operator in the lexer here
> > below).
> >
> > fragment INT                           : ('0'..'9');
> > fragment ALPHA                                 :
> ('a'..'z'|'A'..'Z'|'_');
> > fragment ID                                            : ALPHA
(ALPHA
> |
> INT)*;
> >
> > EXTERNAL_CALL: 'EXT.'! ID;
> >
> >
> > rule: EXTERNAL_CALL '(' { stack.push(new
> > FuncName($EXTERNAL_CALL.getText())); } (expressList { ... } )? ')'
> > expressList: ...;
> >
> >
> > Any idea why the '!' operator looks like not working ? What am I
> doing
> > wrong ?
> >
> > Thank you.
> > Philippe Frankson
> >
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe: http://www.antlr.org/mailman/options/antlr-
> interest/your-
> > email-address
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address


List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: 
http://www.antlr.org/mailman/options/antlr-interest/your-email-address


Colin Macdonald
Senior Consultant
_____________________________________ 

GROUP Business Software
Phone: 770 720 1300 ext: 6132

http://www.gbs.com/