[antlr-interest] v2->v3 Skip chars in Lexer? Terrence?

[Jim Idle]

It is for performance and has been talked about for 4 years, so we don't
need to start it again. If we implement ! then you have to build the
string in to every token and copy it, but basically it is easy to strip
leading and trailing characters as the tokens carry pointers, so get the
start pointer, increment it, get the end point, decrement it, now

Do not use the built in $token.text->chars as this is slow and just for
convenience. The token holds a pointer to the start of the text in the
original input stream, which is greatly faster and you don’t do anything
at all to the token until and if you use it. You know the token type, so
can handle it appropriately. It is a trivial piece of code and you will
want a generic method/function for getting the string anyway. It takes
less time to implement it than to worry about ! not being there any more
:-)

Jim





> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Ruslan Zasukhin
> Sent: Saturday, April 16, 2011 3:37 AM
> To: antlr-interest at antlr.org; Paradigma Software Team List
> Subject: Re: [antlr-interest] v2->v3 Skip chars in Lexer? Terrence?
>
> On 4/16/11 1:18 PM, "Bart Kiers" <bkiers at gmail.com> wrote:
>
> Thank you, Bart.
>
> And I have forget to mention that we using C-target.
> So there is no nice string classes ...
>
> And even with string classes, don't you think guys, that this NEW WAY
> Of dealing of wrap-quotes do not looks  best of the best?
>
>     We must now work with strings?
>     remove first/last chars?
>     remove inside chars?
>         this means do not required copy pasts ...
>
> I have hear that ANTLR3 LEXER going to be much faster of v2 Lexer.
> We did use FLEX instead of v2 Lexer because of that.
>
> And now ... We must do manual job with strings ??
>
> Somehow not best of the best?
> May be it is good idea RETURN BACK
> that simple way to skip that chars right in lexer?
>
>
> > On Sat, Apr 16, 2011 at 12:06 PM, Ruslan Zasukhin
> > <ruslan_zasukhin at valentina-db.com> wrote:
> > ...
> >
> > How to remove that quotes in v3?  :-)
> >
> >
> > Here's a way:
> >
> > DELIMITED
> >   @init {
> >     String q = null;
> >   }
> >   @after {
> >     String text = getText();
> >     // remove the first and last quote, replace all 2 quotes with a
> > single quote
> >     setText(text.substring(1, text.length()-1).replace(q+q, q));
> >   }
> >   :  ( DQUOTE (~DQUOTE | DQUOTE DQUOTE)+ DQUOTE {q = $DQUOTE.text;}
> >      | BQUOTE (~BQUOTE | BQUOTE BQUOTE)+ BQUOTE {q = $BQUOTE.text;}
> >      ) { $type = IDENT; }
> >   ;
> >
> > or create your own token that handles the replacements
> > internally:
> > http://www.antlr.org/wiki/pages/viewpage.action?pageId=1844
> >
> > Regards,
> >
> > Bart.
>
> --
> Best regards,
>
> Ruslan Zasukhin
> VP Engineering and New Technology
> Paradigma Software, Inc
>
> Valentina - Joining Worlds of Information http://www.paradigmasoft.com
>
> [I feel the need: the need for speed]
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address