[antlr-interest] Literals in lexer

[Arnar Birgisson]

Excellent, this was exactly what I was looking for.

Cristian, thank you so much for spending all this time on my problem.

Arnar

> -----Original Message-----
> From: Cristian Amitroaie [mailto:cristian at amiq.ro] 
> Sent: 19. nóvember 2003 22:05
> To: antlr-interest at yahoogroups.com
> Subject: Re: [antlr-interest] Literals in lexer
> 
> 
> I was thinking about adding a tokens section in the lexer:
> 
> tokens {
>    "this"
> };
> 
> or whatever you have, eventually giving it a name to avoid 
> any compiler 
> complaints, ... shortly see the attached example, you'll 
> figure it out...
> 
> Some log (# where "þessi" is your "this"):
> ---
> [cristian at lapc antlr_interest]$ java Main this_lexer.test
> 6 : is
> 4 : þessi // note the different token id
> 6 : lexer
> 6 : good
> 6 : enough
> 6 : þessilexer
> 6 : is
> 6 : good
> 6 : enough
> 6 : indeed
> ---
> 
> Let me know it works ,
> Cristian
> 
> On Tuesday 18 November 2003 19:46, Arnar Birgisson wrote:
> > Hello
> >
> > > How about treating "this" as a literal and adding
> > > {testLiterals=true;} to L_NAME?
> >
> > What do you mean about threathing "this" as a literal? "this" is
> > actually "þessi" in my programming language, so the 
> auto-generated token
> > name chokes the C++ compiler. Besides, my lexing is made up of two
> > lexers, and neither of them are in the same file as the parser
> > definition (so there's no parser-lexer association).
> >
> > The document says to use the "literals" option and there's 
> a link to the
> > options page, but there is no "literals" option documented there.
> >
> > testLiterals is set to true for the whole grammar.
> >
> > Arnar
> >
> > > On Sunday 16 November 2003 20:30, Arnar Birgisson wrote:
> > > > Hello..
> > > >
> > > > I have those rules in my lexer:
> > > >
> > > > L_THIS : "this" ;
> > > > L_NAME : NameChar ( NameChar | '0'..'9' )* ;
> > > > protected NameChar : ( 'a'..'z' | etc. ) ;
> > > >
> > > > Now, the input "thisfile" is taken as two tokens,
> > >
> > > (L_THIS,"this") and
> > >
> > > > (L_NAME,"file"). How do I correct this. The obvious way 
> is to put
> > > > predicates in all the L_* rules I have.. but there must 
> be a simpler
> > > > way.
> > > >
> > > > I have read the documentation on literals in lexical
> > >
> > > anlyzers.. but I
> > >
> > > > can't figure out how to do this.
> > > >
> > > > thanks..
> > > >
> > > > Arnar
> > > >
> > > >
> > > >
> > > >
> > > > Your use of Yahoo! Groups is subject to
> > >
> > > http://docs.yahoo.com/info/terms/
> > >
> > >
> > >
> > >
> > > Your use of Yahoo!
> > > Groups is subject to http://docs.yahoo.com/info/terms/
> >
> > Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
> 
>  
> 
> Your use of Yahoo! 
> Groups is subject to http://docs.yahoo.com/info/terms/ 
> 
> 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/