[antlr-interest] Literals in lexer

Cristian Amitroaie cristian at amiq.ro
Wed Nov 19 14:05:18 PST 2003


I was thinking about adding a tokens section in the lexer:

tokens {
   "this"
};

or whatever you have, eventually giving it a name to avoid any compiler 
complaints, ... shortly see the attached example, you'll figure it out...

Some log (# where "þessi" is your "this"):
---
[cristian at lapc antlr_interest]$ java Main this_lexer.test
6 : is
4 : þessi // note the different token id
6 : lexer
6 : good
6 : enough
6 : þessilexer
6 : is
6 : good
6 : enough
6 : indeed
---

Let me know it works ,
Cristian

On Tuesday 18 November 2003 19:46, Arnar Birgisson wrote:
> Hello
>
> > How about treating "this" as a literal and adding
> > {testLiterals=true;} to L_NAME?
>
> What do you mean about threathing "this" as a literal? "this" is
> actually "þessi" in my programming language, so the auto-generated token
> name chokes the C++ compiler. Besides, my lexing is made up of two
> lexers, and neither of them are in the same file as the parser
> definition (so there's no parser-lexer association).
>
> The document says to use the "literals" option and there's a link to the
> options page, but there is no "literals" option documented there.
>
> testLiterals is set to true for the whole grammar.
>
> Arnar
>
> > On Sunday 16 November 2003 20:30, Arnar Birgisson wrote:
> > > Hello..
> > >
> > > I have those rules in my lexer:
> > >
> > > L_THIS : "this" ;
> > > L_NAME : NameChar ( NameChar | '0'..'9' )* ;
> > > protected NameChar : ( 'a'..'z' | etc. ) ;
> > >
> > > Now, the input "thisfile" is taken as two tokens,
> >
> > (L_THIS,"this") and
> >
> > > (L_NAME,"file"). How do I correct this. The obvious way is to put
> > > predicates in all the L_* rules I have.. but there must be a simpler
> > > way.
> > >
> > > I have read the documentation on literals in lexical
> >
> > anlyzers.. but I
> >
> > > can't figure out how to do this.
> > >
> > > thanks..
> > >
> > > Arnar
> > >
> > >
> > >
> > >
> > > Your use of Yahoo! Groups is subject to
> >
> > http://docs.yahoo.com/info/terms/
> >
> >
> >
> >
> > Your use of Yahoo!
> > Groups is subject to http://docs.yahoo.com/info/terms/
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Main.java
Type: text/x-java
Size: 619 bytes
Desc: not available
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20031120/1c3caff5/Main.bin
-------------- next part --------------
is þessi lexer good enough
þessilexer is good enough indeed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ThisLexer.g
Type: text/x-java
Size: 604 bytes
Desc: not available
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20031120/1c3caff5/ThisLexer.bin


More information about the antlr-interest mailing list