[antlr-interest] Re: Token.filename being ignored - solutions

[mzukowski at yci.com]

See the gcc code for the lexer.  I override makeToken() just fine.  You do
it by declaring that method in the generated lexer (in your .g file).  It's
a subclass of antlr.CharScanner, so it's ok to override it.

http://www.codetransform.com/gcc.html

Monty

-----Original Message-----
From: antlrlist [mailto:antlrlist at yahoo.com] 
Sent: Monday, June 09, 2003 2:33 PM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] Re: Token.filename being ignored - solutions



I already tried that :)

The problem is that makeToken() is protected in antlr.CharScanner, so 
no one can override it on the Lexer. I finally edited makeToken and 
added the filename information to the new Token (it is very easy).

I tend to think that this is a bug; even if CommonToken does not 
provide a "real" implementation setFilename and getFilename, the 
CharScanner should call setFilename on the new token; this way using 
a subclass of Token is very easy. Otherwise one would have to 
manually call _token.setFilename(...) on every Lexer rule ...

What does the community think about this?

Thanks for the feedbak, Monty!

Enrique

--- In antlr-interest at yahoogroups.com, mzukowski at y... wrote:
> Override makeToken().
> 
> Monty
> 
> -----Original Message-----
> From: antlrlist [mailto:antlrlist at y...]
> Sent: Monday, June 09, 2003 7:30 AM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] Re: Token.filename being ignored - 
solutions
> 
> 
> 
> Anyway the CharScanner should fill the filename information in
> antlr.CharScanner.makeToken, should't it? If not, how do I add lex 
> info to my Tokens?
> 
> Enrique.
> 
> --- In antlr-interest at yahoogroups.com, Matt Benson
<gudnabrsam at y...> 
> wrote:
> > Since tokens are an open interface it is quite
> > possible that a token would not come from a file at
> > all.  If you need this why don't you subclass token to
> > have a filename and have your TokenStream
> > implementation then populate that attribute?
> > 
> > -Matt
> > 
> > --- antlrlist <antlrlist at y...> wrote:
> > > 
> > > 
> > > Terrence,
> > > 
> > > I've had this problem for a little while: filenames
> > > are usually
> > > ignored by default in antlr. CommonToken does not
> > > take it in account,
> > > and what's worse antlr.CharScanner.makeToken() does
> > > not fill it.
> > > 
> > > Could you please explain this? Is it a matter of efficiency?
> > > 
> > > According to this link:
> > > 
> > >
> > 
> 
http://java.sun.com/docs/books/tutorial/java/data/stringsAndJavac.html
> > > 
> > > storing a String in CommonToken would add the
> > > overhead of managing 1
> > > pointer - String objects are not created unless you modify them.
> > > 
> > > This is shown specially in this piece of code :
> > > 
> > >   String s1 = "hello";
> > >   String s2 = s1;
> > >   System.out.println("s1 = " + s1
> > >                    + "; s2 = " + s2);
> > >   System.out.println("System.identityHashCode(s1) =
> > > "
> > >                    + System.identityHashCode(s1));
> > >   System.out.println("System.identityHashCode(s2) =
> > > "
> > >                    + System.identityHashCode(s2));
> > > 
> > >   s1 += " world";
> > >   System.out.println("\ns1 = " + s1
> > >                    + "; s2 = " + s2);
> > >   System.out.println("System.identityHashCode(s1) =
> > > "
> > >                    + System.identityHashCode(s1));
> > >   System.out.println("System.identityHashCode(s2) =
> > > "
> > >                    + System.identityHashCode(s2));
> > > 
> > > Here's the output:
> > > 
> > >   s1 = hello; s2 = hello
> > >   System.identityHashCode(s1) = 2452092
> > >   System.identityHashCode(s2) = 2452092
> > > 
> > >   s1 = hello world; s2 = hello
> > >   System.identityHashCode(s1) = 7474923
> > >   System.identityHashCode(s2) = 2452092
> > > 
> > > s1 points to a new address after " world" is
> > > appended.
> > > 
> > > Cheers,
> > > 
> > > Enrique
> > > 
> > > 
> > > 
> > >  
> > > 
> > > Your use of Yahoo! Groups is subject to
> > > http://docs.yahoo.com/info/terms/
> > > 
> > > 
> > 
> > 
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
> > http://calendar.yahoo.com
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/