[antlr-interest] Re: Token.filename being ignored - solutions

antlrlist antlrlist at yahoo.com
Mon Jun 9 07:30:09 PDT 2003


Anyway the CharScanner should fill the filename information in 
antlr.CharScanner.makeToken, should't it? If not, how do I add lex 
info to my Tokens?

Enrique.

--- In antlr-interest at yahoogroups.com, Matt Benson <gudnabrsam at y...> 
wrote:
> Since tokens are an open interface it is quite
> possible that a token would not come from a file at
> all.  If you need this why don't you subclass token to
> have a filename and have your TokenStream
> implementation then populate that attribute?
> 
> -Matt
> 
> --- antlrlist <antlrlist at y...> wrote:
> > 
> > 
> > Terrence,
> > 
> > I've had this problem for a little while: filenames
> > are usually 
> > ignored by default in antlr. CommonToken does not
> > take it in account, 
> > and what's worse antlr.CharScanner.makeToken() does
> > not fill it.
> > 
> > Could you please explain this? Is it a matter of
> > efficiency?
> > 
> > According to this link:
> > 
> >
> 
http://java.sun.com/docs/books/tutorial/java/data/stringsAndJavac.html
> > 
> > storing a String in CommonToken would add the
> > overhead of managing 1 
> > pointer - String objects are not created unless you
> > modify them.
> > 
> > This is shown specially in this piece of code :
> > 
> >   String s1 = "hello";
> >   String s2 = s1;
> >   System.out.println("s1 = " + s1
> >                    + "; s2 = " + s2);
> >   System.out.println("System.identityHashCode(s1) =
> > "
> >                    + System.identityHashCode(s1));
> >   System.out.println("System.identityHashCode(s2) =
> > "
> >                    + System.identityHashCode(s2));
> > 
> >   s1 += " world";
> >   System.out.println("\ns1 = " + s1
> >                    + "; s2 = " + s2);
> >   System.out.println("System.identityHashCode(s1) =
> > "
> >                    + System.identityHashCode(s1));
> >   System.out.println("System.identityHashCode(s2) =
> > "
> >                    + System.identityHashCode(s2));
> > 
> > Here's the output: 
> > 
> >   s1 = hello; s2 = hello
> >   System.identityHashCode(s1) = 2452092
> >   System.identityHashCode(s2) = 2452092
> > 
> >   s1 = hello world; s2 = hello
> >   System.identityHashCode(s1) = 7474923
> >   System.identityHashCode(s2) = 2452092
> > 
> > s1 points to a new address after " world" is
> > appended. 
> > 
> > Cheers,
> > 
> > Enrique
> > 
> > 
> > 
> >  
> > 
> > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/ 
> > 
> > 
> 
> 
> __________________________________
> Do you Yahoo!?
> Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
> http://calendar.yahoo.com


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list