[antlr-interest] Token.filename being ignored - solutions

Matt Benson gudnabrsam at yahoo.com
Fri Jun 6 12:03:13 PDT 2003


Since tokens are an open interface it is quite
possible that a token would not come from a file at
all.  If you need this why don't you subclass token to
have a filename and have your TokenStream
implementation then populate that attribute?

-Matt

--- antlrlist <antlrlist at yahoo.com> wrote:
> 
> 
> Terrence,
> 
> I've had this problem for a little while: filenames
> are usually 
> ignored by default in antlr. CommonToken does not
> take it in account, 
> and what's worse antlr.CharScanner.makeToken() does
> not fill it.
> 
> Could you please explain this? Is it a matter of
> efficiency?
> 
> According to this link:
> 
>
http://java.sun.com/docs/books/tutorial/java/data/stringsAndJavac.html
> 
> storing a String in CommonToken would add the
> overhead of managing 1 
> pointer - String objects are not created unless you
> modify them.
> 
> This is shown specially in this piece of code :
> 
>   String s1 = "hello";
>   String s2 = s1;
>   System.out.println("s1 = " + s1
>                    + "; s2 = " + s2);
>   System.out.println("System.identityHashCode(s1) =
> "
>                    + System.identityHashCode(s1));
>   System.out.println("System.identityHashCode(s2) =
> "
>                    + System.identityHashCode(s2));
> 
>   s1 += " world";
>   System.out.println("\ns1 = " + s1
>                    + "; s2 = " + s2);
>   System.out.println("System.identityHashCode(s1) =
> "
>                    + System.identityHashCode(s1));
>   System.out.println("System.identityHashCode(s2) =
> "
>                    + System.identityHashCode(s2));
> 
> Here's the output: 
> 
>   s1 = hello; s2 = hello
>   System.identityHashCode(s1) = 2452092
>   System.identityHashCode(s2) = 2452092
> 
>   s1 = hello world; s2 = hello
>   System.identityHashCode(s1) = 7474923
>   System.identityHashCode(s2) = 2452092
> 
> s1 points to a new address after " world" is
> appended. 
> 
> Cheers,
> 
> Enrique
> 
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/ 
> 
> 


__________________________________
Do you Yahoo!?
Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
http://calendar.yahoo.com

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list