[antlr-interest] Re: Token.filename being ignored - solutions
antlrlist
antlrlist at yahoo.com
Mon Jun 9 14:32:52 PDT 2003
I already tried that :)
The problem is that makeToken() is protected in antlr.CharScanner, so
no one can override it on the Lexer. I finally edited makeToken and
added the filename information to the new Token (it is very easy).
I tend to think that this is a bug; even if CommonToken does not
provide a "real" implementation setFilename and getFilename, the
CharScanner should call setFilename on the new token; this way using
a subclass of Token is very easy. Otherwise one would have to
manually call _token.setFilename(...) on every Lexer rule ...
What does the community think about this?
Thanks for the feedbak, Monty!
Enrique
--- In antlr-interest at yahoogroups.com, mzukowski at y... wrote:
> Override makeToken().
>
> Monty
>
> -----Original Message-----
> From: antlrlist [mailto:antlrlist at y...]
> Sent: Monday, June 09, 2003 7:30 AM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] Re: Token.filename being ignored -
solutions
>
>
>
> Anyway the CharScanner should fill the filename information in
> antlr.CharScanner.makeToken, should't it? If not, how do I add lex
> info to my Tokens?
>
> Enrique.
>
> --- In antlr-interest at yahoogroups.com, Matt Benson
<gudnabrsam at y...>
> wrote:
> > Since tokens are an open interface it is quite
> > possible that a token would not come from a file at
> > all. If you need this why don't you subclass token to
> > have a filename and have your TokenStream
> > implementation then populate that attribute?
> >
> > -Matt
> >
> > --- antlrlist <antlrlist at y...> wrote:
> > >
> > >
> > > Terrence,
> > >
> > > I've had this problem for a little while: filenames
> > > are usually
> > > ignored by default in antlr. CommonToken does not
> > > take it in account,
> > > and what's worse antlr.CharScanner.makeToken() does
> > > not fill it.
> > >
> > > Could you please explain this? Is it a matter of efficiency?
> > >
> > > According to this link:
> > >
> > >
> >
>
http://java.sun.com/docs/books/tutorial/java/data/stringsAndJavac.html
> > >
> > > storing a String in CommonToken would add the
> > > overhead of managing 1
> > > pointer - String objects are not created unless you
> > > modify them.
> > >
> > > This is shown specially in this piece of code :
> > >
> > > String s1 = "hello";
> > > String s2 = s1;
> > > System.out.println("s1 = " + s1
> > > + "; s2 = " + s2);
> > > System.out.println("System.identityHashCode(s1) =
> > > "
> > > + System.identityHashCode(s1));
> > > System.out.println("System.identityHashCode(s2) =
> > > "
> > > + System.identityHashCode(s2));
> > >
> > > s1 += " world";
> > > System.out.println("\ns1 = " + s1
> > > + "; s2 = " + s2);
> > > System.out.println("System.identityHashCode(s1) =
> > > "
> > > + System.identityHashCode(s1));
> > > System.out.println("System.identityHashCode(s2) =
> > > "
> > > + System.identityHashCode(s2));
> > >
> > > Here's the output:
> > >
> > > s1 = hello; s2 = hello
> > > System.identityHashCode(s1) = 2452092
> > > System.identityHashCode(s2) = 2452092
> > >
> > > s1 = hello world; s2 = hello
> > > System.identityHashCode(s1) = 7474923
> > > System.identityHashCode(s2) = 2452092
> > >
> > > s1 points to a new address after " world" is
> > > appended.
> > >
> > > Cheers,
> > >
> > > Enrique
> > >
> > >
> > >
> > >
> > >
> > > Your use of Yahoo! Groups is subject to
> > > http://docs.yahoo.com/info/terms/
> > >
> > >
> >
> >
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
> > http://calendar.yahoo.com
>
>
>
>
> Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list