[antlr-interest] Possible bug in code generated for Lexer?

[Jim Idle]

Yeah - I think that you need to implement your own token stream as well
as Token. I suspect that if all you want to do is track the file names
then keep a map somewhere in the lexer you can just store an input
reference and the file name (and anything else) then find the file using
the input of the token as a reference.

Because such things are awkward in C, I added some custom fields to the
standard token - a few integers and a void *, so strangely, it is
probably easier to do little things like this. However I don't think it
is difficult to override the stuff in Java - you just have to find the
right example to crib off.

I did think that you work this out from the input, but now I remember
that I wanted to talk to Ter about this for some reason, and now cannot
remember exactly what it was I was going to say but it may have been to
have a setId(String) String getId() pair in the IntStream interface, or
something similar.

Jim



> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Gevik Babakhani
> Sent: Wednesday, June 20, 2007 8:35 AM
> To: Tim Clark
> Cc: antlr-interest
> Subject: Re: [antlr-interest] Possible bug in code generated for
Lexer?
> 
> I am new to this, but I think this is not a bug. See THE BOOK page
> 114/5.10
> the SL3Token will be generated in the Parser code not in the Lexer.
> 
> Hope it helps.
> 
> 
> > Hi
> >
> > I reported this before and got no response from anyone. Hopefully
> this
> > time
> > I'll be more lucky.
> >
> > I have the option:
> >      TokenLabelType=SL3Token
> >
> > in my parser, where SL3Token extends CommonToken. [I want to save
the
> name
> > of included file in the tokens.]
> >