[antlr-interest] Tokens that span across char streams

[Stanislav Sokorac]

I guess the tricky thing will be to insert this functionality without
significantly adding to the run time.. If the stream has to check for
macros, and also mux between the regular stream and the macro definition,
I'm adding two 'if' checks on every single character. Maybe more if I'm also
selectively updating character positions.

I could have the lexer signal to the stream when the switch is needed to
remove one of those, at least.

Or am I over-optimizing here, is lexer already doing way more on every
character than I'm talking about here? I am going to be running into some
significantly large files, so I'd like to avoid overhead wherever I can...

Stan

On Wed, Aug 26, 2009 at 8:31 PM, David-Sarah Hopwood <
david-sarah at jacaranda.org> wrote:

> Stanislav Sokorac wrote:
> > Who keeps track of line numbers and character positions? The char stream,
> or
> > the lexer?
>
> The char stream. It's actually fairly straightforward to override in your
> own subclass -- look at the source code for the consume method of
> ANTLRStringStream, for example:
> <
> http://www.antlr.org/api/Java/_a_n_t_l_r_string_stream_8java-source.html#l00097
> >.
>
> --
> David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090826/e32dced5/attachment.html