[antlr-interest] Re: lexer "modes" for XML parsing etc...

[Oliver Zeigermann]

2005/11/19, Terence Parr <parrt at cs.usfca.edu>:
> On Nov 19, 2005, at 1:19 PM, Oliver Zeigermann wrote:
> > What you describe for v3 final really sounds like lexer modes which -
> > as far as I remember - you don't like?! Finally giving in? Hihihihi ;)
>
> I'm thinking of it as more of a goal-oriented lexer with multiple
> contexts.  Note that it's still better than modes in lex which people
> use to match strings etc...  You should use a separate rule in ANTLR
> for that.  For context-sensitive lexical regions in a single input
> stream, however, context-sensitive lexers can be implemented in a
> modal sense; here the mode implies context which I think is probably
> ok. ;)  I won't call it a mode though ;)
>
> It will be very nice eventually when the parser can say "heh, I'm
> looking for an INT or an ID (and that's it)" so that messages are
> really good and the context is so specific it can built a smaller
> faster predictor.  This solves the List<List<int>> issue of the ">>"
> as it will only be looking for a '>' :)

Oh, so I did not get the point. But, is it really possible to only ask
for a certain set of possible tokens "dynamically"? I mean, the lexer
does not know what the parser will ask for at generation time.

Or did I get it all wrong again?

Oliver