[antlr-interest] Re: lexer "modes" for XML parsing etc...

[Oliver Zeigermann]

2005/11/19, Terence Parr <parrt at cs.usfca.edu>:
>
> On Nov 19, 2005, at 1:30 PM, Oliver Zeigermann wrote:
> >> It will be very nice eventually when the parser can say "heh, I'm
> >> looking for an INT or an ID (and that's it)" so that messages are
> >> really good and the context is so specific it can built a smaller
> >> faster predictor.  This solves the List<List<int>> issue of the ">>"
> >> as it will only be looking for a '>' :)
> >
> > Oh, so I did not get the point. But, is it really possible to only ask
> > for a certain set of possible tokens "dynamically"? I mean, the lexer
> > does not know what the parser will ask for at generation time.
>
> Correct.  The parser would not call Tokens rule it would call a
> different start rule for each unique context in the grammar.  Well,
> unique in the sense that the set of tokens is different.
>
> If you have a rule:
>
> a : A
>    | B
>    | C
>    ;
>
> Rule 'a' might be able to tell the lexer to get one of A,B,C rather
> than any token.  I just don't know how lookahead would work in this
> environment.

Well, that's what I was talking about. How to do this *dynamically*?
Or would you suppose that the lexer knows all kinds of wishes the
parser may have? If it is statical only, lookahead should not be too
hard, you would just have to generate something you would when
something is a fragment, I guess.

Oliver