[antlr-interest] FailedPredicateException leads to infinite loop - bug in the Lexer?

[Cliff Hudson]

So I ended up doing something I think a little similar to this, except that
I get START_TAG followed by UNEXPECTED rather than the other way around - I
could probably fix this by having an action which tosses the appropriate
token or something.  I have managed to keep all modes strictly in the lexer
though, and it is now tolerant of every syntax error I've thrown at it.

Thanks for the assistance.

On Wed, Mar 31, 2010 at 11:27 AM, Ron Burk <ronburk at gmail.com> wrote:

> > My question is then what is
> > the appropriate way to construct the lexer such that it will recover
> > gracefully from that invalid input and NOT go into the infinite loop
> state
> > caused by the thrown exception?
>
> Personally, I try to keep "modes" in the lexer and
> out of the parser. I probably would have had the
> lexer looking past any initial '<' to distinguish the
> various types of things it presages (especially
> since the XML spec seems to make that relatively
> easy). So, for example, this:
>
> > <Program><</Program>
>
> would have returned a token stream like:
>
> TK_START_TAG
> TK_IDENT
> TK_GT
> TK_ILLEGAL
> TK_END_TAG
> TK_IDENT
> TK_GT
>
> I would keep modes like CDATA in the lexer.
> YMMV, many ways to skin a cat, etc.
>