[antlr-interest] Why won't this match...
Gavin Lambert
antlr at mirality.co.nz
Sun Feb 24 23:51:51 PST 2008
At 12:27 25/02/2008, Mark Volkmann wrote:
>I didn't see your post until after I sent my last post.
>Does that mean it isn't possible to do what Alan wants with the
>current lexer implementation or could the grammar be modified to
>do what he wants?
One of the standard workarounds to the "overly optimistic lexer
problem" (as I call it) is to merge all lexer rules with common
prefixes:
fragment BIG_TOKEN 'wibbled'; // just a placeholder
LITTLE_TOKEN
: 'wi'
( ('bbled') => 'bbled' { $type = BIG_TOKEN; } )?
;
SEMI_TOKEN: 'bble';
This works because the syntactic predicate forces a full lookahead
before the branch is taken. BIG_TOKEN is declared as a fragment
since we need to define a token type id for it, and declaring it
in the tokens section causes a warning message for some weird
reason. Creating a throwaway fragment rule handles both of these.
(The other standard workaround is to have a keyword matching table
as Ter already suggested.)
More information about the antlr-interest
mailing list