[antlr-interest] Very simple grammar confusing me
Joachim Schrod
jschrod at acm.org
Wed Nov 10 02:01:20 PST 2010
Oliver Zeigermann wrote:
> Folks!
>
> This is my grammar
>
> ------------------
> SHRASS : '>>=' ;
> SEMI : ';' ;
> GT : '>';
>
> rule : (GT | SEMI | SHRASS)+ ;
> ------------------
>
> I though it should parse
>
>>>;
>
> into a token stream of
>
> GT GT SEMI
>
> but as I see both at runtime as well is in the mToken method it tries
> to match the above input using SHRASS. Which of course fails.
>
> Any hints what I could do to work around that?
John gave you the hint...
To reduce the probability that this happens again: Your basic
problem above is that there is a prefix of a token that is not
covered by any other token rule, namely '>>'.
For ANTLR lexers, *all* possible prefixes of any token must be
matched by another token. Otherwise lexers will fail, as ANTLR
lexers can not backtrack.
The standard solution is to refactor the token grammar. If that
gets too difficult or unmaintainable, one can combine another lexer
generator with ANTLR stream and tree parser capabilities.
HTH,
Joachim
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Joachim Schrod Email: jschrod at acm.org
Roedermark, Germany
More information about the antlr-interest
mailing list