[antlr-interest] please help on lexer rules antlr v3
John B. Brodie
jbb at acm.org
Thu Sep 28 08:18:42 PDT 2006
Pardon me for butting in...
Your grammar is incorrect but also exposes a bug in ANTLR (IMHO). See below.
>...snipped...
>My test string is:
>
>-- axi.reject
>
>followed by a single CRLF
>
>Any idea why the -- axi.reject is still consumed as a comment (on
>channel 99) ?
this in not what i see...
>...snipped...
>---------------------------- grammar --------------------------
>
>grammar TestParser;
>options {k=2; backtrack=true; memoize=true;}
>
>statement: (directive )+ EOF;
>
>directive: DIRECTIVE;
>
>
>LF : '\n' { channel=99; };
>
>CRLF : '\r' ('\n')? { channel=99; };
>
>TAB : '\t' { channel=99; };
>
>SPACE : ' ' { channel=99; };
>
>fragment
>ANYTHING_2_EOL: (~('\n'|'\r' ))* ('\n'|'\r'('\n')?);
>
>fragment
>DIRECTIVE: (SPACE | TAB )* ('axi.locate' | 'axi.reject');
>
>// Single-line comment
>SL_COMMENT: '--' ( DIRECTIVE | ( ANYTHING_2_EOL { channel=99;} ) );
>
>WS : ( TAB | SPACE | CRLF | LF )+ { channel=99; };
>
>
>---------------------------------------------------------------------------
actually, when i run your grammar, i get a parsing error indicating no match
was found for this text: -- axi.reject
in your directive rule you use the token DIRECTIVE. yet the lexer rule
DIRECTIVE is marked as a fragment (e.g. not visible to the parser).
Therefore the directive parsing rule will be never matched (because a
DIRECTIVE is never emitted by the lexer, you have told antlr that it is a
fragment of some other token).
change the directive rule to be:
directive: SL_COMMENT; /*and pick better name*/
and things work much better... ;-)
Question for Dr. Parr - shouldn't Antlr disallow parser references to lexer
fragments?
Hope this helps...
-jbb
More information about the antlr-interest
mailing list