[antlr-interest] v3 lexer problem

[Terence Parr]

Hi.  '.' in the parser matches a token not any random character.  You  
must be very careful how you draw the line between the lexer and the  
parser. also note that protected->fragment in version 3.
Ter
On Nov 27, 2006, at 12:40 AM, EXTERNAL Schnitzer Stephan (Praktikant;  
BEG-PG/EAP5) wrote:

> Hi,
>
> I use v3.0b5 and have a (probably simple) lexer problem.
>
> grammar Test;
> protected ALPHABET : ('a'..'z'|'A'..'Z');
> file    : ('T' . | 'Test')  EOF;
> ----------------
> public static void main(String[] args) throws Exception {	
>    TestLexer lexer = new TestLexer(new ANTLRStringStream("Te"));
>    CommonTokenStream tokens = new CommonTokenStream(lexer);
>    TestParser parser = new TestParser(tokens);
>    parser.file();
> }
> ----------------
>
> I expected "Te" matching to "'T' ."
> But it returns "[mTokens, mT6]: line 1:2 mismatched char: '?' on  
> line 1;
> expecting char 's'
> [file]: line 0:-1 state 0 (decision=1) no viable alt;
> token=[@-1,0:0='<no text>',<-1>,0:-1]"
>
> The method mTokens() reads "Te" and then tries to match "Test", which
> fails.
> However if I use
> file    : ('T' . | 'T''e''s''t')  EOF;
> everything works as expected. But it looks ugly, especially with long
> expressions!
>
> Does anybody know, how I can get the same behaviour using 'Test'  
> instead
> of 'T''e''s''t'?
>
> Regards,
> Stephan
>