[antlr-interest] Tokens match more characters than they should.

Jim Idle jimi at temporal-wave.com
Mon Mar 31 14:08:54 PDT 2008


Well, your lexer doesn't cover the incorrect input, so it is just throwing an error, which the interpreter isn't telling you about. use the debugger instead and you will see the lexer is saying something like "Mismatched character 'a'". The interpreter just shows that the it was consumed by the error recovery. In general a lexer should be able to consume anything and make a token of it, which can then be rejected later or handled better. Something like this as the last lexer rule:

 

 

ERRCHAR: .

   {

      // Call your error routines to say "Unexpected character at xxx" or something sensible

      ...

      $channel = HIDDEN;  // Hide this from the parser

   }

 

Jim

 

From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Chris Dew
Sent: Monday, March 31, 2008 1:37 PM
To: antlr-interest at antlr.org
Subject: [antlr-interest] Tokens match more characters than they should.

 

I'm sure that this is an error in my understanding, as I'm new to antlr.  Hopefully someone can point out my mistake.  I'm using ANTLRWorks 1.1.7, (which says that it is for antlr v3 grammars).

I believe the grammar, attached at the end of the email, should only match "foo bar wibble".  The *interpreter* (built into ANTLRWorks) matches many more characters than I expect.  For example: "fooabarb cwibble" seems to be valid.  It forms 'foo' 'a bar' 'b cwibble'.  Does anyone know why?

Thanks,

Chris.

P.S. The grammar is just a toy grammer, which illustrates a problem I have with a much more complicated grammar.



grammar test;

command    :    FOO BAR WIBBLE    ;

FOO    :    'foo'        ;
BAR    :    'bar'        ;
WIBBLE    :    'wibble'    ;

WS    :    (' '|'\r'|'\n')+ {$channel = HIDDEN;} ;        






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080331/39ce97ef/attachment.html 


More information about the antlr-interest mailing list