[antlr-interest] Lexer not backtracking properly?

Stevenson, Todd (GE Healthcare) Todd.T.Stevenson at ge.com
Fri May 21 08:31:13 PDT 2010


I have been working on a lexer grammar that appears to try to process
one rule but, 
when it fails, does not back track and find the correct rule.
 
Consider this grammar:
 
start
   : LBRACE option RBRACE EOF
   ;
 

option
   : DELETE
   | IMPORT
   | ALL
   ;
 
DELETE  : 'delete';
ALL     : 'all';
IMPORT  : 'import';
 
LBRACE    : '{';
 
RBRACE    : '}';
 
GUID    : LBRACE XDIGIT+ RBRACE;
 
HEXNUMBER : XDIGIT+;
 
WS        : ( ' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
 
fragment
XDIGIT     : '0' .. '9' | 'a' .. 'f' | 'A' .. 'F';
 
------------------------
 
When I use the following input '{all}' it fails because it appears to
try to process the 'GUID'
rule but when it fails, cannot backtrack and process the 'LBRACE' rule.
It works correctly
when I processing the input '{ all}'.  Is this correct behavior?   
 
I know that the GUID rule could probably be a parser rule but I wanted
to not allow embedded 
whitespace so I left it as a lexer rule.
 
 


More information about the antlr-interest mailing list