[antlr-interest] Lexer code not generated as expected?

[Gavin Lambert]

At 08:25 16/12/2009, Peter Boughton wrote:
 >The ANTLR book mentions auto-backtracking (which seems to be 
what
 >is *not* happening here), and which can be turned on with 
"options
 >{backtrack=true;}"
 >
 >Would that not solve this problem?

Unfortunately not; backtracking is a parser feature and doesn't 
operate in the lexer.

 >If so, are there any issues (aside from performance) that might 
be
 >a reason not to have this turned on?

The biggest problem with it (other than performance) is that it 
can hide errors in the grammar, and lead to misleading error 
messages while processing input (since when it finally does 
produce an error, it's likely to be off in some weird part of the 
rules).


But yes, it would be nice if lexing ambiguity could be dealt with 
a little more automatically.  But at the end of the day, fixing it 
up yourself will improve performance, so it's not all bad news.