[antlr-interest] Lexer Bug?

[Alexey Demakov]

Hi Adam,

It's because ANTLR uses so-called approximate lookahead
http://www.antlr.org/doc/glossary.html#Linear_approximate_lookahead
instead of full LL(k) lookahead.

If you define each token in its own protected lexer rule, it will solve the problem
for any order of rules (assuming k >= 2):

protected LG : "<>" ;

protected NE : "!=" ;

NOT_EQUALS : LG | NE ;

LESS_THAN_OR_EQUALS : "<=" ;

Regards,
Alexey
---
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com
 
  ----- Original Message ----- 
  From: Adam Bishop (DSLWN) 
  To: antlr-interest at antlr.org 
  Sent: Friday, February 10, 2006 6:07 AM
  Subject: [antlr-interest] Lexer Bug?


  If I define a lexer rule:

  NOT_EQUALS : "<>" | "!=" ;

   

  It matches the string "<="

   

  The lexer nextToken method shows:

   

  else if ((LA(1)=='!'||LA(1)=='<') && (LA(2)=='='||LA(2)=='>') && (true) && (true) && (true) && (true) && (true) && (true)) {

   

  which is flawed.

   

  I have solved my problem for this by moving the definition of LESS_THAN_OR_EQUALS above the definition of NOT_EQUALS.

   

  Is this a bug, or am I doing something flawed?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20060210/ec2f2226/attachment.html