[antlr-interest] Lexer errors when looking for wrong token

A Z asicaddress at gmail.com
Sun Oct 10 15:31:29 PDT 2010


Hello,

I have a lexer with the following rules:


LBMINUSGT                  : '[->';
LBASRB                     : '[*]';
LBAST                      : '[*';
LBEQUALS                   : '[=';
LBPLUSRB                   : '[+]';
LBRACE                     : '{';
LBRACKET                   : '[';
MINUS                      : '-';

The lexer fails(with an error message) when any string of '[-' or '[*' is
detected. I'm confused why ANTLR cannot tokenize '[-' correctly as LBRACKET
MINUS. It also discards two characters after the failed token. I do not have
a static k defined and ANTLR generates no warnings when compiling. I'm still
debugging but it's slow figuring out how the antlr3dfapredict() function
works. Any help is appreciated.


Test input:
foo[-1]
foo[->saf]
foo[*saf]
foo[+saf]
foo[+]saf]
foo[0]

Test output :
frag.v(1) : lexer error 1 :
        Unexpected character at offset 5, near '1' :
        1]
Token:     SIMPLE_IDENT foo
Token:     SIMPLE_IDENT foo
Token:        LBMINUSGT [->
Token:     SIMPLE_IDENT saf
Token:         RBRACKET ]
Token:     SIMPLE_IDENT foo
Token:            LBAST [*
Token:     SIMPLE_IDENT saf
Token:         RBRACKET ]
Token:     SIMPLE_IDENT foo
Token:     SIMPLE_IDENT f
Token:         RBRACKET ]
Token:     SIMPLE_IDENT foo
Token:         LBPLUSRB [+]
Token:     SIMPLE_IDENT saf
Token:         RBRACKET ]
Token:     SIMPLE_IDENT foo
Token:         LBRACKET [
Token:           DECNUM 0
Token:         RBRACKET ]


More information about the antlr-interest mailing list