AW: [antlr-interest] TokenStreamRecognitionException

[Koehne Kai]

Hi,
 
the problem is that the keywords defined in the tokens section are only detected if there is a lexer rule catching it already!
 
"The literals table [...] is checked after each token is matched, so that the literals effectively override the more general identifier pattern."
http://www.antlr.org/doc/lexer.html#Keywords_and_literals
 
There is no lexer rule that starts with "-", so the keyword MINUS is not detected either.
 
Regards,
 
Kai Koehne

________________________________

Von: antlr-interest-bounces at antlr.org im Auftrag von sunjigang8 at netscape.net
Gesendet: Di 06.06.2006 11:59
An: antlr-interest at antlr.org
Betreff: [antlr-interest] TokenStreamRecognitionException



I defined a token in lexer as  (  MINUS = "-"; ). I use ( minus:MINUS; ) in parser to parser the symbol "-". While TokenStreamRecognitionException was thrown and popped up a window says ( unexpected char: '-' ). I think the char is recongnised but could not form the expected token.

Literals in lexer are consistent with that of parser, that is,  ( public const int MINUS = 31; ) are generated at the beginning of both lexer and parser.

This must be a silly question. I wish you could give a clue.

Robert


options {
    language  = "CSharp";
    namespace = "MiniClean";
}
class MiniCleanLexer extends Lexer;
options {
            k=2;           
        }               
tokens
{
    IMPLEMENTATION="IMPLEMENTATION";
    FUNCTIONAL="FUNCTIONAL";
    STRATEGY="STRATEGY";
    ABSTYPE="ABSTYPE";
    FROM="FROM";
    IMPORT="IMPORT";
    SYSTEM="SYSTEM";
    DEFINITION="DEFINITION";
    MODULE="MODULE";
    MACRO="MACRO";
    TYPE="TYPE";
    RULE="RULE";
    IF="IF";
    OR="||";
    AND="&&";   
    DB_COLUMN= "::";
    BAR= "|";
    DERIVES= "->";
    SEMI= ";";
    COMMA= ",";
    LPAREN="(";
    RPAREN= ")";
    COLUMN= ":";
    LBRACKET= "[";
    RBRACKET= "]";
    PredefinedType = "=>";
    PLUS ="+";
    MINUS = "-";
    DOT=".";
    E="E";
    INT= "INT";
    REAL="REAL";
    CHAR="CHAR";
    BOOL="BOOL";
    STRING="STRING";
    FILE="FILE";
    PROCID="PROCID";
    WORLD="WORLD" ;
    TRUE="TRUE";
    FALSE="FALSE";
    ID_LOWER;
    ID_UPPER;
    FunnyID1;
    NUM;
}


//ID_LOWER: LowerCaseChar (RestChar)*;
//ID_UPPER: UpperCaseChar (RestChar)*;
//FunnyID1:  '@' | '#' | '$' | '%' | '^' | '&'| '?' |'*' | '/' | '='| '<' | '>' | '_' | '`' | '\''|'\"' ;

//NUM:(Digit)+;

protected UpperCaseChar: 'A'..'Z';
protected LowerCaseChar: 'a'..'z';
protected Digit: OctDigit |'8'|'9';
protected OctDigit: '0'..'7' ;
protected RestChar: LowerCaseChar | Digit| UpperCaseChar; //|CharDel|StringDel ; | Class1Char

//protected StringDenot: StringDel (Char)* StringDel ;

//protected Class1Char: '@' | '#' | '$' | '%' | '^' | '&'| '?' | '*' | '+' | '-' | '/' | '='| '<' | '>' | '_' | '.'| '`' | '\''|'\"' ;

protected Class2Char: '|' | '!' | '(' | ')' | '{' | '}'| '[' | ']' | ':' | ';' | '~' | ',' ;

protected Special: '\n' | '\r' | '\f' | '\b' | '\t'| '\\' ;//| StringDel | CharDel; // OctDigit OctDigit OctDigit ;

WS: 
(
     ' '
    |  {setTabSize(4);} '\t'         //NOTE (UpperCaseChar|LowerCaseChar|Class1Char|Class2Char)* ('\r''\n' ){newline();}
    | '\n'
    | '\r' '\n'{newline();}
   
) {$setType(Token.SKIP);} ;





__________________________________________________________________
Switch to Netscape Internet Service.
As low as $9.95 a month -- Sign up today at http://isp.netscape.com/register

Netscape. Just the Net You Need.

New! Netscape Toolbar for Internet Explorer
Search from anywhere on the Web and block those annoying pop-ups.
Download now at http://channels.netscape.com/ns/search/install.jsp