[antlr-interest] keywords testLiteral=true

[mzukowski at yci.com]

IDENTIFIER and DIVIDE have the same lookahead for IDENTIFIER.  So the lexer
is choosing DIVIDE instead of IDENTIFIER.  When DIVIDE calls IDENTIFIER it
overwrites the returned token type with its own.

Best solution is to make IDENTIFIER protected and test for literals in
DIVIDE (and not in IDENTIFIER).

Monty

-----Original Message-----
From: William Lam [mailto:xeenman at yahoo.com] 
Sent: Friday, February 20, 2004 11:29 AM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] keywords testLiteral=true

Hi all,

In my lexer I have a rule SLASH_IDENTIFIER that calls my IDENTIFIER 
rule.  However in the SLASH_IDENTIFIER rule, I get keywords that come 
out of my IDENTIFIER rule.  When I call the IDENTIFIER rule in the 
parser, I do not get keywords.

for example,

SLASH_IDENTIFIER matches this
/foo/keyword

however IDENTIFIER does not match
keyword

class SqlLexer extends Lexer;

options {
    exportVocab = Sql;
    testLiterals = false;
    k = 2;
    caseSensitive = false;
    caseSensitiveLiterals = false;
//    charVocabulary = '\3' .. '\177';
    charVocabulary='\u0000'..'\uFFFE';
}

tokens {
    keyword = "keyword";
}

IDENTIFIER options { testLiterals=true; }
    :    'a' .. 'z' ( 'a' .. 'z' | '0' .. '9' | '_' | '$' | '#' )*
    ;

DIVIDE
    :    '/'  { $setType(SLASH_IDENTIFIER ); }
               ( IDENTIFIER ( '/' IDENTIFIER )* )?
    ;



 
Yahoo! Groups Links



 


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/