[antlr-interest] Re: keywords testLiteral=true

William Lam xeenman at yahoo.com
Fri Feb 20 12:05:42 PST 2004


I changed my identifier rule to be protected, but my SLASH_IDENTIFIER 
rule still absorbs up my keywords

IDENTIFIER options { testLiterals=true; }
    :    ID
    ;

protected
ID
    :    'a' .. 'z' ( 'a' .. 'z' | '0' .. '9' | '_' | '$' | '#' )*
    ;

SLASH_IDENTIFIER options { testLiterals=true;}
    :   ('/' ID)+
    ;

--- In antlr-interest at yahoogroups.com, mzukowski at y... wrote:
> IDENTIFIER and DIVIDE have the same lookahead for IDENTIFIER.  So 
the lexer
> is choosing DIVIDE instead of IDENTIFIER.  When DIVIDE calls 
IDENTIFIER it
> overwrites the returned token type with its own.
> 
> Best solution is to make IDENTIFIER protected and test for literals 
in
> DIVIDE (and not in IDENTIFIER).
> 
> Monty
> 
> -----Original Message-----
> From: William Lam [mailto:xeenman at y...] 
> Sent: Friday, February 20, 2004 11:29 AM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] keywords testLiteral=true
> 
> Hi all,
> 
> In my lexer I have a rule SLASH_IDENTIFIER that calls my IDENTIFIER 
> rule.  However in the SLASH_IDENTIFIER rule, I get keywords that 
come 
> out of my IDENTIFIER rule.  When I call the IDENTIFIER rule in the 
> parser, I do not get keywords.
> 
> for example,
> 
> SLASH_IDENTIFIER matches this
> /foo/keyword
> 
> however IDENTIFIER does not match
> keyword
> 
> class SqlLexer extends Lexer;
> 
> options {
>     exportVocab = Sql;
>     testLiterals = false;
>     k = 2;
>     caseSensitive = false;
>     caseSensitiveLiterals = false;
> //    charVocabulary = '\3' .. '\177';
>     charVocabulary='\u0000'..'\uFFFE';
> }
> 
> tokens {
>     keyword = "keyword";
> }
> 
> IDENTIFIER options { testLiterals=true; }
>     :    'a' .. 'z' ( 'a' .. 'z' | '0' .. '9' | '_' | '$' | '#' )*
>     ;
> 
> DIVIDE
>     :    '/'  { $setType(SLASH_IDENTIFIER ); }
>                ( IDENTIFIER ( '/' IDENTIFIER )* )?
>     ;
> 
> 
> 
>  
> Yahoo! Groups Links



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list