[antlr-interest] keywords appended to identifiers, suggestion s?

[Jim O'Connor]

Hi Dave,
  There is one solution.  There may be others.  A lexer is a CharScanner.
CharScanner has a method testLiteralsTable().

class L extends Lexer;


options {
  testLiterals = false;   // don't automatically test for literals
}
tokens {
   ....
}
{
    public int testLiteralsTable(int ttype) {       //overrides CharScanner
method
     //Useful variables:  text.getBuffer(), text.length()
     //ttype come in as NAME
     //I have built my own hash tables for keywords.  You might be able to
use the tables from the 
     // tokens section. 
    
     //psuedo code
      while( there are members in your tables){
         if(current member is substring of text){
           ttype = type of current member;
           break;
         }//if
       }//while
       return ttype;
    } //testliteral
}
NAME
options {testLiterals=true;}
:   ('A'..'Z')+
    ;


Jim


-----Original Message-----
From: gdave [mailto:dave.green at valley.net]
Sent: Thursday, June 12, 2003 9:32 AM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] keywords appended to identifiers, suggestions?


I have a simple grammar which needs to identify normal runs of 
characters as NAMES, but certain keywords as KEYWORDS. The tricky 
part which I'm unable to solved is that I would like to identify 
keywords even when they are appended to name. Furthermore, I would 
NOT like to identify them when they are embedded in a name. a grammar 
example that fails to meet this need: 

class P extends Parser; 
startRule
    :   
    n:NAME
        {System.out.println("name: " + n.getText());}
    | KEYWORD
        {System.out.println("keyword");}
    ;

class L extends Lexer;

protected 
KEYWORD : "FOO" | "BAR";

protected 
NAME:   ('A'..'Z')+
    ;

KEY_OR_NAME :
 (KEYWORD) => KEYWORD {$setType(KEYWORD);}
 | NAME {$setType(NAME);}
 ;

Test cases that work according to my wishes:
FOO DAVE -> keyword name:DAVE
DAVE FOO -> name:DAVE keyword
FOODAVE -> keyword name:DAVE


Test cases which the grammar mishandles:
DAVEFOO -> name:DAVE keyword
DAVEFOOSUFFIX -> name:DAVEFOOSUFFIX

How can modify the grammar to keep the NAME expression from 
swallowing the trailing keywords? If it did recognize trailing 
keywords, how can I keep it from splitting names because they have 
embedded keywords? thanks in advance for any suggestions. I'm on my 
fourth or fifth pouring over of the antlr manual and haven't found a 
construct that I have been able to use to any advantage in this 
problem. 

Dave



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/