[antlr-interest] Lexer tokens

Donovan, Bob bob.donovan at ugsplm.com
Wed Mar 17 10:42:29 PST 2004


Hi,

I have a lexer rule which matches a string "value" as shown below. I needed to make this a Lexer rule because I have to set a trigger point (boolean) for multiple lexers.

The problem I am having is that sometimes the word "value" and other keywords have different uses..not always a keyword. Most of the keywords are are literals and get translated to thier keyword token in the testLiteralsTable() method. However the VALUE token is not getting translated because its not a Literal, its a lexer rule token.

How can I get around this?

Thanks, Bob Donovan



VALUE			:	"value" 
		{
			isPictureDefinition = false;
		}
		;



	public int testLiteralsTable(int ttype) { 
	 	// Test the token text against the literals table 
    		// Override this method to perform a different literals test
    		int oldType = ttype;
		hashString.setBuffer(text.getBuffer(), text.length()); 
		Integer literalsIndex = (Integer)literals.get(hashString);  
		
		if (literalsIndex != null) { 
			ttype = literalsIndex.intValue();
			// reset the type to ID, if the last token was a message argument
			// this resets the token to ID, just in case 
			// the source contains a keyword where an ID should be.
			if (isMsgArgName(ttype)) {
				ttype = ID;
			}
		} 
		
		secndPred = firstPred;
		firstPred = ttype;
		return ttype; 
	  }


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list