[antlr-interest] How to swich the recognition scope in Lexer

Micheal J open.zone at virgin.net
Wed Jun 20 05:10:45 PDT 2007


> Silvester Pozarnik wrote this on [20 June 2007 13:00]:
> 
> In the antlr 2.7.7 you could change the behaviour of Lexer so 
> that tokens are recognized az literals in special cases by 
> overriding the
> testLiteralsTable() method in CharScanner class. How to the 
> same in antlr 3.0 if you have a grammar as:
> 
> 	grammar test;
> 	tokens {
> 		MYTOKEN = 'mytoken';
> 	}
> 	mygrammar:
> 		{ 
> 		MYTOKEN LPAREN IDENTIFIER RPAREN 
> 		}
> 
> 	LPAREN   : '(' ;
> 	RPAREN   : ')' ;
> 	IDENTIFIER 
> 		: ('a'..'z' | 'A'..'Z' | '\u0080'..'\ufffe') ( 
> Letter | Digit)*;
>     
> 	fragment Letter
> 		: 'a'..'z' | 'A'..'Z' | '_' |'-' |  '\u0080'..'\ufffe';
> 
> 	fragment Digit
> 		: '0'..'9';    
> 
> So that the input "mytoken(mytoken)" is a valid. The first 
> 'mytoken' should be recognized as MYTOKEN, but the second 
> 'mytoken' has to be recognized as an IDENTIFIER. Is there a 
> way to achieve this?

Not to my knowledge (and this applies to V2.x too). Is suspect you need to
change your 'mygrammar' rule:

	mygrammar : MYTOKEN LPAREN (MYTOKEN|IDENTIFIER) RPAREN 

Micheal

-----------------------
The best way to contact me is via the list/forum. My time is very limited.



More information about the antlr-interest mailing list