[antlr-interest] How to swich the recognition scope in Lexer

Silvester Pozarnik silvester.pozarnik at tracetracker.com
Wed Jun 20 04:59:49 PDT 2007


In the antlr 2.7.7 you could change the behaviour of Lexer so that
tokens are recognized az literals in special cases by overriding the
testLiteralsTable() method in CharScanner class. How to the same in
antlr 3.0 if you have a grammar as:

	grammar test;
	tokens {
		MYTOKEN = 'mytoken';
	}
	mygrammar:
		{ 
		MYTOKEN LPAREN IDENTIFIER RPAREN 
		}

	LPAREN   : '(' ;
	RPAREN   : ')' ;
	IDENTIFIER 
		: ('a'..'z' | 'A'..'Z' | '\u0080'..'\ufffe') ( Letter |
Digit)*;
    
	fragment Letter
		: 'a'..'z' | 'A'..'Z' | '_' |'-' |  '\u0080'..'\ufffe';

	fragment Digit
		: '0'..'9';    

So that the input "mytoken(mytoken)" is a valid. The first 'mytoken'
should be recognized as MYTOKEN, but the second 'mytoken' has to be
recognized as an IDENTIFIER. Is there a way to achieve this?

BR
Silvester Pozarnik


More information about the antlr-interest mailing list