[antlr-interest] Howto modify token creation?

[David-Sarah Hopwood]

Indhu Bharathi wrote:
> You can do something like
> 
> ID	: LETTER (LETTER|DIGIT)*
> 	{
> 		String text = getText();
> 		Integer tknType;
> 		if( (tknType=table.get(text))!=null ) {
> 			$type = tknType;
> 		}
> 	}
> 
> The table can be passed to the lexer using some member function. But I don't
> know any clean way how to make sure ANTLR lexer doesn't assign the same int
> to some other token.

YourParser.tokenNames.length should be the first unused token number.

(Obviously, this is relying on an implementation detail, but probably a
fairly stable one.)

Note that if you use token numbers >= tokenNames.length, you should override
getErrorMessage in the parser, so that it doesn't throw an
ArrayIndexOutOfBoundsException when constructing an error message involving
one of these tokens. For example:

@parser::members {
  public String getErrorMessage(RecognitionException e, String[] names) {
    String[] dynamicNames = /* array also containing dynamic token names */;
    return super.getErrorMessage(e, dynamicNames);
  }
}

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com