[antlr-interest] Custom tokens

[Alexey Demakov]

Hi

> How do i extend token class used by lexer to return tokens (say, to include 
> file name in every such token)?

1. Extend token class (e.g. antlr.CommonToken) to add your functionality
2. Set token class in lexer:
lexer.setTokenObjectClass( "<token object full class name>" );

Now lexer will create objects of your class.

3. Insert filter between lexer and parser to set additional fields 
(file name) in token objects. (I use it to link adjacent tokens).
Don't remember exaclty, may be lexer have some method that can be overriden
to set these fields at token construction time.

Regards,
Alexey

-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com