[antlr-interest] lexer suggestions

[Micheal J]

Hi Andy,

> Forgive me if this has been discussed before or if it's not 
> even an issue with more recent versions of ANTLR. I'm using 
> ANTLR 2.7.2.
> 
> I have a couple of minor suggestions for ANTLR-generated lexers:
> 
> 1) Rather than (or in addition to):
> 
> lexer.setObjectClass("some.package.MyToken");
> 
> I'd like to be able to specify my own Token factory:
> 
> lexer.setTokenFactory(TokenFactory myfactory);

This is already present for ANTLR C# targets. The TokenCreator (as the
factory is called) is not an interface but an abstract base class (it
could/should have been an interface). For C#, creating instances via
reflection is [very?] expensive.

> 2) It would also be nice to have Token be an interface, and 
> maybe rename the current Token class to "BaseToken" or 
> something. It's not very likely that anyone would really need 
> their Token class to inherit from something else, but it's 
> possible. It just seems ugly to have those few methods in the 
> Token class that don't do anything. We have set/get methods 
> for column, line, filename, and none of them do anything in 
> the Token class.

Once again, C# has an IToken interface. The Token class still exists however
with all it's methods intact. Think of it as a badly named abstract base
that isn't...erm...abstract. I chose not to change it in the ANTLR 2.x line.
It would have been a gratuitious breaking change with not much benefit.

IToken allows you (in C#) to create an alternative Token class hierarchy if
you so please.

Cheers,

Micheal
ANTLR/C#