[antlr-interest] ANTLR 3 & hidden token management?

[Kay Roepke]

On 7. Aug 2006, at 11:24 Uhr, Loring Craymer wrote:

> From what I've seen, Martin tends to complain about design  
> missteps, and most of his complaints are accurate.  The complaint  
> here is that numbered channels are non-intuitive and it would be  
> better to support symbolic names.  I have to agree--and Ter  
> probably will too when he thinks about it.  One solution would be  
> to add a lexer option, "channels", which takes a string with comma- 
> separated names or some such.  ( channels = "HIDDEN, COMMENTS"; )   
> Or have a "channels" section in the lexer.

Loring and *,

I didn't want to sound unappreciative, so if I did come across that  
way, I'm sorry for that. I was just a bit bewildered by the  
proposition to
discuss syntax changes this late in the game (knowing how such  
discussions go, most of the time...).

As for the channel handling, I agree with you that named channels are  
more descriptive and more intuitive than an arbitrarily chosen  
number, e.g. 99.
I was just trying to make the point that this naming scheme shouldn't  
be handled in the default runtime, because there it could lead to  
misunderstandings as to how things work with channels, but that's  
essentially what you say, too.
Effectively, for the standard out-of-the-box CommonTokenStream class,  
all tokens not on the default channel (== 0) are hidden, so I'm a bit  
uneasy
to call channel 99 the "hidden channel", because every other channel,  
save one, are also "the" hidden channel. It really only makes things  
harder
to explain.
Now, if I plan to use multiple channels, I will hopefully have the  
guts to #define me a descriptive name for that and not use raw  
numbers all
over the place. But that's more the responsibility of the grammar  
author anyway.
Maybe it's more useful to be able to specify channelOverrideMap  
(which maps tokentypes to channels in the CommonTokenStream) somehow  
in the grammar, so that lives close to the definition of the tokens  
themselves, the lexer rules.

But I still think that there are more important things, like shipping  
3.0 final and when the ANTLR grammars are rewritten to be self- 
hosted, maybe
Ter can incorporate the outcome of such discussions as this into ANTLR.

Best regards,

-k
-- 
Kay Röpke <kroepke at classdump.org>
classdump Software