[antlr-interest] Occasionally match hidden channel

[Zenzike]

> This will do it:
>
> ((TokenStream)input ).get( input.index()-1 ).sometokenmethodsuchasgetType()
> == SLASHSLASH

Thanks! I'm amazed at how friendly and quick to reply this community is!

Unfortunately, the problem is a little more subtle than that; there
might be real whitespace elements that I don't want to parse between
SLASHSLASH and OTHER.
However, I thought I solved this by using:

!((CommonTokenStream)input).getTokens(input.LT(-1).getTokenIndex(),
input.LT(1).getTokenIndex(), SLASHSLASH).isEmpty()

But it turns out the problem is more subtle than this ... consider the
following snippet of my grammar (the whole this is rather huge, but
hopefully this will be enough):

display
  : BDISPLAY item
      (  sep item
      | {!((CommonTokenStream)input).getTokens(input.LT(-1).getTokenIndex(),
input.LT(1).getTokenIndex(), SLASHSLASH).isEmpty()}?=>
//        {((LateChannelTokenStream)input).setTokenTypeChannel(SLASHSLASH,
Token.DEFAULT_CHANNEL);}
//        SLASHSLASH+
//        {((LateChannelTokenStream)input).resetTokenTypeChannel(SLASHSLASH);}
//        item
      )* EDISPLAY            -> ^(DISPLAY item+)
  ;

sep
  : SEMICOLON ;
...
SLASHSLASH : '\\\\' {$channel=HIDDEN;};


The trouble is that matching "item" by itself, rather than "sep item",
causes the "multiple alternatives" problem elsewhere in the grammar --
a separation token like "sep" is really needed
in this case. The trouble is that as it stands, the grammar seems
fine, but compiling gives me the following error (due to the semantic
predicate above):

Error creating resources for Aztex. Cause
java.util.NoSuchElementException: no such attribute: description in
template context [outputFile parser genericParser(...) cyclicDFA
if(dfa.specialStateSTs)_subtemplate anonymous cyclicDFAState
cyclicDFAEdge notPredicate evalPredicate(...)]

I expect this is because we need to tell the DFA that there is no
ambiguity here -- which wouldn't be possible for it to detect, since
SLASHSLASH is in the hidden channel.
to try and solve this I implemented a LateChannelTokenStream, that has
the following methods:

	protected int skipOffTokenChannels(int i) {
		int n = tokens.size();
		
		while ( i<n && ((Token)tokens.get(i)).getChannel()!=channel ) {
			if (channelOverrideMap!=null) {
				Integer channelI = (Integer)
					channelOverrideMap.get(new Integer(((Token)tokens.get(i)).getChannel()));
				
				if (channelI.intValue() == channel) {
					((Token)tokens.get(i)).setChannel(channel);
					break;
				}
			}
			i++;
		}
		return i;
	}
	
	public void resetTokenTypeChannel(int ttype) {
		if (channelOverrideMap != null)
			channelOverrideMap.remove(new Integer(ttype));
	}

The idea is to add the SLASHSLASH token to the default channel by
using setTokenTypeChannel(SLASHSLASH, DEFAULT_CHANNEL) where
necessary, then switching it back off with resetTokenTypeChannel()
when it isn't.

For some reason this isn't working at all, though I expect it's
because the DFA isn't enabling the actions when doing a lookahead, and
so the hidden SLASHSLASH isn't visible, and the DFA fails.

To try and counter this, I tried to force backtracking, with the
options {k=0; backtrack=true;} set for this rule, but that doesn't
seem to have helped either.

I know this is a fairly involved problem -- I can supply the rest of
my grammar if it helps, but does anybody have any suggestions?

Thanks lots,

zenzike