[antlr-interest] Matching tokens on hidden channels

[Zenzike]

I am currently trying to match tokens on hidden channels -- this is
sometimes needed when a token is hidden, since generally, these tokens
should not be considered by the parser, but on occasion they should.

I saw a brief mention of "hidden token matching" in a wiki document
that Ter wrote [1], where the syntax:

.TOKEN

matches a particular hidden token TOKEN, and this is indicated by the
"." that precedes TOKEN. It seems that this functionality is not in
ANTLR 3.1.1 yet -- are there any plans to include it?

In order to try and solve this for what I have now, I've tried to use
semantic predicates along these lines:

rule0 : PREV ({NL()}?=> NEXT) ;

where PREV and NEXT are tokens that I want to separate with a hidden
NL token, and NL() is defined as such:

boolean NL() {
  int i = -1;
  while (input.LA(i) != Token.EOF && input.LT(-1).getChannel() == HIDDEN) {
    if (input.LA(i) == NL)
      return true;
    i--;
  }
  return false;
}

This works fine if NEXT is a token, but when it's a rule, I get an
error. For example, where I have:

rule1 : PREV ({NL()}?=> rule2);

I get the following error if I compile with the -debug flag (no error
otherwise):

ANTLR Parser Generator  Version 3.1.1
error(10):  internal error: AztexParser.g :
java.util.NoSuchElementException: no such attribute: description in
template context [outputFile parser genericParser(...) cyclicDFA
if(dfa.specialStateSTs)_subtemplate anonymous cyclicDFAState
cyclicDFAEdge notPredicate evalPredicate(...)]
org.antlr.stringtemplate.StringTemplate.checkNullAttributeAgainstFormalArguments(StringTemplate.java:1276)
org.antlr.stringtemplate.StringTemplate.get(StringTemplate.java:800)

Is this a bug in my code, or is it a problem with debug mode? Is there
another way of achieving what I'm looking for?

Any help would be greatly appreciated.

zenzike


[1] http://www.antlr.org/wiki/display/~admin/2007/06/28/Token+stream+rewriting+with+rewrite+rules