[antlr-interest] Additional tokens not being passed to Create(IToken token) ?

[Dejas Ninethousand]

Consider the following grammar file snippet:

grammar CSharpDoc;

options {
    language='CSharp';
    output=AST;
    backtrack=true;
}

tokens {
NAMESPACE_COMPLETE; // I am a imaginary token, I have no definition
elsewhere in the file.
}

...

namespace_decl
    :    NAMESPACE IDENT -> NAMESPACE IDENT NAMESPACE_COMPLETE;

NAMESPACE
    :     'namespace';

IDENT
    :   ID_START (ID_PART)*
    ;

...


I am using a custom TreeAdaptor and have overloaded the Create(IToken token)
method to create my own SyntaxTree internally "on the side" within my custom
tree adapter as tokens flow through Create.  I have used this technique
before with other grammars without a problem.  But for some reason in this
implementation I never see any NAMESPACE_COMPLETE tokens passed to Create.
So, for example:

  public override object Create(IToken token)
        {
            if (token != null)
            {
                switch (token.Type)
                {
                    case(CSharpDocLexer.NAMESPACE_COMPLETE):
                    {
                            // program never reaches this case
                    }
                      ... // do custom tree construction in the rest of the
function
                }
            }
            return base.Create(token);
       }

I can't for the life of me figurure out why, in this instance, i'm not
seeing that extra token.

So for some statement like "namespace cat" I would expect to see the
following tokens pass trough the Create method in this order:

NAMESPACE
IDENT
NAMESPACE_COMPLETE

but what I am seeing is:

NAMESPACE
IDENT

What am I missing?

Thanks,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20081021/4f2f933d/attachment.html