[antlr-interest] Does the C runtime support token rewrite?

[Kay Röpke]

Hi Andy!

On Apr 12, 2009, at 6:59 PM, Andy Grove wrote:

> My grammar defines whitespace as:
>
> Whitespace : ( '\t' | ' ' | '\r' | '\n' )+ 	{ $channel = HIDDEN; }
> 	;

funny, that should cause them to be there.

> Do you know how I can access tokens on the hidden channel in C? I  
> also generate Java parser from the same grammer and the whitespace  
> tokens are available there.

After setting up my inputstream/lexer/tokenstream, i'm doing this:

	ctx->token_vector = ctx->tokens->getTokens(ctx->tokens);

Where ctx->token_vector is
  	pANTLR3_VECTOR token_vector;

Later I'm retrieving the tokens by index (because this entire thing is  
wired up to a script language - which uses 1-based indices, hence the  
-1):

	token = token_vector->get(token_vector, ndx-1);

I'm definitely seeing all tokens on all channels. I'm assuming you  
haven't set any channels to be discarded using discardOffChannel or  
discardSet?
Discarding happens during fillBuffer, so those tokens would never end  
up in the tokens vector.

Which runtime library version are you using (I'm on the latest  
released one, 3.1.3 IIRC)? Worst case I'd fire up the debugger and  
watch fillBuffer() do its thing.

cheers,
-k