[antlr-interest] Accessing HIDDEN tokens in the C target.

[Robin Green]

On Thu, Aug 28, 2008 at 5:39 PM, Robin Green <rgreen at google.com> wrote:

>
>
> On Thu, Aug 28, 2008 at 5:07 PM, Jim Idle <jimi at temporal-wave.com> wrote:
>
>> Yes but get returns a pointer to a TOKEN, which has a super pointer to the
>> common token (off the top of my head), if it is returning a
>> pANTLR3_COMMON_TOKEN already (it might be) then you don't need the super.
>>
>>
> get() in antlr3tokenstream.c returns a pANTLR3_COMMON_TOKEN.
>
> My problem is that I'm walking through this stream and I never see a token
> that's not "channel=0". My iinput source has plenty of inline comments and
> whitespace, so I'm starting to suspect that I'm calling the wrong "get()"
> here.
>
> #define    INPUT PARSER->tstream
> #define ISTREAM INPUT->istream
>
> Since I use "ANTLR3_INT32 max_size = INPUT->istream->cachedSize;" to get
> the number of tokens in the stream, should I be using ISTREAM or INPUT in my for my get() call?
> What's the difference?
>
>
Apologies for a dumb question there. Please ignore it.

It seems my out of channel tokens are being discarded by the tokenizer in
the function "fillBuffer()" in antlr3tokenstream.c, so there never were any
tokens in the stream to find. How do I tell the tokenizer not to discard my
tokens? I'm guessing adding a

   token_stream->discardOffChannel = FALSE;

just after creating my pANTLR3_COMMON_TOKEN_STREAM would do the job? Is
there a standard function I should be using instead?

- Robin Green
  Google, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080828/5b8acf57/attachment.html