[antlr-interest] Accessing HIDDEN tokens in the C target.

Jim Idle jimi at temporal-wave.com
Thu Aug 28 16:33:02 PDT 2008


On Thu, 2008-08-28 at 16:16 -0700, Robin Green wrote:
> Hello!
> 
> 
> Longtime listener, firstime, er, caller.
> 
> 
> I am using the "C" target and I am trying to get access to the
> $channel=HIDDEN tokens from the parser, but only inside one rule. The
> FAQ and advice on the mailing list that I have read talk about
> subclassing the ANTLR3_COMMON_TOKEN_STREAM class and rewriting the
> skipOffTokenChannels() function to send back every token in the
> stream.


When you want to do this in one rule, just use get it directly starting
from the current index. Say you want to look back from a particular
token in a rule to see if there was a comment on channel 2 (please note
that I have not compiled this, just typed it in from memory). Basically
though you can do anything you can do in Java (more in fact), just by
looking a ta Java example and realizing that the C methods are all the
same names (just about) but instead of  x.y(z), you use x->y(x, z).

rule
: f=FUNCTION x y z
    {
        int sIndex;
       pANTLR3_COMMON_TOKEN tok;

       sIndex = $f->getTokenIndex($f) - 1;   // Index for first token
  
      // Now look back up looking for tokens
     tok = INPUT->get(INPUT, sIndex);

    if (tok->getChannel(tok) == whatever) { .... }

// and so on
    
    }
;

Use the C runtime for cribbing the functions and also the Doxygen API
docs (see API docs on antlr home page). You are well advised to read
through all of this. Also use the search at: http://antlr.markmail.org/
- it is very good.

If you really want to override a function, then copy the function from
the runtime as a start point and put it in your own source. Then in your
grammar use:

@parser::apifuncs
 {
    // Install custom error message display
    //
    RECOGNIZER->displayRecognitionError = produceError;
 }



This piece of code will run after all the standard methods are set up
and you can install your own version of any function whatsoever. No need
to alter the standard runtime.


> 
> 
> 1. It's unclear how to "subclass" the objects in the C target library,
> especially how to rewrite a "member" function.
> 
> 
> 2. Even if I do so, how do I switch between token stream as I enter a
> rule and restore it as I leave? I understand the examples that push
> the current stream onto a stack for "including" files, but these
> streams are complete in themselves, so when they swap back they only
> have to pick up where they left off. I really need to switch on and
> off the skipping of off-channel tokens.
> 
> 
> What's technique should I use? I am leaning towards writing my own
> "get()" in the lookahead:
> 
> 
> shader
>   : ( options{greedy=false;}: ( val = ~('technique')
>     {
>       ANTLR3_INT32 index = $val.index + 1;
>       ANTLR3_INT32 max_size = INPUT->istream->cachedSize;
>       while(index < max_size) {
>          pANTLR_COMMON_TOKEN token = INPUT->get(INPUT, index);
>          if (token->getChannel(token) == HIDDEN) {
>            // add lookahead token to the output
>         }
>       }
>     }
>     )* )
>   ;
> 
> 
> and using it inside a parser rule to directly diddle with the token
> stream, but I was wondering if there is a better way that doesn't
> involve me customizing the C target library (which will require
> re-customizing every time I update the antlr package). Is there a way
> to inject my own behavior into the C libs without recompiling it?
> 
> 
> 
> Thanks in advance,
> 
> 
> - Robin Green
>   Google Inc.
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080828/c9a926ba/attachment.html 


More information about the antlr-interest mailing list