[antlr-interest] Comments / White-Space and All The Rest

Randall R Schulz rschulz at sonic.net
Fri Nov 24 11:30:31 PST 2006


Terence,

On Friday 24 November 2006 10:45, Terence Parr wrote:
> Hi. You are probably trying to execute code with in the lexer, but
> you should really simply matched lexical items in the lexer and leave
> processing to the parser. Place comments on a hidden token channel 
> so that the parser does not see them in terms of parsing,

Naturally, that's what I do.


> but actions 
> can still access the comments.

How? The Wiki page 
<http://www.antlr.org/wiki/pages/viewpage.action?pageId=1057> alludes 
to doing this, but lack specifics on how to code such actions.


> All non-skipped tokens are sent to 
> the parser in the token stream and just by looking backwards and
> forwards in the token stream you can see all of the hidden channel
> tokens. For example, at the start of the method, look one token to
> the left of the first token of the method definition and you will
> find out if there is a comment there. :)

How do I get at the tokens? How do I navigate the token stream? I notice 
that TokenStream.LB(int) is protected, so I cannot even call it from 
parse action code.

I also notice that rules that return a single value (which is all but 
two of the rules in my grammar) don't trigger the generation of return 
types that descend from ParserRuleReturnScope. In those cases, how do I 
get at the start and stop tokens?


> CommonTokenStream is your friend. :)
>
> Ter


Randall Schulz


More information about the antlr-interest mailing list