[antlr-interest] Comments / White-Space and All The Rest

[Terence Parr]

Hi. You are probably trying to execute code with in the lexer, but  
you should really simply matched lexical items in the lexer and leave  
processing to the parser.  Place comments on a hidden token channel  
so that the parser does not see them in terms of parsing, but actions  
can still access the comments.  All non-skipped tokens are sent to  
the parser in the token stream and just by looking backwards and  
forwards in the token stream you can see all of the hidden channel  
tokens. For example, at the start of the method, look one token to  
the left of the first token of the method definition and you will  
find out if there is a comment there. :)

CommonTokenStream is your friend. :)

Ter
On Nov 24, 2006, at 8:21 AM, Randall R Schulz wrote:

> Hi,
>
> I would like to integrate comments into the stream of top-level
> constructs recognized by my ANTLR (v3) grammar.
>
> Specifically, I'd like to treat comments in the input something like
> JavaDoc comments in that I accumulate them (merge their contents in a
> single StringBuilder instance) as they appear and then when the parser
> encounters a top-level construct (the analogy would be a method or
> field definition in Java source code) associate the accumulated
> comment, if any, with that construct.
>
> I've had partial success, but there's one problem: It appears all the
> comments are processed before any of the parsing happens. Thus the
> comments don't properly interleave with the non-comment content and
> don't get associated with the proper constructs. Instead, they all
> appear at the beginning, merged into a single comment block.
>
> How can I get the kind of behavior I want? Namely that comment action
> code is processed in sequence with the parsing process?
>
>
> Thanks.
>
> Randall Schulz