[antlr-interest] Preserving ALL comments!

[Andy Tripp]

Damir Kirasić wrote:

> Thank you for your answer.
>
> Yes, what you suggest is all right but
> it requires a lot of work. Completely new
> layer between lexer and parser.
> And we already have that hidden channels and hidden
> objects that are supposed to solve such problems.
>
> Any easier solution?
>
> Damir
>
To just remove comments/whitespace shouldn't be more than a few lines of 
code. Just loop through the List
of tokens, removing all the ones of type Newline, Whitespace, Comment, 
and CPPComment. I wouldn't call
that a whole "layer", just a few lines of code that you'd quickly write 
in Java (or whatever) that will accomplish
what you want.

And the hidden channels seem like the wrong solution...they split the 
comments and whitespace into
a separate stream of tokens to be independently processed. But if you 
really want to know which
comments go with which code, you can't do your processing independently. 
You need to treat
the comments and whitespace in the context of the stream of "real" tokens.