[antlr-interest] New user

[John Allen Green]

Antlr has a great feature called a "hidden token filter". You let the
whitespace go through as tokens from your lexer, and then the filter
decides what tokens go through to the parser for it to turn into nodes. At
the same time, those "hidden tokens" which were filtered out are added as
extra links hanging off the regular AST nodes. That way you can walk the
AST for regular semantic work, and you can also examine AST nodes for
attached hidden tokens. I use this heavily for my work.

In http://www.antlr.org/articles.html , look for "Preserving Whitespace
During Translation".

Regards,
John



--On 10/09/2002 2:34 PM -0700 Matt Benson wrote:

> I believe I begin to understand.  Accounting for
> comments in the Java grammar, for example, would
> necessitate liberally sprinkling the tokens throughout
> the parser rules, to represent the fact that they can
> occur anywhere.  The end of having the tokens in the
> tree might not be worth the effort and resulting
> unreadable parser rules, correct?  The other option
> you mentioned, adding comments as attributes of nodes,
> I do not understand.  Would this approach be
> applicable to my situation and if so how is it most
> easily accomplished?
> 
> TIA,
> Matt
> 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/