[antlr-interest] [antlr-dev] Including whitespace in AST

Jim Idle jimi at temporal-wave.com
Fri Jun 15 01:40:52 PDT 2012

If you want to see them as tokens in the tree, then after the tree is
produced by the parser, iterate through the tokens and set all the
whitespace tokens back to the default channel.

This is easier to deal with if your whitespace is:

WS: (‘ ‘ | ‘\t’)+;

And not one WS token per character. However you will still need to have WS?
all over your tree grammar to handle it.


*From:* Mike Lischke [mailto:mike at lischke-online.de]
*Sent:* Friday, June 15, 2012 2:30 PM
*To:* Jim Idle
*Subject:* Re: [antlr-dev] Including whitespace in AST


thanks for your answer.

Each tree node will tell you the starting and ending token (the tokens that
get what the node represents). Hidden tokens (rather than skipped tokens)
are still within the span, but the parser does not see them. So, if you
want the whitespace, you will need to access the tokens directly using code
at the point in the tree walk where you need to see it.

That's what I considered option B: during the tree walk use the token
stream to get hidden tokens. Not as elegant as it could be but an
acceptable solution I think.


More information about the antlr-interest mailing list