[antlr-interest] always create an ast node for an optional token

[Johannes Luber]

Felix Dorner schrieb:
> Hey,
> 
> I have some problems with ambiguities of declarations/expressions. I 
> therefore introduced an additional Token VAR='var' that's required prior 
> to a declaration.
> At some points however, I want to make that var token optional, because 
> it's clear from the context that there is a variable declaration, like 
> in a for statement:
> 
> Tokens{
> VAR='var'
> }
> 
> declaration: VAR ID -> ^(VAR ID)
> 
> forStatement : 'for' (VAR? ID 'in'...) -> ^(FOR ^(VAR ID ...))
> 
> You see, that I'd like to put a var node in the tree in all cases, even 
> if it's ommited in the text in unambiguous situations. It seems like 
> antlr doesn't like it this way, I get RewriteEmptyStreamExceptions in 
> this case.
> 
> I resolved it by placing literal 'var's into the rules and make VAR an 
> imaginary token, to decouple their meanings:
> 
> Tokens{
> VAR;
> }
> 
> declaration : 'var' ID -> ^(VAR ID)
> for: 'for' '(' 'var'? ID ...)   -> ... ^(VAR ID)
> 
> I guess that's an okay way to do it. Any other solutions?

I use an imaginary token named OPTIONAL like this:

forStatement : 'for' (VAR? ID 'in'...) -> ^(FOR ^(OPTIONAL VAR?) ^(ID ...))

So your normal code expects only the OPTIONAL part and if you don't need 
to look at VAR, you can ignore it.

Johannes