[antlr-interest] BUG: Constant (literal) token is incorrectly assumed to be present in AST rewrite

[Johannes Luber]

Foust schrieb:
> Austin  wrote:
>> At the same time, this is pretty clearly a bug: antlr is confusing two
>> tokens that look the same, but one of which doesn't exist.
> 
> The way to do this is to create a synthetic token (called "imaginary", for
> some reason), by adding a section to your grammar just after the "options":
> 
> tokens {
> 	T_TERCIARY = '?';
> }
> 
> Then use that token in your tree output:
> 
> 	-> ^(TERCIARY logical_or expression? conditional_expr?) ;
> 
> Brent

An imaginary token is defined like

tokens {
IMAGINARY;
}

Note that no assignment to a particular input string happens - that is 
the reason why such tokens are imaginary tokens.

> 
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Austin Hastings
>> Sent: Wednesday, August 13, 2008 6:07 PM
>> To: antlr-interest
>> Subject: [antlr-interest] BUG: Constant (literal) token is incorrectly
>> assumed to be present in AST rewrite
>>
>> Howdy,
>>
>> I have a C-like grammar with this entry:
>>
>> conditional_expr
>>     : logical_or ( '?' expression ':' conditional_expr )?
>>        -> ^( '?' logical_or expression? conditional_expr?)
>>     ;
>>
>> I am using antlr v3.1. In this case, the generated code assumes that
>> the
>> rewrite-rule's '?' token is the same as the possibly-nonexistent '?' in
>> the matching rule, and tries to dereference the "stream" that would
>> contain that token, if it were present.
>>
>> In the case where no conditional expression is present, the rewrite
>> should collapse into ^( '?' logical_or )
>>
>> I suspect that there's a better way to handle this particular rewrite
>> scenario. I'd like to hear it.
>>
>> At the same time, this is pretty clearly a bug: antlr is confusing two
>> tokens that look the same, but one of which doesn't exist.
>>
>> =Austin
>