[antlr-interest] Non-imaginary tokens in rewrite rules

[Jan Obdrzalek]

Hello,

I arrived at the following problem. Consider this rule:

foobar:
   'foo' -> ^(FB 'foo')
|  'bar' -> ^(FB 'foo')
;

Here FB is an imaginary token. It goes through antlr with no complaints, but
crashes when encountering 'bar' during parsing. looking at the generated
code, it looks as though ANTLR treats imaginary and non-imaginary tokens on
the right hand side differently: If a token is imaginary (FB), it is created
using adaptor.create. If not, it is read from the corresponding stream_xx
using stream_xx.next(). However, that stream is filled at the beginning of
the rule __only__ if there is such token in the input. [In other words if
the token number of  'foo' is 41 and of 'bar' is 42, what happens is that
the stream_42 is filled with 'bar' token, but the stream_41 (which is empty)
is read, resulting in an exception] This puzzles me quite a lot. Why not
just create 'foo' using adaptor.create, as it is the case for imaginary
tokens?

Any ideas, comments?

Regards,
Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20070822/83538747/attachment-0001.html