[antlr-interest] Adding imaginary tokens to the TokenStream in the PARSER? (Bad idea)

[Terence Parr]

On Jul 30, 2007, at 9:59 AM, Susan Jolly wrote:

> OK, I've now answered my own question in the negative. One does NOT  
> want to
> actually insert new tokens into an existing TokenStream because the
> TokenStream is a List, not a LinkedList, and each Token keeps track  
> of its
> index into this list.  If you were to insert a new Token somewhere  
> in the
> middle of the List, you'd have to reset all of the indices for Tokens
> following the insert.
>
> As its comments make clear, the "fiendishly clever" TokenRewriteStream
> doesn't actually insert Tokens.
>
> I need to insert an object, not just text.

Hi Susan,

this should be no problem. the rewrite engine actually accepts  
objects not strings. When it goes to string, then it evaluates  
everything.

> I had been using imaginary Tree
> nodes but saw some awkwardness.  I'll need to think about this some  
> more.

The lexer is free to insert as many imaginary nodes as it wants; for  
example, a Python lexer needs to insert a number of DEDENT tokens.  
The parser should not be inserting tokens into the stream that it is  
currently parsing. perhaps you could explain your problem a little  
bit more and I can give you a better answer.

Ter