[antlr-interest] Adding imaginary tokens to the TokenStream in the PARSER? (Bad idea)

Susan Jolly easjolly at ix.netcom.com
Mon Jul 30 09:59:27 PDT 2007


OK, I've now answered my own question in the negative. One does NOT want to
actually insert new tokens into an existing TokenStream because the
TokenStream is a List, not a LinkedList, and each Token keeps track of its
index into this list.  If you were to insert a new Token somewhere in the
middle of the List, you'd have to reset all of the indices for Tokens
following the insert.

As its comments make clear, the "fiendishly clever" TokenRewriteStream
doesn't actually insert Tokens.

I need to insert an object, not just text. I had been using imaginary Tree
nodes but saw some awkwardness.  I'll need to think about this some more.

Susan




More information about the antlr-interest mailing list