[antlr-interest] TreeNodeStream has broken index() method

Philippe Faes philippe at faes.net
Wed Dec 23 06:22:34 PST 2009


Dear all,

I've noticed that a CommonTreeNodeStream does not increment its index.
In other words, if I consume a token from the stream, then input.index()
remains 0.

This is bad for tree walkers, because the recover() method will think that
is encounters the same error twice, and skip over the token. See
BaseRecognizer.recover()

I think the problem is in LookaheadStream.remove().
int p is the index, but it gets reset.

  public T remove() {
        T o = get(0);
        p++;
        // have we hit end of buffer and not backtracking?
        if ( p == data.size() && markDepth==0 ) {
            // if so, it's an opportunity to start filling at index 0 again
            clear(); // size goes to 0, but retains memory
        }
        return o;
    }

Things work for me if I skip the clear(); instruction, but perhaps there is
a better solution that can be patched in the next official release.

kind regards

Philippe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091223/4a7cb44b/attachment.html 


More information about the antlr-interest mailing list