[antlr-interest] Non-consumed token

[Graham Mer]

Hello list,

When I run a test sample against my AST-generating grammar in
ANTLRWorks 1.4.2 debugger, it generates the expected parse tree and
AST according to the diagrams generated in the debugger, but shows a
few of the tokens in the input panel in yellow. I can't find any
documentation on this, but from the preference panel, it appears that
this indicates a "non-consumed token." What does that mean, and does
it need to be fixed, or is it just additional information about a
correct parse? If it needs to be fixed, how do I fix or debug it?

I thought that the AST-generating grammar was working correctly, but I
hit problems when trying to parse the resulting AST that cause ANTLR
to run out of memory (infinite recursion?), so I suspect the yellow
tokens are trying to tell me that the AST-generating parse is in fact
not working correctly.

Finally, is there documentation for ANTLRWorks and its debugger
somewhere? I don't see any on the documentation page or the ANTLRWorks
page at antlr.org, and a google search for "antlr non-consumed token"
returns nothing useful.

Thanks.