[antlr-interest] Puzzle, and solution.

[Clifford Heath]

Something that's puzzled me when using ANTLRWorks to debug grammars
is that sometimes when running a rule in the interpreter, the parse
tree ends in "No viable alternative" when fed input that should match,
just because of the existence of rules that aren't activated.

As an example, I have an expression grammar that works just fine by
itself. When I include it in a larger grammar, which uses expressions
only in certain contexts, the same expression that parsed fine now
won't parse... until I add a token that *could* follow an expression.

The expression rule then matches, and of course doesn't consume the
extra token. It seems that the expression rule only matches the
current input if, when looking ahead, the next token (or EOF) is a
valid next token. The added token may be any token that is a valid
successor to an expression in any rule of the larger grammar, even
if that rule can't be called!

When you enter an expression to test the behaviour of an "expression"
rule in the interpreter, you need to either add a plausible next
token to the end, or add a rule like this:

junk: expression EOF;

and then any valid expression will parse without adding a token,
because there's an implicit EOF at the end of what you entered...
even though the rule "junk" never gets called anywhere in the wider
grammar.

Just a trap I thought I might be able to help someone else avoid.
I'm sure it's expected behaviour, but it's a bit non-intuitive at
first.

Clifford Heath.