[antlr-interest] chat message parser rule?

[Gavin Lambert]

At 09:16 27/01/2009, ian eyberg wrote:
 >Running this through antlrworks I am hitting my 'chatrule'
 >every time but it will start chopping up the
 >player's message into other tokens that I have defined.
 >
 >My question: why wouldn't the ALPHA token always be
 >chosen over the other tokens?

Lexing is done independently of parsing (and before parsing 
begins).

Essentially, when the lexer is "between" tokens it will look at 
the input and decide which of the lexer rules best fits the 
input.  Then it will consume input using that rule until the rule 
is no longer satisfied, and then it will exit out and look for 
another lexer rule to satisfy.

When you have overlapping definitions (such as ALPHA and LAMB), 
this can lead to differences in behaviour depending on the 
ordering of the input.  (eg. "rack" => LAMB, "racket" => LAMB 
ALPHA, "track" => ALPHA)