[antlr-interest] Help controlling parser decisions

[Randall R Schulz]

On Tuesday 24 July 2007 10:56, Ted Villalba wrote:
> Hi,
>
> I have a grammar that contains tokens that are sometimes operators,
> sometimes not, depending on the context. The set of operators
> overlaps with the set of all words that can be acceptable tokens.
> Trouble is, depending on the order of my lexer rules,  the parser
> will recognize all such tokens (AND , OR ,NEAR) as operators, or will
> recognize none of them as operators.

This qualifies as a FAQ, I think.

Lexical analysis is not driven by parser rules. It is an entirely 
separate, earlier phase of input processing. There is no way to produce 
different token types from any given input depending on context.

Another way of thinking about it is that while the parser is driven by 
top-down invocation of rules (along with all the look-ahead and 
backtracking that may be involved), there is no such orchestration of 
the lexical analyzer. It just chunks up the characters (or breaks down 
the total sequence) into a linear chain of sub-sequences of the total 
input.


> ...
>
> Ted


Randall Schulz