[antlr-interest] Understanding Lexer rules

Johannes Luber jaluber at gmx.de
Wed Feb 20 07:05:21 PST 2008


Mark Volkmann schrieb:
> On Feb 20, 2008 8:50 AM, Johannes Luber <jaluber at gmx.de> wrote:
>> No. Rereading the text, I suppose one could be confused about that, as
>> it isn't as clear as it could be. Is adding "Longer matches are
>> preferred over shorter matches. If one has two tokens KEY='key'; and
>> KEYWORD='keyword';, then the input 'keyword' will match KEYWORD, even if
>> KEY comes first." enough?
> 
> Does the same rule about matching the maximum length apply when the
> lexer rules use the cardinality operators (?, * and +)? I recommend
> adding something about that and not just showing examples with fixed
> literal values.
 >
> In case of a tie in the number of characters matched by multiple lexer
> rules, does the first one win?

I honestly never thought about that, but I believe that the first one 
has to win, or the "keyword vs. identifier" disambiguation wouldn't 
work. Can you whip up some tests to confirm this with a bigger test set? 
I am short of time. :(

Johannes


More information about the antlr-interest mailing list