[antlr-interest] Understanding Lexer rules

[Darien Hager]

On 2/20/08, Johannes Luber <jaluber at gmx.de> wrote:
> Mark Volkmann schrieb:
>
> No. Rereading the text, I suppose one could be confused about that, as
>  it isn't as clear as it could be. Is adding "Longer matches are
>  preferred over shorter matches. If one has two tokens KEY='key'; and
>  KEYWORD='keyword';, then the input 'keyword' will match KEYWORD, even if
>  KEY comes first." enough?

I think the reason it is conterintuitive to me is that I assume a
different model for the lexer, and with it comes different assumptions
about behavior.

1, flawed model: The lexer loops through all tokens to find the first
one that matches what it sees when it looks ahead into the character
stream. When it finds one that matches, it moves on to the next point
in the stream and continues.

2, better model: The lexer keeps a set of "still possible" tokens, and
inches through the stream character by character, thinning the set as
appropriate, until only one possibility remains.

Is #2 closer to the actual behavior?
-- 
Darien Hager
Developer
Etelos, Inc.
darien at etelos.com

http://www.etelos.com
"Revolutionizing the way applications are developed, distributed and consumed."

This e-mail message, including attachments, may contain confidential
information for the sole use of the intended recipient(s). If you are
not the intended recipient, then this is notice that any use,
disclosure, dissemination, distribution or copying is strictly
prohibited. If you have received this message in error please contact
the sender by reply mail and destroy all copies of the original
message.

This e-mail message, including attachments, may contain confidential information for the sole use of the intended recipient(s). If you are not the intended recipient, then this is notice that any use, disclosure, dissemination, distribution or copying is strictly prohibited.  If you have received this message in error please contact the sender by reply mail and destroy all copies of the original message.