[antlr-interest] Understanding Lexer rules
Loring Craymer
lgcraymer at yahoo.com
Wed Feb 20 12:53:25 PST 2008
#2 is a pretty fair way of describing what a DFA does (how ANTLR's LL(*) works). Additionally, ANTLR automatically generates an mTokens rule which is an exhaustive list of TOKEN rules as alternatives :
mTokens : A | B | C .. Z ;
In cases of ambiguity, the ordering of alternatives matters; for the lexer, that means the order in which they appear in the mTokens rule. At present, I believe that that order reflects the order in which the rules are defined (IIRC, the rule names are kept in a OrderedHashSet).
--Loring
----- Original Message ----
> From: Darien Hager <darien.hager at etelos-inc.com>
> To: antlr-interest at antlr.org
> Sent: Wednesday, February 20, 2008 9:52:25 AM
> Subject: Re: [antlr-interest] Understanding Lexer rules
>
> On 2/20/08, Johannes Luber wrote:
> > Mark Volkmann schrieb:
> >
> > No. Rereading the text, I suppose one could be confused about that, as
> > it isn't as clear as it could be. Is adding "Longer matches are
> > preferred over shorter matches. If one has two tokens KEY='key'; and
> > KEYWORD='keyword';, then the input 'keyword' will match KEYWORD, even if
> > KEY comes first." enough?
>
> I think the reason it is conterintuitive to me is that I assume a
> different model for the lexer, and with it comes different assumptions
> about behavior.
>
> 1, flawed model: The lexer loops through all tokens to find the first
> one that matches what it sees when it looks ahead into the character
> stream. When it finds one that matches, it moves on to the next point
> in the stream and continues.
>
> 2, better model: The lexer keeps a set of "still possible" tokens, and
> inches through the stream character by character, thinning the set as
> appropriate, until only one possibility remains.
>
> Is #2 closer to the actual behavior?
> --
> Darien Hager
> Developer
> Etelos, Inc.
> darien at etelos.com
>
> http://www.etelos.com
> "Revolutionizing the way applications are developed, distributed and consumed."
>
> This e-mail message, including attachments, may contain confidential
> information for the sole use of the intended recipient(s). If you are
> not the intended recipient, then this is notice that any use,
> disclosure, dissemination, distribution or copying is strictly
> prohibited. If you have received this message in error please contact
> the sender by reply mail and destroy all copies of the original
> message.
>
> This e-mail message, including attachments, may contain confidential information
> for the sole use of the intended recipient(s). If you are not the intended
> recipient, then this is notice that any use, disclosure, dissemination,
> distribution or copying is strictly prohibited. If you have received this
> message in error please contact the sender by reply mail and destroy all copies
> of the original message.
>
>
____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
More information about the antlr-interest
mailing list