[antlr-interest] Understanding Lexer rules

Loring Craymer lgcraymer at yahoo.com
Wed Feb 20 12:53:25 PST 2008


#2 is a pretty fair way of describing what a DFA does (how ANTLR's LL(*) works).  Additionally, ANTLR automatically generates an mTokens rule which is an exhaustive list of TOKEN rules as alternatives :  
mTokens : A | B | C .. Z ;

In cases of ambiguity, the ordering of alternatives matters; for the lexer, that means the order in which they appear in the mTokens rule.  At present, I believe that that order reflects the order in which the rules are defined (IIRC, the rule names are kept in a OrderedHashSet).

--Loring

----- Original Message ----
> From: Darien Hager <darien.hager at etelos-inc.com>
> To: antlr-interest at antlr.org
> Sent: Wednesday, February 20, 2008 9:52:25 AM
> Subject: Re: [antlr-interest] Understanding Lexer rules
> 
> On 2/20/08, Johannes Luber  wrote:
> > Mark Volkmann schrieb:
> >
> > No. Rereading the text, I suppose one could be confused about that, as
> >  it isn't as clear as it could be. Is adding "Longer matches are
> >  preferred over shorter matches. If one has two tokens KEY='key'; and
> >  KEYWORD='keyword';, then the input 'keyword' will match KEYWORD, even if
> >  KEY comes first." enough?
> 
> I think the reason it is conterintuitive to me is that I assume a
> different model for the lexer, and with it comes different assumptions
> about behavior.
> 
> 1, flawed model: The lexer loops through all tokens to find the first
> one that matches what it sees when it looks ahead into the character
> stream. When it finds one that matches, it moves on to the next point
> in the stream and continues.
> 
> 2, better model: The lexer keeps a set of "still possible" tokens, and
> inches through the stream character by character, thinning the set as
> appropriate, until only one possibility remains.
> 
> Is #2 closer to the actual behavior?
> -- 
> Darien Hager
> Developer
> Etelos, Inc.
> darien at etelos.com
> 
> http://www.etelos.com
> "Revolutionizing the way applications are developed, distributed and consumed."
> 
> This e-mail message, including attachments, may contain confidential
> information for the sole use of the intended recipient(s). If you are
> not the intended recipient, then this is notice that any use,
> disclosure, dissemination, distribution or copying is strictly
> prohibited. If you have received this message in error please contact
> the sender by reply mail and destroy all copies of the original
> message.
> 
> This e-mail message, including attachments, may contain confidential information 
> for the sole use of the intended recipient(s). If you are not the intended 
> recipient, then this is notice that any use, disclosure, dissemination, 
> distribution or copying is strictly prohibited.  If you have received this 
> message in error please contact the sender by reply mail and destroy all copies 
> of the original message.
> 
> 




      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping


More information about the antlr-interest mailing list