[antlr-interest] Look-ahead changes when rules lifted up

[mzukowski at yci.com]

You can't go changing lexers from the parser.  The parser has already
recieved k more tokens beyond the token you are looking at in the parser.
You must switch lexers from the lexer.

Monty

-----Original Message-----
From: Paul J. Lucas [mailto:dude at darkfigure.org]
Sent: Monday, December 02, 2002 7:23 PM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] Look-ahead changes when rules lifted up


Given:

	elementConstructor
		: XML_TAG_START { pushContentLexer(); } QNAME attributeList
           (xmlEmptyElement | xmlNonEmptyElement) { popLexer(); } ;

	xmlEmptyElement
		: XML_EMPTY_ELEMENT_END ;

	xmlNonEmptyElement
		: XML_END_TAG (elementContent)*
		  XML_END_TAG_START QNAME (S)? XML_TAG_END ;

where:

	XML_TAG_START: '<' ;
	XML_EMPTY_ELEMENT_END: "/>" ;
	XML_END_TAG: '>' ;
	XML_END_TAG_START: "</" ;

with sufficient look-ahead, this doesn't work properly.  When parsing:

	<a></a>, <b/>
	      ^______________here

it needlessly looks-ahead at the ',' even though looking at the '>' alone is
sufficient.  The trouble is that the ',' is parsed in the wrong lexer (it
hasn't been popped yet).

If, however, I move the popLexer() action code out of the
elementConstructor rule and (redundantly) into the xmlEmptyElement and
xmlNonEmptyElement rules as in:

	xmlEmptyElement
		: XML_EMPTY_ELEMENT_END { popLexer(); } ;

	xmlNonEmptyElement
		: XML_END_TAG (elementContent)*
		  XML_END_TAG_START QNAME (S)? XML_TAG_END { popLexer(); } ;

then everything works fine: it no longer looks ahead past the '>' at
the ',', the lexer is popped, and the previous lexer returns the comma
correctly.

Why should moving the action code make a difference?  This is really
annoying and took dumb-luck and lot of trial-and-error to figure out
how to work around this.

- Paul


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/