[antlr-interest] Tool for testing lexers/parsers/tree parsers

[mzukowski at bco.com]

Jonathan Bachrach posted a question here on 8/31/01 entitled "single
identifier as java expression".  The problem was that calling the java
grammar's expr rule would not work if the input was just one identifier.  I
believe that I remember him saying the solution was to make a separate rule
to parse an expr followed by EOF.  The Java grammar generates this code in
unaryExpressionNotPlusMinus:
...
if (synpred handled here)
{...
}
else if ((_tokenSet_19.member(LA(1))) && (_tokenSet_20.member(LA(2)))) {
	postfixExpression();
	if (inputState.guessing==0) {
		astFactory.addASTChild(currentAST, returnAST);
	}
} 
else {
	throw new NoViableAltException(LT(1), getFilename()); 
}

_tokenSet_20 doesn't have EOF in it, and you will get a NoViableAltException
thrown.  I found this out by regenerating with options
{codeGenBitsetTestThreshold = 999999;} which forces no token sets to be
generated unless they have more than 999999 elements.

So the answer is no, you may not always be able to test individual rules if
they are not expecting an EOF to follow.

Monty

> -----Original Message-----
> From: bob mcwhirter [mailto:bob at werken.com]
> Sent: Thursday, January 10, 2002 1:00 PM
> To: antlr-interest at yahoogroups.com
> Subject: RE: [antlr-interest] Tool for testing lexers/parsers/tree
> parsers
> 
> 
> On Thu, 10 Jan 2002 mzukowski at bco.com wrote:
> 
> > There is a potential problem in testing specific rules in a 
> parser in that
> > the EOF token may not be part of the token sets used for 
> lookahead in
> > decisions.  This may also be a problem for "protected" 
> lexer rules.  If you
> > find such a problem I would recommend simply writing a new 
> rule with the EOF
> > included.  For example:
> > 
> > exprTestRule: expr EOF;
> > 
> > In fact I would recommend doing this for any rules you are 
> going to test to
> > make sure that all of your test input is used.  Otherwise 
> you might only
> > parse as much as the rule wants to see at the same time you would be
> > assuming that your entire input has been parsed.
> 
> Would it be possible, though, to test normal rules, without 
> the EOF, and
> then check that nextToken() produces EOF?
> 
> 	ie:
> 
> 		parser.exprRule();
> 		assertEquals( TokenTypes.EOF, lexer.nextToken() );
> 
> This is at least what I've done to test hand-written recursive descent
> parsers and lexers (for SAXPath).
> 
> 	-bob
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
http://docs.yahoo.com/info/terms/ 



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/