[antlr-interest] Should a Lexer barf on any input file?

[mzukowski at yci.com]

> I added a driver loop similar to that below to my under-development 
> Lexer to be able to test it as I go.
> 
>    while ( !done ) {
>       Token t = lexer.nextToken();
>       PrintLine("Token: "+t);
>    }
> 
> My query is this: is it possible for the Lexer to barf on a file that 
> the combined Parser/Lexer would accept?
> 
> Basically, I'm trying to determine if it is possible for the Lexer 
> alone to reject a file that the Parser/Lexer combo accepts.

All a parser does is (indirectly through TokenBuffer) call lexer.nextToken()
over and over until EOF.  Unless you are doing something nasty like having
parser feedback to the lexer then I wouldn't expect to see a problem.  I
test my lexers like that all the time.  For instance:

		BASICLexer lexer = new BASICLexer ( dis );
	      lexer.setTokenObjectClass("ArevToken");
		t = lexer.nextToken();
		while (t.getType() != Token.EOF_TYPE ) {
		    String type = BASICParser._tokenNames[t.getType()];
                System.out.println( type + ": '"+t.getText()+"'" + " " +
t.toString() );
		    t = lexer.nextToken();
		}

Monty

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/