[antlr-interest] Should a Lexer barf on any input file?

mzukowski at yci.com mzukowski at yci.com
Thu May 23 06:53:43 PDT 2002

> I added a driver loop similar to that below to my under-development 
> Lexer to be able to test it as I go.
>    while ( !done ) {
>       Token t = lexer.nextToken();
>       PrintLine("Token: "+t);
>    }
> My query is this: is it possible for the Lexer to barf on a file that 
> the combined Parser/Lexer would accept?
> Basically, I'm trying to determine if it is possible for the Lexer 
> alone to reject a file that the Parser/Lexer combo accepts.

All a parser does is (indirectly through TokenBuffer) call lexer.nextToken()
over and over until EOF.  Unless you are doing something nasty like having
parser feedback to the lexer then I wouldn't expect to see a problem.  I
test my lexers like that all the time.  For instance:

		BASICLexer lexer = new BASICLexer ( dis );
		t = lexer.nextToken();
		while (t.getType() != Token.EOF_TYPE ) {
		    String type = BASICParser._tokenNames[t.getType()];
                System.out.println( type + ": '"+t.getText()+"'" + " " +
t.toString() );
		    t = lexer.nextToken();



Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 

More information about the antlr-interest mailing list