[antlr-interest] Re: Lexer makes 2 valid tokens when there is only 1 invalid one

Alan Oursland alan at oursland.net
Thu Apr 17 11:41:22 PDT 2003


I had the same problem in scheme. My workaround would look something like this:

TOKEN	:	(NUMBER DELIMITER) 	=> NUMBER 		{ $setType(NUMBER); }
	|	(IDENTIFIER DELIMITER) 	=> DELIMITER 	{ $setType(IDENTIFIER); }
	;

Note that there is only one non-protected token type (TOKEN), but the type of
the returned token is changed during processing.

Alan


> -----Original Message-----
> From: micheal_jor [mailto:open.zone at virgin.net]
> Sent: Thursday, April 17, 2003 12:52 PM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] Re: Lexer makes 2 valid tokens when there is
> only 1 invalid one
>
>
> > Here is a valid SQL stmt: select 123 w from table;
> >
> > Here is an invalid SQL stmt: select 123w from table;
>
> I defer to Monty's post on this matter. You would have to either use
> a validating predicate as he advises or, you might want to check for
> whitespace in the hidden TokenStream in your parser.
>
> For "123 w" the Lexer would generate NUMBER WHITESPACE IDENTIFIER
> For "123w"  the Lexer would generate NUMBER            IDENTIFIER
>
> > In the first stmt, the "w" is a column alias for the
> constant "123";
> > in the 2nd stmt, "123w" is an invalid column name. My problem is
> that
> > I need to weed out bad stmts, like the 2nd one, but I cannot do
> that
> > if my lexer converts it to a valid stmt, like the first one. That's
> > why the parser cannot capture this problem - it doesn't think there
> > is one.
>
> It just needs to use more of the info available (e.g. the hidden
> TokenStream).
>
> Cheers,
>
> Micheal
>
>
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list