[antlr-interest] Re: Representing state in lexer

cgodfrey86 cgodfrey at epnet.com
Wed May 14 08:50:32 PDT 2003


I can do a definition like the following, where a PATTERN token is 
returned as either quoted or not, but I still have issues  when 
scanning gt. If gt appears after a quote (not necessarily immediately 
after the quote), I want to return gt as type PATTERN.

PATTERN : QUOTE (ALLOWCHARS)+ QUOTE ;

TERM 	:
	("gt")=> "gt"
	{$setType(GT_OP);}
 	| (">")=> ">"
	{$setType(GT_OP);}
	|("ge")=> "ge" 
	{$setType(GE_OP);}
	|(">=")=> ">="
	{$setType(GE_OP);}
	|("lt")=>"lt" 
	{$setType(LT_OP);}
	|("<")=>"<"
	{$setType(LT_OP);}
	|("le")=>"le"
	{$setType(LE_OP);}
	|("<=")=>"<="
	{$setType(LE_OP);}
	|("eq")=>"eq" 
	{$setType(EQ_OP);}
	|("=")=>"="
	{$setType(EQ_OP);}
	|("-")=>"-"
	{$setType(DASH);}
	| ("or") => "or"
	{$setType(OR_OP);}
	| ("and") => "and"
	{$setType(AND_OP);}
	| ("not") => "not"
	{$setType(NOT_OP);}
	|(('a'..'z')('a'..'z') WS) => ('a'..'z')('a'..'z')
      	{
		$setType(S_TAG);
        }
        | ('w'INT)=>'w'INT
	{$setType(W_OP);}
        | ('n'INT)=>'n'INT
	{$setType(N_OP);}
	|
	(ALLOWCHARS)+
	{$setType(PATTERN);}
     	;




--- In antlr-interest at yahoogroups.com, "Anthony W Youngman" 
<Anthony.Youngman at E...> wrote:
> What is it non-deterministic with? In other words, what other 
construct
> recognised by the lexer could a PATTERN be confused with?
> 
> You will have to define an unquoted pattern as a subset of that 
other
> construct, and then use $settype to change the type to PATTERN if 
that's
> what you've got.
> 
> Cheers,
> Wol
> 
> -----Original Message-----
> From: cgodfrey86 [mailto:cgodfrey at e...] 
> Sent: 14 May 2003 15:56
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] Re: Representing state in lexer
> 
> 
> Thank you for the suggestion.
> 
> The main issue that I am running into with this is that a PATTERN 
> token can appear either inside or outside of quotes - so I won't 
> recognize a PATTERN outside of quotes. When I try and define it 
> without quotes, I have lexical nondeterminism
> 
> PATTERN : QUOTE (ALLOWCHARS)+ QUOTE ;
> 
> 
> 
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/ 
> 
> 
> 
> This transmission is intended for the named recipient only. It may 
contain private and confidential information. If this has come to you 
in error you must not act on anything disclosed in it, nor must you 
copy it, modify it, disseminate it in any way, or show it to anyone. 
Please e-mail the sender to inform us of the transmission error or 
telephone ECA International immediately and delete the e-mail from 
your information system.
> 
> Telephone numbers for ECA International offices are: Sydney +61 (0)
2 9911 7799, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 
and New York +1 212 582 2333.


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list