[antlr-interest] Handling lexical nondeterminism in Tokens

Mark R. Diggory mdiggory at latte.harvard.edu
Sun Feb 5 10:42:09 PST 2006


I'm still working on building a Parser for our query syntax. I've 
encountered an issue with nondeterminism. I've included my grammar file:

My question is how can I assure that the boolean predicate AND not the 
quoted string literal "you AND I" do not collide? I'd be very thankful 
to anyone with comments about obvious problems with my grammar file.

thanks,
Mark

> class SearchQueryParser extends Parser;
>     options
>     {
>           k=3;
>         exportVocab=SearchQuery;
>         buildAST = true;   // uses CommonAST by default
>         
>     }
>
>
> expr
>     :    
>         mexpr ((AND|OR|NOT) mexpr)*
>     ;
>
> mexpr
>     :    
>         LITERAL^ | IDENTIFIER^ ((EQUALS|NOT_EQUALS|LT|LTE|GT|GTE) 
> LITERAL^)+
>     ;
>
>
> atom
>       :    
>           IDENTIFIER | LEFT_PAREN! expr RIGHT_PAREN!
>     ;
>
> class SearchQueryLexer extends Lexer;
>     options
>     {
>         charVocabulary='\3'..'\377';
>     }
>
> WS
>     :
>         ('\n' | ' ' | '\t' | '\r')+
>         {
>             $setType(Token.SKIP);
>         }
>     ;
>
>
> protected
> SINGLE_QUOTE_STRING
>     :
>         '\''! (~('\''))* '\''!
>     ;
>
> protected
> DOUBLE_QUOTE_STRING
>     :
>         '"'! (~('"'))* '"'!
>     ;
>
> LITERAL
>     :
>         SINGLE_QUOTE_STRING | DOUBLE_QUOTE_STRING
>     ;
>
> IDENTIFIER
>
>     options
>     {
>         testLiterals=true;
>     }
>
>     :     
>         ('\241'..'\377'|'a'..'z'|'A'..'Z'|'_') 
> ('\241'..'\377'|'a'..'z'|'A'..'Z'|'-'|'_'|'0'..'9'|'.')*    
>     ;
>
> LEFT_PAREN
>     :    '('        ;
>
> RIGHT_PAREN    
>     :    ')'        ;
>
> NOT
>     :    ("NOT"|"not")    ;
>
> AND
>     :    ("AND"|"and")    ;
>
> OR
>     :    ("OR"|"or")        ;
>
> EQUALS
>     :    '='        ;
>
> NOT_EQUALS
>     :    "<>"    ;
>
> LT
>     :    '<'        ;
>
> LTE
>     :    "<="    ;
>
> GT
>     :    '>'        ;
>
> GTE
>     :    ">="    ;




More information about the antlr-interest mailing list