[antlr-interest] MismatchedTokenException encountered when moving from literal to lexical token

[Randall Barnhart]

Hey ANTLR Folks,

I am trying to generate a grammar for a Prolog like language, and I am 
running into a problem when I try to provide more than 1 token alternative 
for a comma separated list.  I've provided the full grammar below for 
reference, but here is the rule that has the problem:

When I changed -
body
	: subStatement (','^ subStatement)*
	;

to the following -
body
	: subStatement (CONJ^ subStatement)*
	;

where -
CONJ : 'AND' | 'and' | ',' ;

I now get a MismatchedTokenException when I use a ',' whereas before it was 
recognizing it just fine.  Specifically:

recoverFromMismatchedToken
BR.recoverFromMismatchedToken
line 2:24 mismatched input ',' expecting ENDSTMT

Using the following input to match on the "clause" rule:
pred1(cp1, np1, dp1) :-
	cp1 := "predicateCode1",
	np1 := 2 * 3 / 10,
	dp1 := 12/31/2007 12:00 AM
	;

Also, I don't get the error if I use 'AND' or 'and'.  Ultimately I think it 
is a conflict with some of the other rules that use ',' to signifiy a list 
of expressions, because when I change those to different symbols then I 
don't get this error.

I think I should maybe use a syntactic or semantic predicate, but I'm not 
real sure on how to go about doing this.  Does anyone have any specific 
solutions, ideas, or advice that might lead me in the right direction?

Thanks,
Randall Barnhart



grammar Test;

options {
	output=AST;
}

tokens {
	RULE;
	HEAD;
	BODY;
	STMTCONJ;
	STMTDISJ;
	FACT;
	VARLIST;
	LITLIST;
	NEGATE;
	EXPR;
	VARDECL;
	VARASSIGN;
	PREDCALL;
	EXPRLIST;
}

clause
	: rule
	| fact
	;

rule
	: head ':-' body ENDSTMT
	  -> ^(RULE head ^(BODY body))
	;

head
	: predicateName '(' varList ')'
	  -> ^(HEAD predicateName varList)
	;

body
	: subStatement (CONJ^ subStatement)*
	;

fact
	: predicateName '(' literalList ')' ENDSTMT
	  -> ^(FACT predicateName literalList)
	;

predicateName
	: ID
	;

varDecl
	: type ID
	  -> ^(VARDECL type ID)
	;

varAssign
	: ID ':=' expr
	  -> ^(VARASSIGN ID expr)
	;

varList
	: ID (',' ID)* -> ^(VARLIST ID+)
	;

type
	: 'Code'
	| 'Numeric'
	| 'Date'
	;

predicateInvocation
	: predicateName '(' exprList ')'
	  -> ^(PREDCALL predicateName exprList)
	;

subStatement
	: expr -> ^(EXPR expr)
	| varDecl
	| varAssign
	| predicateInvocation
	;

exprList
	: expr (',' expr)*
	  -> ^(EXPRLIST expr+)
	;

expr
	: boolAndExpr (OR^ boolAndExpr)*
	;

boolAndExpr
	: equalityExpr (AND^ equalityExpr)*
	;

equalityExpr
	: relationalExpr ((EQUALS|NOTEQUALS)^ relationalExpr)*
	;

relationalExpr
	: addExpr ((LT|LTEQ|GT|GTEQ)^ addExpr)*
	;

addExpr
	: multExpr ((PLUS|MINUS)^ multExpr)*
	;

multExpr
	: powExpr ((MULT|DIV|MOD)^ powExpr)*
	;

powExpr
	: unaryExpr (POW^ unaryExpr)*
	;

unaryExpr
	: primaryExpr
	| NOT^ primaryExpr
	| MINUS primaryExpr -> ^(NEGATE primaryExpr)
	;

primaryExpr
	: '('! expr ')'!
	| literal
	;

literalList
	: literal (',' literal)*
	  -> ^(LITLIST literal+)
	;

literal
	: BOOLEAN
	| CODE
	| NUMERIC
	| DATE
	| ID
	;

OR	: '||';
AND	: '&&';
EQUALS	: '==' ;
NOTEQUALS
	: '!=' ;
LT	: '<' ;
LTEQ	: '<=' ;
GT	: '>' ;
GTEQ	: '>=' ;
PLUS	: '+' ;
MINUS	: '-' ;
MULT	: '*' ;
DIV	: '/' ;
MOD	: '%' ;
POW	: '^' ;
NOT	: '!' | 'NOT' | 'not';

BOOLEAN : 'true' | 'false' ;
CODE	: '\"' ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')* '\"' ;
NUMERIC	: ('0'..'9')+ ('.' ('0'..'9')+)? ;
DATE	: MO1 MO2 '/' D1 D2 '/' YEAR ' ' H1 H2 ':' MI1 MI2 ' ' ('AM'|'PM') ;

fragment
MO1	: ('0'..'1') ;
fragment
MO2	: ('0'..'9') ;
fragment
D1	: ('0'..'3') ;
fragment
D2	: ('0'..'9') ;
fragment
YEAR	: ('0'..'9') ('0'..'9') ('0'..'9') ('0'..'9') ;
fragment
H1	: ('0'..'1') ;
fragment
H2	: ('0'..'9') ;
fragment
MI1	: ('0'..'5') ;
fragment
MI2	: ('0'..'9') ;

ENDSTMT	: ';' ;
CONJ	: 'AND' | 'and' | ',';
DISJ	: 'OR' | 'or';


ID	: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')* ;

WS	: (' '|'\r'|'\t'|'\u000C'|'\n') {$channel=HIDDEN;} ;

_________________________________________________________________
Kick back and relax with hot games and cool activities at the Messenger 
Café. http://www.cafemessenger.com?ocid=TXT_TAGHM_SeptHMtagline1