[antlr-interest] Mismatched Token with String and Newline

[Gavin Lambert]

At 22:25 26/10/2007, Robert.Klaus at innovations.de wrote:
 >I have a problem with lexing Strings and I'm thinking I have 
done
 >something wrong. When entering the grammar below in ANTLRWorks 
and
 >use the interpreter with the input "\n" it works fine. When I
 >enter a line break after the "\n" I get a 
MismatchedTokenException.

First: ANTLRworks isn't really designed for debugging lexers, just 
parsers.  So when you're facing a lexer problem it's better to 
write a test harness outside of ANTLRworks and step through it 
with your debugger of choice.

 >fragment CHARACTER
 >	:
 >		
 >		('\\' ('n'|'t'|'r'|'\\'|'"'))		
 >		| ('\\u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR)
 >	;

"ESCAPE" would probably be a better name than "CHARACTER".

Also, given ANTLR's trouble dealing with common prefixes, you 
should probably rewrite this to:

fragment CHARACTER
   :  '\\'
      (  'n' | 't' | 'r' | '\\' | '"'
      |  'u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR
      )
   ;

 >NEWLINE :	'\r'? '\n' ;
 >WS 	:	(' ' | '\t' | '\f')+ ;

You're not skipping or hiding the NEWLINE, so it's going to get 
emitted.  This means that your given input string ought to get 
lexed to STRING NEWLINE EOF, and you're only accepting STRING 
EOF.  Hence the second of the two errors you reported.