[antlr-interest] Mismatched Token with String and Newline

[Robert.Klaus at innovations.de]

Ahh,

Thanks! Skipping tokens helps alot if you don't forget it ...

Bye

Robert

> -----Original Message-----
> From: Gavin Lambert [mailto:antlr at mirality.co.nz] 
> Sent: Friday, October 26, 2007 11:54 AM
> To: Klaus Robert; antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Mismatched Token with String and Newline
> 
> At 22:25 26/10/2007, Robert.Klaus at innovations.de wrote:
>  >I have a problem with lexing Strings and I'm thinking I 
> have done  >something wrong. When entering the grammar below 
> in ANTLRWorks and  >use the interpreter with the input "\n" 
> it works fine. When I  >enter a line break after the "\n" I 
> get a MismatchedTokenException.
> 
> First: ANTLRworks isn't really designed for debugging lexers, 
> just parsers.  So when you're facing a lexer problem it's 
> better to write a test harness outside of ANTLRworks and step 
> through it with your debugger of choice.
> 
>  >fragment CHARACTER
>  >	:
>  >		
>  >		('\\' ('n'|'t'|'r'|'\\'|'"'))		
>  >		| ('\\u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR)
>  >	;
> 
> "ESCAPE" would probably be a better name than "CHARACTER".
> 
> Also, given ANTLR's trouble dealing with common prefixes, you 
> should probably rewrite this to:
> 
> fragment CHARACTER
>    :  '\\'
>       (  'n' | 't' | 'r' | '\\' | '"'
>       |  'u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR
>       )
>    ;
> 
>  >NEWLINE :	'\r'? '\n' ;
>  >WS 	:	(' ' | '\t' | '\f')+ ;
> 
> You're not skipping or hiding the NEWLINE, so it's going to 
> get emitted.  This means that your given input string ought 
> to get lexed to STRING NEWLINE EOF, and you're only accepting 
> STRING EOF.  Hence the second of the two errors you reported.
> 
>