[antlr-interest] Mismatched Token with String and Newline
Gavin Lambert
antlr at mirality.co.nz
Fri Oct 26 02:53:50 PDT 2007
At 22:25 26/10/2007, Robert.Klaus at innovations.de wrote:
>I have a problem with lexing Strings and I'm thinking I have
done
>something wrong. When entering the grammar below in ANTLRWorks
and
>use the interpreter with the input "\n" it works fine. When I
>enter a line break after the "\n" I get a
MismatchedTokenException.
First: ANTLRworks isn't really designed for debugging lexers, just
parsers. So when you're facing a lexer problem it's better to
write a test harness outside of ANTLRworks and step through it
with your debugger of choice.
>fragment CHARACTER
> :
>
> ('\\' ('n'|'t'|'r'|'\\'|'"'))
> | ('\\u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR)
> ;
"ESCAPE" would probably be a better name than "CHARACTER".
Also, given ANTLR's trouble dealing with common prefixes, you
should probably rewrite this to:
fragment CHARACTER
: '\\'
( 'n' | 't' | 'r' | '\\' | '"'
| 'u' HEX_CHAR HEX_CHAR HEX_CHAR HEX_CHAR
)
;
>NEWLINE : '\r'? '\n' ;
>WS : (' ' | '\t' | '\f')+ ;
You're not skipping or hiding the NEWLINE, so it's going to get
emitted. This means that your given input string ought to get
lexed to STRING NEWLINE EOF, and you're only accepting STRING
EOF. Hence the second of the two errors you reported.
More information about the antlr-interest
mailing list