[antlr-interest] Newline not recognized

[Gavin Lambert]

At 18:36 14/01/2008, Cory Isaacson wrote:

>I have a rule like this:
>
>r : MYSTARTTOKEN '\n' NEXTTOKEN ;
>
>When I parse it doesn't see the '\n' char.
>
>If I put the '\n' into to Lexer rule like this:
>
>NEXTTOKEN : '\n' 'xyz' ;
>
>It works fine, but this won't meet my needs as I need to get the 
>text of NEXTTOKEN (without the newline).
>
>Any idea what is wrong in what I'm doing?

Most likely, you've already got a different token representing a 
'\n'.  Whenever you use a string literal in a parser rule, it 
creates a hidden new token that matches that literal.  If your 
lexer is choosing to generate a different token over that one, 
though (eg. you have a WS rule), then it'll never match.

If whitespace (or just newlines) are significant in your grammar, 
then make sure you don't hide them at the lexer level, and that 
you're not mixing lexer and parser rules.  (I generally find it 
easier to remember the separation if you avoid using string 
literals in parser rules altogether.)