[antlr-interest] Basic predicate question re: lexer

Dukie Banderjee dukie_banderjee at hotmail.com
Sun Jun 14 18:09:56 PDT 2009


Hi,
I'm working on a parser for a file format that can contain text and delimiters. One of the delimiters is a ':', and you can escape the delimiter by following it with a '?' such as ':?'.

I'd like to have the lexer consider the ':?' as part of the TEXT token, and ':' match the SEPARATOR token. So I tried this:
file        : contents+ EOF ;

contents    : TEXT
        | CSEP
        ;

CSEP:  ':';
TEXT: ('a'..'z'|'A'..'Z'|'0'..'9'|' '|'-'|','|'.'|'/'|COLON)+ ;
fragment COLON: ':?' ;

This didn't work, as the input "hello:" caused an error. I guess it was expecting to continue a TEXT token (next char would be '?'), and met with an EOF instead.

I imagine there's a way to do this, perhaps with predicates, but I'm not experienced enough to see the obvious solution. Can anyone help?

Thanks,

Rob

_________________________________________________________________
Create a cool, new character for your Windows Live™ Messenger. 
http://go.microsoft.com/?linkid=9656621
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090614/40345f16/attachment.html 


More information about the antlr-interest mailing list