[antlr-interest] problem matching any 'other' text

[Nigel Sheridan-Smith]

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Prashant Deva
> Sent: Saturday, 26 March 2005 7:45 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] problem matching any 'other' text
> 
> Hi,
> I have a little problem. I have this grammer (the antlr 2.7.xx
> grammar, that is) and i am trying to add a rule to match 'any other
> text'. basically text that is not matched by any rule.
> But there seems to be a conflict with it and Whitespace rule.
> Here are the rules-
> 
> TEXT_ERROR : (options{ greedy = false;}: .)+ ; //match any text not
> matched by any other rule
> 
> WS_LOOP
>        :       (       // grab as much WS as you can
>                        options {greedy=true;} :
>                WS |   COMMENT
>                )*
>        ;
> 
> Thing is, when i type a whitespace, it is detected as a 'TEXT_ERROR'
> token instead of a ws token. how do i correct the rules so that the ws
> is handled correctly.
> 


It looks like these are rules in the Lexer... In that case, you need to be
very careful about how tokens are generated. You can use protected Lexer
rules to avoid some conflicts, but not in all cases.

I assume your whitespace/newline rules are something like this:

WS: NL | ' ' | '\t';
protected NL: '\n' | '\r' { newline(); };  /* Simplified example */

If you want a rule that matches anything else, you can do this:

ANYTHINGELSE: ( ~(' ' | '\t' | '\n' | '\r') )+ ;

But this will conflict with any other lexer rules you write, because you
need to have at least the first few characters distinguishable (or use a
syntactic predicate). Otherwise, this rule could potentially grab any
character input up until the next white-space. Which I'm pretty sure you
don't want!

So what exactly are you trying to achieve?

Nigel

--
Nigel Sheridan-Smith
PhD research student

Faculty of Engineering
University of Technology, Sydney
Phone: 02 9514 7946
Fax: 02 9514 2435