[antlr-interest] collecting tokens without invoking parser rules...

[Alan Lehotsky]

> From: "Alan Lehotsky" <ALehotsky at ABINITIO.COM>
> Date: January 17, 2011 4:40:02 PM EST
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] collecting tokens without invoking parser rules...
> 
> 
> Using Antlr 3.2 with language=C as a target
> 
> For parsing Teradata's stored-procedure language (SPL), we have the issue 
> of context-sensitive token hiding.
> 
> I'm trying to use rules for SQL statements embedded in SPL that just 
> swallow the tokens, so we have rules like:
> 
> 
>        swallow_to_semi :   ~ (  SEMI  ) * ;
> 
>                update_stmt :  UPDATE swallow_to_semi;
> 
> .......
> This feels like something that should be relatively easy to do, but I 
> don't seem to be able to figure out exactly how to make it happen and I 
> haven't hit upon the right search terms to find an appropriate example in 
> the Antlr-interest archives or the Wiki.


Answering my own question...

It turns out that I was trying TOO hard to make this work.  This simple rule seems to completely resolve
the problems I was having:


	swallow_to_semi :

                     ( { LT(1)->getType(LT(1)) != SEMI }?.)+

i.e. collect any token that ISN'T a SEMI.  Not sure why the original rule didn't do exactly the same thing, but this
doesnt get confused by trying to do any parser-rule reductions on the token stream.