[antlr-interest] Token stream filter

[Ric Klaren]

Hi,

On Wed, Jun 02, 2004 at 11:26:53AM +0100, Anthony Youngman wrote:
> Okay. Doing my own homework :-) Would this work (needing cleaning up, I
> don't know what to throw etc ...)?
> 
> filter=DEREMER
> 
> protected DEREMER [returns token] : (
> 	(newline|semi)
> 	(id:IDENT {if text != "REM" throw tokenmatchexception}|"*"|"!")
> 	("=" {throw exception} | "(" {throw exception})?
> 	({greedy=false} .)
> 	newline)
> 	returns newline;

filter=<rule> only works in lexers AFAIK.

> I want to filter the token stream coming out of the lexer (and I can't
> see how to do it using the lexer's filter - can I do the same thing in
> the parser?). Anyways, I want to match the following sequence
> 
> (newline | semi) ("rem" | "*" | "!" ) ~( "=" | "(" ) ({greedy=false} .)
> newline)

So to put it in parser Lingo:

(NEWLINE | SEMI) ("rem" | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )

There's only a set of closures missing by the looks of it ?

Looks like Monty's filter example might be usefull...

http://www.codetransform.com/filterexample.html

> eating everything EXCEPT the newline, which needs to be passed through
> to the parser. Note also, that while "*" and "!" are tokens, "rem" is an
> ident. 

So it's:

(NEWLINE | SEMI) (rem:ID | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )

> Of course, if I need "rem" to be a token, presumably if this rule
> fails I can convert it back to an ident?

Should be possible to modify the ident to a REM token I guess.

I guess I'd handwrite a small parser for the above might be simpler than
coaxing antlr in doing it. Although I'm not sure you migth get away with
some semantic predicate fun.

> Or do I need to write my own token stream to do this (if so, where do I
> look for an example?)

See the link to Monty's site.

Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
  "I think we better split up."
  "Good idea. We can do more damage that way."
  --- Ghostbusters



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/