[antlr-interest] Token stream filter

Anthony Youngman Anthony.Youngman at ECA-International.com
Wed Jun 2 07:29:52 PDT 2004


Thanks. That's pretty much what I was looking for. Now I merely have to
understand it :-)

And it's even the case that I'm writing a grammar for the same language
:-) although a different dialect. I don't know if you remember Rob's
example of a program with all the varying usages of REM? A *nightmare*
(and the one I'm trying to deal with :-) but using a different technique
to him.

Note for Ter - if you haven't saved that post, go look it up. It'd be
nice if Antlr had some way of trying to cope - it's a perfect example of
something that a lexer/parser could handle pretty easily, but a separate
lexer and parser would have nightmares over...

Cheers,
Wol 

-----Original Message-----
From: Ric Klaren [mailto:klaren at cs.utwente.nl] 
Sent: 02 June 2004 15:17
To: antlr-interest at yahoogroups.com
Subject: Re: [antlr-interest] Token stream filter

Hi,

On Wed, Jun 02, 2004 at 11:26:53AM +0100, Anthony Youngman wrote:
> Okay. Doing my own homework :-) Would this work (needing cleaning up,
I
> don't know what to throw etc ...)?
> 
> filter=DEREMER
> 
> protected DEREMER [returns token] : (
> 	(newline|semi)
> 	(id:IDENT {if text != "REM" throw tokenmatchexception}|"*"|"!")
> 	("=" {throw exception} | "(" {throw exception})?
> 	({greedy=false} .)
> 	newline)
> 	returns newline;

filter=<rule> only works in lexers AFAIK.

> I want to filter the token stream coming out of the lexer (and I can't
> see how to do it using the lexer's filter - can I do the same thing in
> the parser?). Anyways, I want to match the following sequence
> 
> (newline | semi) ("rem" | "*" | "!" ) ~( "=" | "(" ) ({greedy=false}
.)
> newline)

So to put it in parser Lingo:

(NEWLINE | SEMI) ("rem" | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )

There's only a set of closures missing by the looks of it ?

Looks like Monty's filter example might be usefull...

http://www.codetransform.com/filterexample.html

> eating everything EXCEPT the newline, which needs to be passed through
> to the parser. Note also, that while "*" and "!" are tokens, "rem" is
an
> ident. 

So it's:

(NEWLINE | SEMI) (rem:ID | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )

> Of course, if I need "rem" to be a token, presumably if this rule
> fails I can convert it back to an ident?

Should be possible to modify the ident to a REM token I guess.

I guess I'd handwrite a small parser for the above might be simpler than
coaxing antlr in doing it. Although I'm not sure you migth get away with
some semantic predicate fun.

> Or do I need to write my own token stream to do this (if so, where do
I
> look for an example?)

See the link to Monty's site.

Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++
-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++
-------
  "I think we better split up."
  "Good idea. We can do more damage that way."
  --- Ghostbusters



 
Yahoo! Groups Links



 



****************************************************************************

This transmission is intended for the named recipient only. It may contain private and confidential information. If this has come to you in error you must not act on anything disclosed in it, nor must you copy it, modify it, disseminate it in any way, or show it to anyone. Please e-mail the sender to inform us of the transmission error or telephone ECA International immediately and delete the e-mail from your information system.

Telephone numbers for ECA International offices are: Sydney +61 (0)2 8272 5300, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 and New York +1 212 582 2333.

****************************************************************************



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list