[antlr-interest] Token stream filter
Ric Klaren
klaren at cs.utwente.nl
Wed Jun 2 07:17:18 PDT 2004
Hi,
On Wed, Jun 02, 2004 at 11:26:53AM +0100, Anthony Youngman wrote:
> Okay. Doing my own homework :-) Would this work (needing cleaning up, I
> don't know what to throw etc ...)?
>
> filter=DEREMER
>
> protected DEREMER [returns token] : (
> (newline|semi)
> (id:IDENT {if text != "REM" throw tokenmatchexception}|"*"|"!")
> ("=" {throw exception} | "(" {throw exception})?
> ({greedy=false} .)
> newline)
> returns newline;
filter=<rule> only works in lexers AFAIK.
> I want to filter the token stream coming out of the lexer (and I can't
> see how to do it using the lexer's filter - can I do the same thing in
> the parser?). Anyways, I want to match the following sequence
>
> (newline | semi) ("rem" | "*" | "!" ) ~( "=" | "(" ) ({greedy=false} .)
> newline)
So to put it in parser Lingo:
(NEWLINE | SEMI) ("rem" | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )
There's only a set of closures missing by the looks of it ?
Looks like Monty's filter example might be usefull...
http://www.codetransform.com/filterexample.html
> eating everything EXCEPT the newline, which needs to be passed through
> to the parser. Note also, that while "*" and "!" are tokens, "rem" is an
> ident.
So it's:
(NEWLINE | SEMI) (rem:ID | "*" | "!") ~( "="| "(" ) ( {greedy=false} . )
> Of course, if I need "rem" to be a token, presumably if this rule
> fails I can convert it back to an ident?
Should be possible to modify the ident to a REM token I guess.
I guess I'd handwrite a small parser for the above might be simpler than
coaxing antlr in doing it. Although I'm not sure you migth get away with
some semantic predicate fun.
> Or do I need to write my own token stream to do this (if so, where do I
> look for an example?)
See the link to Monty's site.
Cheers,
Ric
--
-----+++++*****************************************************+++++++++-------
---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755 ----
-----+++++*****************************************************+++++++++-------
"I think we better split up."
"Good idea. We can do more damage that way."
--- Ghostbusters
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list