[antlr-interest] Re: A simple prepocessor with a lexer?

[Terence Parr]

Ack...a context-sensitive problem.  Ok, perhaps a two pass system where 
the first pass reads the separator and then converts them all to a 
single, normalized separator.  Then the real parser can walk the output 
and easily parse the input. :)  Think preprocessor.

Ter
On Tuesday, October 7, 2003, at 05:36 PM, maaxxxcal wrote:

> Hummm ok, I promise this is the last post on that topic. I realize
> that maybe I should have been more clear about what I was trying to
> achieve from the beginning.
> So the class you're talking about is LexerSharedInputState, but I'm
> not sure it would do for me because I don't know when to stop the
> inner Lexer. Let me explain.
> One of the effect of my pragma is to define what the statement
> separator is going to be for the next portion of code. I already have
> a lexer and parser that know how to parse a single statement, with no
> separator. Now what I'm trying to do is to wite a lexer that will
> process my script, understand the pragmas and cut statements to feed
> them to my single-statement lexer/parser.
>
> Here's the structure that the scripts I have to parse can have:
> -- PRAGMA separator=@
> -- PRAGMA error=on
> <statement>@
> <statement>@
> -- PRAGMA error=off
> -- PRAGMA separator=%
> <statement>%
> <statement>%
>
> What's interesting in this problem is that an important part of the
> syntax (the statement separator) is only defined at runtime. I
> believe there is no way to do that with only 1 lexer/parser with
> ANTLR because it has to be able to predict statically what the
> alternatives are.
>
> So what strategy would you use?
> Thanks again for your help!!
>
>> Oh that's different.  Simplest thing to do is create another lexer
> that
>> handles that text and then, using a SharedLexerInputState (or
> whatever
>> it's called), create an instance of the new lexer in the outer
> lexer
>> and then make a while loop around it's nextToken().
>>
>> You can grep for that class name in the examples probably.
>>
>> Ter
>> --
>> Professor Comp. Sci., University of San Francisco
>> Creator, ANTLR Parser Generator, http://www.antlr.org
>> Co-founder, http://www.jguru.com
>> Co-founder, http://www.knowspam.net enjoy email again!
>> Co-founder, http://www.peerscope.com pure link sharing
>
>
>
>
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
>
>
>
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com pure link sharing




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/