[antlr-interest] Re: A simple prepocessor with a lexer?

[Terence Parr]

On Monday, October 6, 2003, at 04:28 PM, maaxxxcal wrote:

> Wow thanks for your quick answer, it worked really well. I didn't
> know about the "filter" option.
> One last thing though: I'd like to generate one token for each piece
> of text that is between my pragmas.

Oh that's different.  Simplest thing to do is create another lexer that 
handles that text and then, using a SharedLexerInputState (or whatever 
it's called), create an instance of the new lexer in the outer lexer 
and then make a while loop around it's nextToken().

You can grep for that class name in the examples probably.

Ter


> How would you do that?
> I tried to use filter=RULE to catch all the filtered text, but I
> don't know how to express "anything until the next pragma", since I
> cannot use ~PRAGMA (if PRAGMA is the rule defining the syntax of my
> pragma).
>
> With the option filter=CODE, I tried
>
> CODE: . ; (equivalent to filter=true)but that creates a token for
> every character
> CODE: (.)*; but that consumes all the characters till EOF
>
> What startegy would you use to effectively catch all the text between
> the pragmas?
>
> Thanks again for your help!
>
>
> --- In antlr-interest at yahoogroups.com, Terence Parr <parrt at c...>
> wrote:
>> Hi.  Damn yahoogroups sucks.  Post from home didn't make it.
> Anyway,
>> try this:
>>
>> class L extends Lexer;
>>
>> options {
>> 	filter=true;
>> }
>>
>> CMT : "--" ( PRAGMA | (options{greedy=false;}:.)* '\n' ) ;
>>
>> protected
>> PRAGMA : "<pragma"> ... ;
>>
>> On Saturday, October 4, 2003, at 08:21 PM, maaxxxcal wrote:
>>
>>> I am trying to write a very simple preprocessor but I'm not quite
>>> sure how to go about it.
>>> Here's the problem, I need to process pragmas that are really
> inside
>>> single-line comments in a SQL script (not necessarily from the
> start
>>> of line).
>>> For instance, I can have input of the form:
>>>
>>> <sql>
>>> ...
>>> <sql> -- <pragma>
>>> ...
>>> <sql>
>>>
>>> What is the simplest way to achieve this with ANTLR? Is a lexer
>>> enough as long as the syntax of the pragma is not too complex?
>>> I tried writing such a lexer but I'm having trouble defining a
> rule
>>> that would match "anything until --"
>>>
>>> Thanks for your help!
>>>
>>>
>>>
>>>
>>> Your use of Yahoo! Groups is subject to
>>> http://docs.yahoo.com/info/terms/
>>>
>>>
>>>
>> --
>> Professor Comp. Sci., University of San Francisco
>> Creator, ANTLR Parser Generator, http://www.antlr.org
>> Co-founder, http://www.jguru.com
>> Co-founder, http://www.knowspam.net enjoy email again!
>> Co-founder, http://www.peerscope.com pure link sharing
>
>
>
>
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
>
>
>
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com pure link sharing




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/