[antlr-interest] Extends the lexer in C target

Jim Idle jimi at temporal-wave.com
Wed Mar 3 09:57:23 PST 2010


Fabien,

Look at the generated code and the 'constructor' for your lexer. If you trace through that code you will see that the mTokens() is just a static function, the pointer to which is installed in the lexer structures. You ca install your own.

However, pre-processors are easy enough and though you have to add extra code to your lexer rules, it is much easier (and more maintainable) to set the channel of the token to a member variable value or call SKIP. Then implement the pre-processor in the lexer. Sometimes you cannot do this though. For instance the C# pre-processor is implemented in the lexer, but the VB pre-processor is its own up front program (which I implemented as a parser grammar myself).

In your case though, I don't think that you want to override mToken(), you want to override the nextToken and nextTokenStr functions. Just copy them from antlr3lexer.c, modify as needed and perform the SKIP operation when your flag is set to 'off'. You then install the pointer to your version of nextToken() in the lexer structures after you have created your lexer and before anything calls in to it.


As I say though, personally I think it is neater to explicitly code for this in the lexer rules.

Jim

-----Original Message-----
From: Fabien Antoine [mailto:fab.antoine at gmail.com] 
Sent: Wednesday, March 03, 2010 12:02 AM
To: Jim Idle
Subject: Re: [antlr-interest] Extends the lexer in C target

Hi Jim and thanks for your answer.

I saw all what you said concerning the lexer implementation and it's
precisely the problem.

I want to create a kind of preprocessor that interprets statements only
if a constant is defined. And I don't want to manage a Preprocessor
token in all my rules.

I tried to reproduce the system used in the C# parser done by Andrew
Bradnan but its output target is C#. Then he can extend the generated
lexer and override the "mTokens" function. By overriding the "mTokens"
function, he can push the next token into the hidden channel if the
preprocessor decided that current statements must not be returned.

Then the only thing I want to do is to add this "filter" to the lexer.
I'm fully open to other "cleaner" solutions...

Fabien

On 02/03/2010 16:53, Jim Idle wrote:
> mTokens is generated but you can install any function you like in its place by adding code in the initialization. You should see that the address of the static function is just stored in the lexer structures. Just replace the pointer with a pointer to your own function. Read through the source though to find out what you need to do in that function and the functions that it invokes.
>
> That said, if you are trying to override this function, then you are really replacing the lexer. Perhaps you should tell us why you want to override that and what you are trying to achieve - maybe there is a better way.
>
> Jim
>   






More information about the antlr-interest mailing list