[antlr-interest] Possible bug in C runtime: Unable to override emit() in lexer

[Michael Carns]

Jim,

I'm planning on using the custom pointer to stash my data.  When I first 
looked those fields, it looked to me like they are designed to be set by 
action blocks in the grammar.  I have some common metadata that needs to 
go along with every token, and I'd rather not have to insert and 
maintain an action block for every single lexer rule.  That's what 
started me down the path of overriding emit.  It provided a common point 
where I could adjust those values.  Is there a better place to insert my 
code?

Thanks,
Mike

On 4/26/2011 8:05 AM, Jim Idle wrote:
> No, you override both of these with your own function, and your own
> function should call your own version of emit if the built in one is not
> what you want.
>
> However most people will not need to use a customized token type and emit
> overrides and so on. There is a pointer and 3 integers in the token that
> can be used to store any additional information that you need.
>
>
> Jim
>
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Michael Carns
>> Sent: Tuesday, April 26, 2011 12:42 AM
>> To: antlr-interest at antlr.org
>> Subject: [antlr-interest] Possible bug in C runtime: Unable to override
>> emit() in lexer
>>
>> In order to use a customized token type, I believe I need to override
>> emit() and emitNew() within my ANTLR3_LEXER instance.  However, my
>> function isn't getting called.  I'm using version 3.1.3 of the C
>> runtime.
>>
>> The culprit appears to be this call to emit() within nextTokenStr() in
>> antlr3lexer.c:
>>
>> Line 268:
>>                  if (lexer->rec->state->token == NULL)
>>                  {
>>                      // Emit the real token, which adds it in to the
>> token stream basically
>>                      //
>>                      emit(lexer);
>>                  }
>>
>> Shouldn't it be "lexer->emit(lexer)"?  As written, the call to emit()
>> will bind to the local version of emit() within antlr3lexer.c and won't
>> call my customized version.  I've looked at more recent versions in the
>> source repository, and the code appears to be unchanged.  Is this a
>> real bug, or am I going down the wrong road in my attempt to customize
>> the generated tokens?
>>
>> Thanks,
>> Mike Carns
>>
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>