[antlr-interest] 3.1 TokenSource

[Jim Idle]

Flattery will get you everywhere ;-)

 

I have documented this change in the examples for 3.1 but will document it in the changes from 3.0 section of the docs soon - apologies in the meantime. When the interface changed for 3.1, the token source pointer had to move down a few levels and it was already a bit awkward to find this (intuitively at least), so the C examples for 3.0 will not work with 3.1 generation.

 

To make this all a bit easier, there is now a new macro (I generally avoid these as debugging with them is a night mare but here it is justified). The macro is called TOKENSOURCE and basically it knows how to pull out the tokensource pointer from a lexer context pointer. The 3.1 C/C++ examples show how to use it, and here is a snippet:

 

tstream = antlr3CommonTokenStreamSourceNew(ANTLR3_SIZE_HINT, TOKENSOURCE(lxr));

 

That should be all you need :-)

 

Jim

 

From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Christian Schladetsch
Sent: Thursday, April 10, 2008 2:34 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] 3.1 TokenSource

 

Hello [Jim],

First off, huge kudos to Jim for the C codegen for ANTLR. I've read 80% of it, and it is a thing of beauty. I've not seen better C code in my 15 years as a professional C/C++ developer.

Secondly, now that I finally have a build of 3.1 ANTLR, i get the followng error:

    error C2039: 'tokSource' : is not a member of 'ANTLR3_LEXER_struct'

from C++ source:

    pPiLexer lxr = PiLexerNew(input);
    pANTLR3_COMMON_TOKEN_STREAM tstream = antlr3CommonTokenStreamSourceNew(ANTLR3_SIZE_HINT, lxr->pLexer->tokSource);

PiLexer is generated from Pi.g using custom build rules in VS2008:

1>ANTLR Parser Generator  Version 3.1b1 (??)  1989-2007
1>Generating PiParser.c
1>Generating PiParser.h
1>Generating PiLexer.c
1>Generating PiLexer.h
1>Compiling...
1>Main.cpp
1>[obscured]\rhocompiler\main.cpp(11) : error C2039: 'tokSource' : is not a member of 'ANTLR3_LEXER_struct'
1>        e:\dev\antlr\antlr-2008-04-09.10\runtime\c\include\antlr3lexer.h(48) : see declaration of 'ANTLR3_LEXER_struct'
1>Build Time 0:02
1>Build log was saved at "file://c:\Users\user\Desktop\MabPrefix\Projects\RhoCompiler\Debug\BuildLog.htm"
1>RhoCompiler - 1 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

I have spent some time investigating this. It is clear that that the lexer should indeed supply a source of tokens. But I see no way for PiLexer (which is auto-generated) to return a token source. 

Regards,
Christian.

PS. What's with the Java thing. When will Terr move to C++ and make? ;)



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080410/d8221f58/attachment.html