[antlr-interest] Is there a safe and easy way to reuse LEXER and PARSER objects on C target?

[Jim Idle]

This is available as the reuse() method on the input stream, lexer, and
parser. If used then the lexer will also reuse the tokens from the last
run and avoid any malloc.

However, you will need to use the snapshot in perforce to get that. Or
just wait a few weeks for the next release.

jim

> -----Original Message-----
> From: Ruslan Zasukhin [mailto:ruslan_zasukhin at valentina-db.com]
> Sent: Wednesday, April 20, 2011 2:42 AM
> To: Jim Idle
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Is there a safe and easy way to reuse
> LEXER and PARSER objects on C target?
>
> On 12/11/09 6:17 PM, "Jim Idle" <jimi at temporal-wave.com> wrote:
>
> And I hope this is already fixed during last 1.5 year?   :-)
>
> I bet that re-create lexer/parser objects is much slower.
> In v2 we have very fast drop this idea.
> And v2 ANTLR was able easy enough reuse  lexer/parser and treeparser.
>
>
> > Ok - that is probably a bug. I guess nobody tried to do that before
> > ;-). For now, you will have to recreate the parser each time until I
> can fix it.
> >
> > Jim
> >
> > From: Ronghui Yu [mailto:stoneyrh at gmail.com]
> > Sent: Friday, December 11, 2009 5:01 AM
> > To: Jim Idle
> > Cc: antlr-interest
> > Subject: Re: [antlr-interest] Is there a safe and easy way to reuse
> > LEXER and PARSER objects on C target?
> >
> > Here is my pseudo code:
> >
> > pLexer->pLexer->rec->reset(pLexer->pLexer->rec);
> > pInputStream->data = (pANTLR3_UINT8)GetText().c_str(); sizeBuf =
> > pInputStream->(ANTLR3_UINT32)GetText().length();
> > pInputStream->reset(pInputStream);
> > pLexer->pLexer->setCharStream(pLexer->pLexer,pInputStream);
> > pTokenStream->free(pTokenStream);
> > pTokenStream = antlr3CommonTokenStreamSourceNew(TOKEN_SIZE_HINT,
> > TOKENSOURCE(pLexer));
> > pTokenStream->discardOffChannelToks(pTokenStream,ANTLR3_FALSE);
> > pParser->pParser->setTokenStream(pParser->pParser,pTokenStream-
> >tstrea
> > pParser->pParser->m);
> >
> >
> > It works most of the time, but occasionally violation access occurs.
> > It doesn't work if applied to a grammar file importing another
> > grammar. For example, I have a a keywords.g file is imported to the
> > main grammar. When setting the token stream on the main grammar, the
> > token string of embedded parser does not get updated automatically.
> >
> > On Fri, Dec 11, 2009 at 1:11 AM, Jim Idle <jimi at temporal-wave.com>
> wrote:
> > To be honest, I would think you can hardly measure the time taken to
> > create them, but you have to call the reset() methods and set the
> > character stream and the token stream. There were issues with that at
> > one point but I think I have fixed them all now. You can see how to
> > reuse the lexer by looking at the examples in the examples download.
> > Specifically the Java parser example will help here.
> >
> > Jim
> >
> > From: antlr-interest-bounces at antlr.org
> > [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Ronghui Yu
> > Sent: Thursday, December 10, 2009 8:21 AM
> > To: antlr-interest
> > Subject: [antlr-interest] Is there a safe and easy way to reuse LEXER
> > and PARSER objects on C target?
> >
> > Hi, All,
> >
> > On my project, I have a parser for parsing different statements again
> > and again. In order to save a little time on initialization, I would
> > like to reuse the LEXER and PARSER objects created the first time,
> something like this:
> >
> > if (bInitialized)
> > {
> >     reinitialize();
> > }
> > else
> > {
> >    initialize();
> >    bInitialized = true;
> > }
> >
> > The problem now is how to write reinitialize() safely. I have no idea
> > on which fields of LEXER or PARSER objects must be reset to which
> > status. Then my current code works most of the time, but it
> encounters
> > NULL pointer occasionally(I am sure the grammar file is good because
> > if I don't reuse the LEXER and PARSER objects, everything goes fine).
> >
> > Anybody could give me some ideas?
> >
> > Thanks in advance.
> >
> > --
> > ===========================
> > Regards
> > Ronghui Yu
> >
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-
> address
> >
> >
> >
> > --
> > ===========================
> > Regards
> > Ronghui Yu
> >
> >
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-
> address
>
> --
> Best regards,
>
> Ruslan Zasukhin
> VP Engineering and New Technology
> Paradigma Software, Inc
>
> Valentina - Joining Worlds of Information http://www.paradigmasoft.com
>
> [I feel the need: the need for speed]
>