[antlr-interest] How to tell the lexer to stop?(C runtime target )

[chain one]

I think I have found the way to tell the lexer to stop.The way is :
1. Place a BOOL variable "if_lexer_need_stop" in the generated lexer .c file
: BOOL if_lexer_need_stop = FALSE;
2. Override the function nextTokenStr and function myNextToken :
    modify this line from
   "if  (lexer->input->istream->_LA(lexer->input->istream, 1) ==
ANTLR3_CHARSTREAM_EOF)"
   to
  "if
 (if_lexer_need_stop ==TRUE||lexer->input->istream->_LA(lexer->input->istream,
1) ==        ANTLR3_CHARSTREAM_EOF)"

3. Install the new nextToken() method:
    a). Add this line after calling antlr3LexerNewStream in the
function testlexLexerNewSSD:
       "ctx->pLexer->rec->state->tokSource->nextToken = myNextToken;"
4. Modify one terminal lex rule to the form like:
    SEMI : ';' {if_lexer_need_stop = TRUE;}
;





On Fri, Jan 16, 2009 at 8:33 AM, chain one <chainone at gmail.com> wrote:

> Thanks JIm
> You suggestion is very useful to me.
> Here is my another concern to the method you provided:
> If the file is very large, would it lead the lexer to read though the
> whole file? If it is, I think it would be slower than the method of
> just telling the lexer to stop.
>
> On 1/16/09, Jim Idle <jimi at temporal-wave.com> wrote:
> > chain one wrote:
> >>
> >> I want to only parse the beginning of a file. And I don't want the
> >> recognizer to beak the whole file into huge number of tokens which will
> >> make the memory consuming very high.
> >> So after the lexer having recognized one specified TOKEN such as ";", I
> >> want to tell the lexer to stop, and pass the tokens to the parser.
> >> Is it possible in C runtime target? If it is, could you tell me how?
> >>
> >>
> >> SEMI: ';'  { // throw the EOF token?}
> >> ;
> >>
> > If the trigger point is as simple as this, then why not just:
> >
> > SEMI : ';' .* ;
> >
> > Remember that the text is not copied into the token, it is just a pointer
> to
> > the start and end of the text and so does not take up any memory.
> >
> > Jim
> >>
> >>
> >> Thanks
> >>
> >> ________________________________
> >>
> >> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> >> Unsubscribe:
> >> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >>
> >
> >
> >
>
> --
> Sent from my mobile device
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090116/82f6692b/attachment.html