[antlr-interest] Aggressive vs. Lazy Buffering in ANTLR

Foolish Ewe foolishewe at hotmail.com
Tue Dec 19 10:05:58 PST 2006


Hi Kay:

Thanks to you for the hints.  I'm somewhat restricted in which code I can 
publish without
managerial/legal approval.  I have modified the Expr example from the wiki 
to have some
additional debugging output and created a "MyLazyInputStream" class.  The 
constructor
for this new class is pretty simple, it does an assignment of the input 
stream and allocates
storage to buffer input and does little else (the class' stream position 
pointers, etc.
all use compile time initialization).

Since my last post, I've taken a closer look at my code and discovered that 
I
had statements of the form

        CommonTokenStream tokens = new CommonTokenStream(lexer);
        System.out.println("After TokenStream Constructor, tokens = " + 
tokens + ">");

Instead of dumping the reference address, this appaeared to somehow invoke 
fillbuffer,
when I changed System.out.println to no longer try to print the tokens 
object/reference,
then controll passed into the parser which I think then blocks on 
tokens.fillbuffer();
I removed the dumps of the tokens and parser objects, now the software hangs 
so that
the Eclipse debugger can't break it in the parser.prog (unusual because I 
was able to do this
with all previous examples).

I still suspect that the tokens.fillbuffer() method aggressively buffers 
tokens, but I'll
need to check it more carefully to confirm this, since I'm not sure of why I 
get no output
when I enter
a = 3 + 4

In the Eclipse debugger (I don't know of anyway to get Eclipse to pass an 
EOF on input, it
appears to filter out control D).

Regards:

Bill M.

>From: Kay Roepke <kroepke at classdump.org>
>To: Foolish Ewe <foolishewe at hotmail.com>
>CC: antlr-interest at antlr.org
>Subject: Re: [antlr-interest] Aggressive vs. Lazy Buffering in ANTLR
>Date: Tue, 19 Dec 2006 17:32:28 +0100
>
>
>On 19. Dec 2006, at 16:59 , Foolish Ewe wrote:
>
>>However, I tried making a simple input method that uses Lazy  buffering 
>>(i.e. reads characters
>>on demand extending the buffer).  This lazy input stream no longer  blocks 
>>on the constructor,
>>however, now I've noticed that the CommonTokenStream constructor  appears 
>>to block.
>
>What do you do in your input stream's constructor? The  CommonTokenStream 
>constructors shouldn't block,
>they just set the members.
>
>>It appears to be blocking in fillBuffer, but I don't see an  explicit call 
>>in the constructor.
>
>fillBuffer is invoked on the first call to LT. Maybe you're seeing  that? 
>Though it doesn't sound
>likely based on what you described...
>
>>The impact of this is that it negates the interactive support that  the 
>>lazy buffering of my
>>home-grown input stream class, so I'd like to fix it.
>>
>>Is there a lazy buffering (buffer as input becomes available/on  demand) 
>>variant of TokenStream
>>support or have I made some error?
>
>No there's nothing like that in the standard distribution. Could you  post 
>some code?
>
>cheers,
>
>-k
>
>--
>Kay Röpke
>http://classdump.org/
>
>
>
>

_________________________________________________________________
Get free, personalized online radio with MSN Radio powered by Pandora 
http://radio.msn.com/?icid=T002MSN03A07001
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Test.java
Type: application/octet-stream
Size: 1035 bytes
Desc: not available
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20061219/7e64b509/attachment.obj 


More information about the antlr-interest mailing list