[antlr-interest] streaming parser?

[Jim Idle]

On 09/19/2009 12:12 PM, Karl Meissner wrote:
> Are there any good article about using antlr in the context of a 
> steaming parser?
Search the list for prior discussions, but all you really need is to 
write your own input stream that gives EOF back to the caller when a 
complete "unit" is detected, then keep setting up the input stream 
again. Presumably there is something in the incoming packets that 
indicates a safe place to call something a "unit" of work?

>
> I have text files which I need to filter and search as they download, 
> packet by packet.  The packets may fragment the text tokens so I need 
> some sort of stateful stream....
>
>
> It would be really nice if I could arrange for the parse to run 
> deterministically. I am hoping to use Antlr in a highly optimized, low 
> latency application.   boost regex was not up to the task...
Depends what you mean by deterministically? You mean that after nnms of 
cpu or elapsed time that it gives up control to an embedded system 
control loop? Or that it always takes the same time? You will want to 
increase the value of -Xmaxdfaedges to a large number to avoid DFAs 
being generated as tables (inline Ifs are much faster because of branch 
prediction and avoiding memory cache misses).

BTW - I presume that you are using the C target here as you have no 
chance of achieving the kind of performance you seem to be asking for 
from the other targets.

Jim