[antlr-interest] Reading from sys.stdin with ANTLR-Python (for an REPL)

[Darien Hager]

On 2/18/08, Benjamin Niemann <pink at odahoda.de> wrote:
>
> 
> ANTLR out of the box is not really suited for such incremential parsing -
> that's not a Python specific issue. Building an incremential TokenStream
> is
> not a big deal - I have the code already, and I'm planning for a long time
> now to clean it up and add it to the Python runtime... But the logic to
> detect when a statement is complete resides in the parser and I don't know
> a solution for that problem...
>

I think there was some talk a while back about incremental parsing in ANLTR,
but I'm not sure what came of it. It would be pretty cool to use it for
things like syntax-highlighting tools, etc.


On 2/17/08, *Aaron Lebo* <aaron.m.lebo at gmail.com> wrote:
> So I'm having no issue with that. What I am wanting to do is an REPL and I
need to interactively parse the data coming from sys.stdin.
> The problem is that all of the 'Stream' classes from the antlr3 package
read all input on creation, so passing sys.stdin or sys.stdin.read()
> causes the REPL to hang, as something like antlr3.ANTLRInputStream(
sys.stdin) reads infinitely.





I dabbled with parsing input data incrementally a little while back with a
simple grammar. Just an outline for what sorta-kinda-might-have worked (Java
runtime)

A) Define a parser rule to match one single statement, when expecting a
series.
B) Define "junk" tokens for things like unterminated quoted strings, invalid
token fragments terminated by EOF, etc.
C) Subclass the parser/lexer to disable automatic recovery and to generally
throw exceptions more readily.
D) Have an inner and outer loop.
E) The outer loop makes a new token stream starting with any leftovers
from the last token stream. IT then appending the incoming data.
F) The inner loop tries uses the parser and token stream to match as many
statements as it can, marking it's progress in the stream each time. When it
fails (parser exception), it breaks out of the loop.

-- 
Darien Hager
Developer
Etelos, Inc.
darien at etelos.com

http://www.etelos.com
"Revolutionizing the way applications are developed, distributed and
consumed."

This e-mail message, including attachments, may contain confidential
information for the sole use of the intended recipient(s). If you are not
the intended recipient, then this is notice that any use, disclosure,
dissemination, distribution or copying is strictly prohibited. If you have
received this message in error please contact the sender by reply mail and
destroy all copies of the original message.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080219/a1a06cf7/attachment-0001.html