[antlr-interest] parsing python

[Terence Parr]

On Feb 27, 2004, at 11:16 PM, Kaleb Pederson wrote:

> On Friday 27 February 2004 5:03 pm, Terence Parr wrote:
>> Well, I've embarked on my parser for python.  First task was to
>> autoconvert the distribution grammar to ANTLR format.  Then I jumped 
>> on
>> the lexical issues.  Eewwwwwwwwwww!  Should never watch sausage being
>> made or python being lexically analyzed.  I think Humans can get 
>> pretty
>> used to this weird indentation thing as it's nice visually.  Getting
>> the lexer to handle the weird structure and the fun exceptions to the
>> rules, is no picnic.
>
> Lol.  Weird is all a matter of definition.
>
>> Anyway, I have something that looks like a parser with a
>> PythonTokenStream that does the right INDENT/DEDENT imaginary token
>> generation, but I have to add in a few random "allow trailing commas
>> for no reason" (COMMA)? subrules.  Then I will attempt a symbol table
>> manager so I can learn the semantics of symbol table lookup.
>
> Hmm.  IIRC, the python grammar is very well defined, so shouldn't you 
> just be
> able to take what's there and make it LL(k)? (hopefully without too 
> much
> work?)

It's LL(1) 'cept for the optional commas and semis, which require k=2.

> As far as the commas and Python, I presume you are talking about tuple
> packing/unpacking? Maybe on the print statement (indicating no 
> newline)?

Stuff like that and

a=1; b=2;

and

foo(1,3,)

that's just wrong in my book, but I'm a language implementor not 
designer ;)

>> I'll post something when it's ready for public consumption. :)
>
> One of the reasons that I chose antlr is that I have heard that Python 
> support
> is underway?  How is it coming?  Is there anything that I can do to 
> help?

Well, a few people are playing around.  We have a python generator that 
works with jython, but nobody has translated the libraries to python 
yet.  I'm madly learning python as fast as I can ;)

> I'll gladly test it out.  I tried out several different Python parsers

Should be something in the file sharing area you can try out.

> (LR/LALR) for some work projects, but the debugging information they 
> provided
> made them nearly useless so I ended up writing them by hand.  I 
> recently ran
> across a few more, but was hoping I could wait for antlr to pick up 
> Python
> support.
>
>> Nice language, python....just a little kinky lexically. ;)
>
> Oh yeah.  I still have to get used to the lexer parsing structure and 
> how it
> stands apart from all the other ones that I have seen up to this 
> point.  I
> suppose then that it wasn't the grammar that was giving you 
> problems....

Nope...grammar is autotranslated from distribution.  The lexer is 
totally nasty with context-sensitive whitespace etc...  For example, 
NEWLINE must be sent to the parser as it is statement separate etc..., 
but inside (...), [...], {...} whitespace indentation and newlines are 
totally ignored and not sent to the parser.

> Thanks for the feedback and the tool.  I really like how I can look at 
> LA(k)
> and FIRST(set).  They make it really easy to handle things that were
> previously much more difficult.

:)

Ter
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!
Cofounder, http://www.peerscope.com pure link sharing





 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/