[antlr-interest] Antlr3.4 Python bugs, workarounds

Terence Parr parrt at cs.usfca.edu
Thu Feb 9 10:51:48 PST 2012


Hi Benjamin (Wolf), thanks very much.

I've recently change the main v3 repository to antlr/antlr3 not antlr/antlr. Since I am a newbie at git/github, could you send me a pull request to that new repository for the same fix? then I can integrate it.

I've just left a redirect signal in the readme file at antlr/antlr so hopefully people will see that now.

thansk!
Ter
On Feb 4, 2012, at 1:13 AM, Benjamin S Wolf wrote:

> For reference, I believe I've found the culprit code. I was browsing
> the antlr network on github and I came across this commit:
> https://github.com/vslavik/antlr/commit/bbf349ab7d7ccb468604f95b28b08eefd694206f
> 
> ie. the code probably looked like:
> 
> <dfa.specialStateSTs:{state | \r\n
> if s == <i0>: <! compressed special state numbers 0..n-1 !>
>    <state>}; separator="\nel">
> 
> which caused a newline to be output before "if",
> and I changed it to
> 
> <dfa.specialStateSTs:{state | \n
> if s == <i0>: <! compressed special state numbers 0..n-1 !>
>    <state>}; separator="\nel">
> 
> which fixed the issue for me,
> and vslavik changed it to
> 
> <dfa.specialStateSTs:{state | if s == <i0>: <! compressed special
> state numbers 0..n-1 !>
>    <state>}; separator="\nel">
> 
> which apparently fixed the issue for him.
> 
> On Wed, Dec 28, 2011 at 2:47 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>> 
>> On Dec 27, 2011, at 5:08 PM, Benjamin Niemann wrote:
>>>> Hi Benjamin,
>>>> 
>>>> I was meddling around with the stg templates for Python in trying to
>>>> fix some other bugs I reported in another thread, and after updating
>>>> the files in antlr-3.4-complete.jar this problem was alleviated.
>>>> 
>>>> I narrowed down the diff and discovered that the stg templates in the
>>>> original jar all had DOS line endings (that is, \r\n instead of just
>>>> \n), and that removing all the carriage returns in
>>>> org/antlr/codegen/templates/Python/Python.stg solved the issue of the
>>>> elif being split across a newline.
>>>> 
>>>> That certainly explains why it only showed in antlr-3.4-complete.jar,
>>>> since the templates included with antlr-3.4.tar.gz did not have the
>>>> carriage returns. :)
>>> 
>>> Good catch, thanks a lot for figuring that out.
>>> That seems like a bug in stringtemplate to me - I thought it was
>>> smarter about dealing with line endings.
>>> 
>>> Ter:
>>> Was the jar built on a windows box?
>> 
>> nope. mac os x.
>> 
>>> I assume perforce adds the CRLFs
>>> when checking out the files under windows - the files are stored as
>>> "text", i.e. line endings are converted to the native system.
>> 
>> that's correct i think.
>> 
>>> Unless ST can be taught to deal with that properly, we could store the
>>> templates as binary in the repository - but that could be messy when
>>> someone actually wants to edit them under windows and it's hard to
>>> notice when CRs creep back in.
>>> Or avoid building jars on windows ;)
>>> This probably affects other targets as well, but those are probably
>>> less picky about some extra whitespace here and there. Could lead to
>>> some obscure bugs though.
>> 
>> ST emits proper newlines per platform and should read \r\n just like \n.  BUT…apparently not ;)  I'm adding to list to check ST v4 to see how it works.
>> 
>> Ter



More information about the antlr-interest mailing list