[antlr-interest] Antlr3.4 Python bugs, workarounds

[Benjamin S Wolf]

Sure. I'll note that the fixes I had there were for a different set of
issues. But I can add in my fixes for the two here, then put the pull
request on antlr/antlr3.

On Thu, Feb 9, 2012 at 10:51 AM, Terence Parr <parrt at cs.usfca.edu> wrote:
> Hi Benjamin (Wolf), thanks very much.
>
> I've recently change the main v3 repository to antlr/antlr3 not antlr/antlr. Since I am a newbie at git/github, could you send me a pull request to that new repository for the same fix? then I can integrate it.
>
> I've just left a redirect signal in the readme file at antlr/antlr so hopefully people will see that now.
>
> thansk!
> Ter
> On Feb 4, 2012, at 1:13 AM, Benjamin S Wolf wrote:
>
>> For reference, I believe I've found the culprit code. I was browsing
>> the antlr network on github and I came across this commit:
>> https://github.com/vslavik/antlr/commit/bbf349ab7d7ccb468604f95b28b08eefd694206f
>>
>> ie. the code probably looked like:
>>
>> <dfa.specialStateSTs:{state | \r\n
>> if s == <i0>: <! compressed special state numbers 0..n-1 !>
>>    <state>}; separator="\nel">
>>
>> which caused a newline to be output before "if",
>> and I changed it to
>>
>> <dfa.specialStateSTs:{state | \n
>> if s == <i0>: <! compressed special state numbers 0..n-1 !>
>>    <state>}; separator="\nel">
>>
>> which fixed the issue for me,
>> and vslavik changed it to
>>
>> <dfa.specialStateSTs:{state | if s == <i0>: <! compressed special
>> state numbers 0..n-1 !>
>>    <state>}; separator="\nel">
>>
>> which apparently fixed the issue for him.
>>
>> On Wed, Dec 28, 2011 at 2:47 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>>>
>>> On Dec 27, 2011, at 5:08 PM, Benjamin Niemann wrote:
>>>>> Hi Benjamin,
>>>>>
>>>>> I was meddling around with the stg templates for Python in trying to
>>>>> fix some other bugs I reported in another thread, and after updating
>>>>> the files in antlr-3.4-complete.jar this problem was alleviated.
>>>>>
>>>>> I narrowed down the diff and discovered that the stg templates in the
>>>>> original jar all had DOS line endings (that is, \r\n instead of just
>>>>> \n), and that removing all the carriage returns in
>>>>> org/antlr/codegen/templates/Python/Python.stg solved the issue of the
>>>>> elif being split across a newline.
>>>>>
>>>>> That certainly explains why it only showed in antlr-3.4-complete.jar,
>>>>> since the templates included with antlr-3.4.tar.gz did not have the
>>>>> carriage returns. :)
>>>>
>>>> Good catch, thanks a lot for figuring that out.
>>>> That seems like a bug in stringtemplate to me - I thought it was
>>>> smarter about dealing with line endings.
>>>>
>>>> Ter:
>>>> Was the jar built on a windows box?
>>>
>>> nope. mac os x.
>>>
>>>> I assume perforce adds the CRLFs
>>>> when checking out the files under windows - the files are stored as
>>>> "text", i.e. line endings are converted to the native system.
>>>
>>> that's correct i think.
>>>
>>>> Unless ST can be taught to deal with that properly, we could store the
>>>> templates as binary in the repository - but that could be messy when
>>>> someone actually wants to edit them under windows and it's hard to
>>>> notice when CRs creep back in.
>>>> Or avoid building jars on windows ;)
>>>> This probably affects other targets as well, but those are probably
>>>> less picky about some extra whitespace here and there. Could lead to
>>>> some obscure bugs though.
>>>
>>> ST emits proper newlines per platform and should read \r\n just like \n.  BUT…apparently not ;)  I'm adding to list to check ST v4 to see how it works.
>>>
>>> Ter
>