[antlr-interest] Antlr3.4 Python bugs, workarounds

[Benjamin S Wolf]

For reference, I believe I've found the culprit code. I was browsing
the antlr network on github and I came across this commit:
https://github.com/vslavik/antlr/commit/bbf349ab7d7ccb468604f95b28b08eefd694206f

ie. the code probably looked like:

<dfa.specialStateSTs:{state | \r\n
if s == <i0>: <! compressed special state numbers 0..n-1 !>
    <state>}; separator="\nel">

which caused a newline to be output before "if",
and I changed it to

<dfa.specialStateSTs:{state | \n
if s == <i0>: <! compressed special state numbers 0..n-1 !>
    <state>}; separator="\nel">

which fixed the issue for me,
and vslavik changed it to

<dfa.specialStateSTs:{state | if s == <i0>: <! compressed special
state numbers 0..n-1 !>
    <state>}; separator="\nel">

which apparently fixed the issue for him.

On Wed, Dec 28, 2011 at 2:47 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>
> On Dec 27, 2011, at 5:08 PM, Benjamin Niemann wrote:
>>> Hi Benjamin,
>>>
>>> I was meddling around with the stg templates for Python in trying to
>>> fix some other bugs I reported in another thread, and after updating
>>> the files in antlr-3.4-complete.jar this problem was alleviated.
>>>
>>> I narrowed down the diff and discovered that the stg templates in the
>>> original jar all had DOS line endings (that is, \r\n instead of just
>>> \n), and that removing all the carriage returns in
>>> org/antlr/codegen/templates/Python/Python.stg solved the issue of the
>>> elif being split across a newline.
>>>
>>> That certainly explains why it only showed in antlr-3.4-complete.jar,
>>> since the templates included with antlr-3.4.tar.gz did not have the
>>> carriage returns. :)
>>
>> Good catch, thanks a lot for figuring that out.
>> That seems like a bug in stringtemplate to me - I thought it was
>> smarter about dealing with line endings.
>>
>> Ter:
>> Was the jar built on a windows box?
>
> nope. mac os x.
>
>> I assume perforce adds the CRLFs
>> when checking out the files under windows - the files are stored as
>> "text", i.e. line endings are converted to the native system.
>
> that's correct i think.
>
>> Unless ST can be taught to deal with that properly, we could store the
>> templates as binary in the repository - but that could be messy when
>> someone actually wants to edit them under windows and it's hard to
>> notice when CRs creep back in.
>> Or avoid building jars on windows ;)
>> This probably affects other targets as well, but those are probably
>> less picky about some extra whitespace here and there. Could lead to
>> some obscure bugs though.
>
> ST emits proper newlines per platform and should read \r\n just like \n.  BUT…apparently not ;)  I'm adding to list to check ST v4 to see how it works.
>
> Ter