[antlr-interest] Antlr3.4 Python bugs, workarounds

[Benjamin S Wolf]

On Thu, Nov 10, 2011 at 10:05 AM, Rampon Jerome <ramponjerome at yahoo.fr> wrote:
>
> Hi Ben,
>
> I have seen this message below on the web.
> I add similar issues with python target on 3.4 (I did not try with 3.3 for
> ex) for an HDL grammar
> (pretty large)
>
> The elif keyword was split as you mentionned.
> Did you get any final answer ? Could be reproduced ? Fixed ?
> Thanks
> Jerome

Hi Jerome,

I was told this was a known bug but not reproducible, so I provided
the smallest test case I could generate later in the thread. I haven't
received any confirmation yet, so I don't believe it's been fixed.

I can suggest the workaround I provided earlier. If that is too much
to do, note that this only arises when k is set. I managed to refactor
my rules to the point where it was not necessary to lex with the k and
backtrack options set, so I no longer need to do this myself.

--Ben

On Sun, Oct 30, 2011 at 9:48 PM, Benjamin S Wolf <jokeserver at gmail.com> wrote:
> On Fri, Oct 28, 2011 at 11:56 AM, Benjamin Niemann <pink at odahoda.de> wrote:
>> On Fri, Oct 7, 2011 at 11:17 PM, Benjamin S Wolf <jokeserver at gmail.com> wrote:
>>> I've discovered two bugs in working with the Python target to generate lexers.
>>>
>>> The first is in that when supplying k, the generated code for special
>>> state transitions is invalid: the "elif" keyword is split across lines
>>> as "el\nif", so the python interpreter crashes upon reading "el".
>>
>> That's been reported before, but I have problem reproducing it. Are
>> you using antlr-3.4-complete.jar (I can only see this problem with
>> that build) or did you build it yourself from antlr-3.4.tar.gz (or
>> something else completely)?
>
> Using antlr-3.4-complete.jar, yes.
>
> I tried reducing my test case as far as I could, but it won't generate
> the DFA with the special state transitions if I use fewer than 57
> rules. Eventually I tried generating 57 rules to trigger it, and here
> is the (excruciatingly ugly) result. (Note that I have rules that
> start with 'a' and rules that start with 'b'. This triggers the
> "el\nif" case. If I only used 'a' rules, the special state transition
> would only need to check "if s == 0" since no other character is
> valid.)
>