[antlr-interest] Antlr3.4 Python bugs, workarounds

[Benjamin Niemann]

On Fri, Oct 7, 2011 at 11:17 PM, Benjamin S Wolf <jokeserver at gmail.com> wrote:
> I've discovered two bugs in working with the Python target to generate lexers.
>
> The first is in that when supplying k, the generated code for special
> state transitions is invalid: the "elif" keyword is split across lines
> as "el\nif", so the python interpreter crashes upon reading "el".

That's been reported before, but I have problem reproducing it. Are
you using antlr-3.4-complete.jar (I can only see this problem with
that build) or did you build it yourself from antlr-3.4.tar.gz (or
something else completely)?

> I've
> been working around this in vim by running the following command in
> the generated file:
>
> :%s/\( \+\)el\n\1/\1el/
>
> (Replaces "el\n" preceded and followed with n spaces by n spaces
> followed by "el".)
>
> The second is in the Python antlr3 library. Calling getTokens() on a
> CommonTokenStream will return all but the last token. This is because
> the slice notation [start:stop] is inclusive on the left and exclusive
> on the right, but stop is set to len(self.tokens) - 1.
> http://www.antlr.org/api/Python/antlr3_8py-source.html#l01733
>
> This can be fixed by finding the following lines in getTokens() (in
> antlr3/streams.py):
>
> if stop is None or stop >= len(self.tokens):
>    stop = len(self.tokens) - 1
>
> and changing them to
>
> if stop is None or stop > len(self.tokens):
>    stop = len(self.tokens)
>
> or it can be worked around by using the tokens attribute directly.

Will have a look at that, thanks for reporting.

-Ben


>
> --Ben
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>