[antlr-interest] ANTLR 2.7.5 C# runtime in a multi-threaded environment hangs

[Jim Crozman]

We use ANTLR to parse our query language in our data server. This is a
multi user system so we service many simultaneous requests. It is a .NET
product so we use C# as the language. We have been using 2.7.2.1 for
quite a while and I finally got around to moving up to 2.7.5 a few weeks
ago.

Yesterday we were running a stress test and we got hung in an infinite
loop in ASTPair.advanceChildToEnd().
The problem was the NextSibling chain wrapped on itself. When exploring
how this happened we discovered that some code had been added to ASTPair
since the 2.7.2.1. There is a new static Queue in the class and two
static methods GetInstance and PutInstance. The parser generated code
now calls GetInstance to create and ASTPair and then calls PutInstance
when it is done with it. I am assuming this was done to recycle the
ASTPairs and avoid new and garbage collect.

The ASTPairs are kept in a System.Collections.Queue and reused. The
Queue class is only thread safe if it is "public static" and then only
if you are careful what you do with it. Thus the parser is not thread
safe. Our many threads all parsing at the same time must have gotten the
same node and got all messed up in the references.

Why not just new an ASTPair when required and let the garbage collector
clean it up? This is what 2.7.2.1 did. It costs just about nothing to do
a "new" in a garbage collected environment (just move the heap pointer)
and there is no delete. The only impact is the garbage collect. Rapidly
turning over objects get cleanned up very efficiently in the
generational .NET garbage collector. Probably less cycles than all the
code to enqueue and dequeue the nodes especially if you need critical
sections to make it thread safe. A thread static for the queue would
work but that is slow too.

We changed GetInstance to just always do a "new ASTPair" and stubbed the
PutInstance. Everything seems to work great then. We did a quick search
for other static variables and there didn't seem to be any other ones of
concern.

Another option that I tried was to eliminate the Get & Put Instance and
change the ASTPair from a class to a struct. This way there is no new on
the heap at all. It is a value class on the stack. There are a few other
chages required to make this work such as passing the currentAST nodes
(ASTPair objects) by ref to addASTChild and makeASTRoot in the
ASTFactory. There was also a virtual copy method that didn't seem to be
needed.

Can the next patch release can get rid of this problem? If not please
explain the rationale behind this change and what I can do to use ANTLR
in a multithreaded environment (many simultaneous parsers running). I
hate having a modified version.

Thanks, ANTLR is great!

Jim Crozman
Senior Software Architect
webPLAN inc.