[antlr-interest] Infinite lexer exception allocation loop in Ctarget C parser, antlr3.0

Hardy, Stephen Stephen_Hardy at rabbit.com
Tue Jul 3 09:29:04 PDT 2007


Jim,

thanks for that.  Yeah, I already added the \x support, but it sowed the
seeds of doubt that some other user syntax or lexical error may cause
the same problem.  I use Fedore core 3, and it seemed to recover by
killing the task once it asked for all 2Gbytes, but I don't know if XP
is equally robust.

BTW, really nice job on the C target.  Much better documentation than
most open source projects.  I'm really looking forward to the C++
target, if only to eliminate the explicit passing of the 'this' pointer.

Just a few points: 

1) I basically had little trouble getting it to work.  The only real
problem was that the sources don't come with a configure script already
built, so one has to use autoconf.  To cut a long story short, I
eventually gave up on the GNU tools because makefile.in wasn't available
so I couldn't successfully run automake.  It turned out to be easy to
create a config.h file from scratch, so that's what I ended up doing.
Nevertheless, you might want to add a working configure script and
makefile.am's as is done with most source distros, to avoid
inconvenience to those with back-level autoconf tools.  (Admittedly I'm
all at sea with autoconf tools, so it's probably my stupidity).

2) Before I realized you had already ported Terrence's C grammar/Java
target to a C target, I was attempting to do so myself.  While doing
this, I found out that the @header stuff is put into both the .c and the
.h files.  I really wanted to put the @header stuff in just the .h file
(is that unreasonable?) so I decided to tweak the string template file.
Lo and behold, the changes to the string templates were not noticed by
the antlr tool.  What am I missing?  I thought it was just a matter of
changing 

$(ANTLR_HOME)/src/org/antlr/codegen/templates/C/C.stg

but it obviously isn't that simple...  Surely I don't have to rebuild
antlr - that would mean I'd have to get a java compiler - argh!

Regards,
SJH


> -----Original Message-----
> From: Jim Idle [mailto:jimi at temporal-wave.com] 
> Sent: Monday, July 02, 2007 7:25 PM
> To: Hardy, Stephen; antlr-interest at antlr.org
> Subject: RE: [antlr-interest] Infinite lexer exception 
> allocation loop in Ctarget C parser, antlr3.0
> 
> I t sounds like an issue with the grammar that exposes an issue in the
> runtime, though something tells me that this may be related to an
> extremely recent change to deal with something related. 
> 
> Thanks for pointing this out - I will endeavor to fix this tomorrow or
> at least provide an answer/work around (which might be to allow \x
> sequences ;-).
> 
> Jim
> 
> > -----Original Message-----
> > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > bounces at antlr.org] On Behalf Of Hardy, Stephen
> > Sent: Monday, July 02, 2007 1:31 PM
> > To: antlr-interest at antlr.org
> > Subject: [antlr-interest] Infinite lexer exception 
> allocation loop in
> > Ctarget C parser, antlr3.0
> > 
> > Hi all,
> > I'm tasked with some C-to-C translation, and have been 
> using the ANSI
> C
> > grammar with the C target as a starting point, with antlr3.0.
> > 
> > The grammar inadvertently omits the possibility of using \x (hex)
> > escapes in a literal string, and this causes an infinite memory
> > allocation loop when the parser is run against tokens such as
> > "\x00\x00".  The offending sequence of code is in mSTRING_LITERAL(),
> > which calls mEscapeSequence(), which in turn allocates an exception
> > struct (CONSTRUCTEX()) when it fails to understand the \x.
> > Unfortunately, the calling code is a for(;;) loop which does not
> > advance
> > the token stream, hence the lexer will allocate forever.
> > 
> > Sorry I'm really new to this, so it may be my fault, but it 
> looks like
> > it may be a C target problem.  Didn't see any similar problem
> mentioned
> > in the most recent archive.
> > 
> > Regards,
> > SJH
> 


More information about the antlr-interest mailing list