[antlr-interest] Re: Help with Java grammar

cliftonccraig ccc at icsaward.com
Tue Mar 9 11:18:01 PST 2004


I found some issues with my AppendInputStream. I'm including my
revised code. Once again, can anyone chime in and show me where I'm
re-inventing the wheel? Is there a simpler way to do what I'm doing?

Cliff

    static class AppendInputStream extends FilterInputStream
    {
        InputStream additional;
        private static final int EOF = -1;
        private boolean endOfFirst = false, firstMarked = false,
secondMarked = false;
        private int markCount = 0, markLimit = 0;

        public AppendInputStream(InputStream in, String additional)
        {
            this(in,additional.getBytes());
        }

        public AppendInputStream(InputStream in, byte[] add)
        {
            this(in, new ByteArrayInputStream(add));
        }

        public AppendInputStream(InputStream in, InputStream add)
        {
            super(in);
            additional = add;
        }

        public int read() throws IOException
        {
            int val = EOF;
            if (false==endOfFirst)
            {
                val = super.read();
            }
            if (EOF != val)
            {
                trackMarking(1);
                return val;
            }
            else
            {
                endOfFirst = true;
                conditionallyMark();
                return additional.read();
            }
        }

        public int read(byte b[]) throws IOException
        {
            return read(b, 0, b.length);
        }

        public int read(byte b[], int off, int len) throws IOException
        {
            if(off + len > b.length)
                throw new IndexOutOfBoundsException("Cannot read " +
len + " bytes from offset " + off + " in array of length " + b.length);
            int val = EOF;
            if (false==endOfFirst)
            {
                val = super.read(b, off, len);
            }
            if (EOF != val)
            {
                trackMarking(val);
                if( len > val )
                {
                    endOfFirst = true;
                    val += readAdditional(b, val + off, len - val);
                }
            }
            else
            {
                endOfFirst = true;
                val = readAdditional(b, off, len);
            }
            return val;
        }

        /**
         * Keep track of the # of bytes read into our marking.
         * @param val
         */
        private void trackMarking(int val)
        {
            if(firstMarked) markCount += val;
        }

        private int readAdditional(byte[] b, int off, int len) throws
IOException
        {
            conditionallyMark();
            return additional.read(b, off, len);
        }

        private void conditionallyMark()
        {
            //Just-in-time marking. If the 1st is marked but the 2nd
hasn't been marked
            //and have haven't read past our mark limit we mark it
right before our 1st
            //attempt to read into it.
            if(firstMarked && false==secondMarked && markLimit -
markCount > 0)
            {
                additional.mark(markLimit - markCount);
                secondMarked = true;
            }
        }

        public int available() throws IOException
        {
            if (endOfFirst)
            {
                return additional.available();
            }
            else
            {
                return super.available();
            }
        }

        public void close() throws IOException
        {
            super.close();
            additional.close();
        }

        public synchronized void mark(int readlimit)
        {
            markLimit = readlimit;
            if(endOfFirst && false==firstMarked && false==secondMarked)
            {
                additional.mark(readlimit);
                secondMarked = true;
            }
            else
            {
                super.mark(readlimit);
                firstMarked = true;
            }
        }

        public boolean markSupported()
        {
            return super.markSupported() && additional.markSupported();
        }

        public synchronized void reset() throws IOException
        {
            if(firstMarked)
            {
                super.reset();
                firstMarked = false;
            }
            markCount = 0; markLimit = 0;

            if (secondMarked)
            {
                additional.reset();
            }
        }

        public long skip(long n) throws IOException
        {
            if(endOfFirst)
                return additional.skip(n);
            else
                return super.skip(n);
        }
    }

--- In antlr-interest at yahoogroups.com, "cliftonccraig" <ccc at i...> wrote:
> Thanks Ric,
> 
> Still no luck. I tried your suggestion and while it went through the
> ANTLR generator ok it didnt stop the OutOfMemory error. It appears my
> parser is still getting hung up on the last line being a single
> comment. I'm trying another ugly solution to my problem that is not
> ANTLR related and maybe someone can help me here. I am trying to
> append a newline character to each stream fed into my parser but I'm
> not satisfied with what I have. I created an AppendInputStream that
> extends a filtered stream. I need this to perform as fast as possible
> and I didn't find anything in the JDK that looked like it would help.
> Basically what I want to do is wrap the FileInputStream that I give my
> parser in my custom AppendInputStream and hand it off like so:
> Reader r = new InputStreamReader( new AppendInputStream( new
> FileInputStream(javaFile), "\r\n" ) )
> 
> It sounds like overkill but I just don't see any easier way to do
> this. I thought I saw a way to chain or concatenate to InputStreams
> together as one in some book or article but I can't remember. Please,
> could someone help me with either solution. I feel so stumped.
> 
> Cliff
> 
> The source to my AppendInputStream is below:
>     private static class AppendInputStream extends FilterInputStream
>     {
>         InputStream additional;
>         private static final int EOF = -1;
>         private boolean endOfFirst = false, firstMarked = false,
> secondMarked = false;
>         private int markCount = 0, markLimit = 0;
> 
>         public AppendInputStream(InputStream in, String additional)
>         {
>             this(in,additional.getBytes());
>         }
> 
>         protected AppendInputStream(InputStream in, byte[] add)
>         {
>             this(in, new ByteArrayInputStream(add));
>         }
> 
>         protected AppendInputStream(InputStream in, InputStream add)
>         {
>             super(in);
>             additional = add;
>         }
> 
>         public int read() throws IOException
>         {
>             int val = EOF;
>             if (false==endOfFirst)
>             {
>                 val = super.read();
>             }
>             if (EOF != val)
>             {
>                 trackMarking(1);
>                 return val;
>             }
>             else
>             {
>                 endOfFirst = true;
>                 conditionallyMark();
>                 return additional.read();
>             }
>         }
> 
>         public int read(byte b[]) throws IOException
>         {
>             return read(b, 0, b.length);
>         }
> 
>         public int read(byte b[], int off, int len) throws IOException
>         {
>             if(off + len > b.length)
>                 throw new IndexOutOfBoundsException("Cannot read " +
> len + " bytes from offset " + off + " in array of length " +
> b.length);
>             int val = EOF;
>             if (false==endOfFirst)
>             {
>                 val = super.read(b, off, len);
>                 trackMarking(val);
>             }
>             if (EOF != val)
>             {
>                 if( len > val )
>                 {
>                     endOfFirst = true;
>                     val += readAdditional(b, val + off, len - val);
>                 }
>             }
>             else
>             {
>                 endOfFirst = true;
>                 val = readAdditional(b, 0, b.length);
>             }
>             return val;
>         }
> 
>         /**
>          * Keep track of the # of bytes read into our marking.
>          * @param val
>          */
>         private void trackMarking(int val)
>         {
>             if(firstMarked) markCount += val;
>         }
> 
>         private int readAdditional(byte[] b, int off, int len) throws
> IOException
>         {
>             conditionallyMark();
>             return additional.read(b, off, len);
>         }
> 
>         private void conditionallyMark()
>         {
>             //Just-in-time marking. If the 1st is marked but the 2nd
> hasn't been marked
>             //and have haven't read past our mark limit we mark it
> right before our 1st
>             //attempt to read into it.
>             if(firstMarked && false==secondMarked && markLimit -
> markCount > 0)
>             {
>                 additional.mark(markLimit - markCount);
>                 secondMarked = true;
>             }
>         }
> 
>         public int available() throws IOException
>         {
>             if (endOfFirst)
>             {
>                 return additional.available();
>             }
>             else
>             {
>                 return super.available();
>             }
>         }
> 
>         public void close() throws IOException
>         {
>             super.close();
>             additional.close();
>         }
> 
>         public synchronized void mark(int readlimit)
>         {
>             markLimit = readlimit;
>             if(endOfFirst && false==firstMarked &&
> false==secondMarked)
>             {
>                 additional.mark(readlimit);
>                 secondMarked = true;
>             }
>             else
>             {
>                 super.mark(readlimit);
>                 firstMarked = true;
>             }
>         }
> 
>         public boolean markSupported()
>         {
>             return super.markSupported() && additi
> onal.markSupported();
>         }
> 
>         public synchronized void reset() throws IOException
>         {
>             if(firstMarked)
>             {
>                 super.reset();
>                 firstMarked = false;
>             }
>             markCount = 0; markLimit = 0;
> 
>             if (secondMarked)
>             {
>                 additional.reset();
>             }
>         }
> 
>         public long skip(long n) throws IOException
>         {
>             if(endOfFirst)
>                 return additional.skip(n);
>             else
>                 return super.skip(n);
>         }
>     }
> 
> 
> --- In antlr-interest at yahoogroups.com, Ric Klaren <klaren at c...> wrote:
> > On Tue, Mar 09, 2004 at 02:08:09PM -0000, cliftonccraig wrote:
> > > I just tried this:
> > > SL_COMMENT
> > > 	:	"//"
> > > 		(~('\n'|'\r'))* ('\n'|'\r'('\n')?)
> > > 		{
> > > //*CCC- Allow comments to flow through to the rewrite engine
> > > //		    $setType(Token.SKIP);
> > > 		    newline();
> > > 		}
> > > 		|
> > > 		"//" (~('\n'|'\r'))*
> > > 	;
> > 
> > How about this? 
> > 
> > SL_COMMENT
> >    :  "//" 
> >       ( ~('\n'|'\r') )*                         // not a newline
> part...
> >       ( ('\n'|'\r'('\n')? { newline(); } ) )?   // optional newline
> >    ;
> > 
> > If this gives trouble generate the lexer with -traceLexer and see
> where it
> > gets stuck. (or check with debugger)
> > 
> > There's a few dirty tricks you can do with EOF checks that work by
> checking
> > LA(i) for EOF in the init action of a closure rule, but I don't
> think these
> > should be necessary for this. (unless I'm missing the point
> somewhere)
> > 
> > > And I got an warning saying:
> > > D:\scm\tools\parsers\grammar\ANTLR\java.g:1235: warning:lexical
> > > nondeterminism between alts 1 and 2 of block upon
> > > D:\scm\tools\parsers\grammar\ANTLR\java.g:1235:     k==1:'/'
> > > D:\scm\tools\parsers\grammar\ANTLR\java.g:1235:     k==2:'/'
> > > D:\scm\tools\parsers\grammar\ANTLR\java.g:1235:    
> > > k==3:'\u0003'..'\t','\u000b','\u000c','\u000e'..'\uffff'
> > > D:\scm\tools\parsers\grammar\ANTLR\java.g:1235:    
> > >
> k==4:<end-of-token>,'\u0003'..'\t','\u000b','\u000
> c','\u000e'..'\uffff'
> > 
> > Don't worry too much about warnings like these ;) Read the source
> for what
> > antlr generated for the rule and it often becomes obvious if the
> > parser/lexer will do the right thing. (and it helps in getting a
> feel for
> > things)
> > 
> > Cheers,
> > 
> > Ric
> > -- 
> >
> -----+++++****************************************
> *************+++++++++-------
> >     ---- Ric Klaren ----- j.klaren at u... ----- +31 53 4893722  ----
> >
> -----+++++****************************************
> *************+++++++++-------
> >  Time what is time - I wish I knew how to tell You why - It hurts to
> know -
> >           Aren't we machines - Time what is time - Unlock the door
> >                - And see the truth - Then time is time again
> >                 From: 'Time what is Time' by Blind Guardian



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list