[antlr-interest] finish/stop parsing without closing reader

Horst Dehmer horst.dehmer at inode.at
Sat Oct 24 06:07:38 PDT 2009


Hello!

For one special parser rule in my grammar I have to stop parsing and read
the rest of the input from the reader as (compressed) binary data. Is there
any way to instruct the parser to stop reading further tokens when the top
most rule was successfully recognized?

Any help is highly appreciated!

- Horst

More details:  The rule recognizes header information for the then following
binary data:

d_data returns [D_DATA pdu]
@init {
  pdu = null;
}
  : 'D4|' oigId = oig_id '|' syncpointId = syncpoint_id '|'
uncompressedLength = length '|'
    {
      SyncpointDescriptor syncpoint = ...
      pdu = new D_DATA(syncpoint);
    }
  ;

The size of the binary data following the last Œ|¹ can become quite big and
I have to store them as a file to disk.
After the header is recognized the parser returns but the input reader is
closed. Without using EOF in the rule, it seems the additional bytes are
consumed from ANTLRReaderStream:

CharStream charStream = new ANTLRReaderStream(reader);
PduLexer lexer = new PduLexer(charStream);
TokenStream tokenStream = new CommonTokenStream(lexer);
PduParser parser = new PduParser(tokenStream);
...

With a trailing EOF in the rule the parser naturally complains about the
additional information: line 1:36 extraneous input '<binary data>' expecting
EOF
The test case shows proper recognition of the T2/D4 along with syncpoint
token, but reading Œ<binary data>¹ fails due to a closed reader/stream.

@Test
public void parse_whole() throws RecognitionException, IOException {
    final BigDecimal OIG = new BigDecimal("10903008203000000001");
    final String SYNCPOINT = "82737";
    final long LENGTH = 40;
    final String FORMAT = "T2|D4|%20.0f|%s|%d|<binary data>";
    final String MESSAGE = String.format(FORMAT, OIG, SYNCPOINT, LENGTH);

    StringReader reader = new StringReader(MESSAGE);
    T_PDU t_pdu = parserDriver.parse(reader); // OK.
    D_PDU d_pdu = ((T_DATA) t_pdu).getPdu(); // OK.
    D_DATA d_data = (D_DATA) d_pdu; // OK.
    SyncpointDescriptor token = d_data.getToken();
    
    assertEquals(OIG, token.oigId); // OK.
    assertEquals(SYNCPOINT, token.syncpointId); // OK.
    assertEquals(LENGTH, token.uncompressedLength); // OK.

    try {
        // reader should be positioned at the rest of the message, i.e.
Œ<binary data>¹.
        char[] buffer = new char["<binary data>".length()];
        reader.read(buffer);
    }
    catch (IOException exception) {
        // NOT OK:
        // java.io.IOException: Stream closed
    }
}


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091024/10e85b79/attachment.html 


More information about the antlr-interest mailing list