[antlr-interest] SUCCESS! (mostly) detecting transitions in stanza-based files

[Chris Black]

Just a follow up to let everyone know I've resolved the unexpected 
token: null problem. Ric noted that I wasn't handling/matching EOFs in 
my token filter (StanzaParser). I changed to using an eol rule and the 
spurrious errors from the downline parser went away. I am still not 
getting any prints from the catchall rule where I return a self-built 
EOF token, but I think this is because the downstream parser must stop 
as soon as a nextToken call gives and EOF (which came from a matching 
rule). I also did a bit of cleaning up and extra prints. I'd like to 
restate my call for a usable TokenQueue in the default ANTLR package 
since I've found this token filtering approach to be quite helpful. See 
http://www.codetransform.com/filterexample.html for Monty's article that 
I based my filter on.

Thanks again to all!
Chris

Here is what I have now:
header {
   package mypackage;
   import antlr.*;
}

class StanzaParser extends Parser;
options {
   importVocab=CSV;
   k=2;
}

tokens {
   STANZASEPARATOR;
}

{
   MyTokenQueue queue = new MyTokenQueue(8);
     public void consume() {
       try {
           queue.append(LT(1));
       } catch(TokenStreamException e) {
           System.err.println("error in consume");
           System.err.println(e);
           e.printStackTrace();
       }
     super.consume();
   }
     public Token nextToken() throws TokenStreamException {
       Token ret;
       if(queue.length() <= 0) {
           try {
               line();
           } catch(RecognitionException e) {
               System.err.println("recog exception in nextToken");
               System.err.println(e);
               e.printStackTrace();
           }
           catch(TokenStreamException e) {
               System.err.println("tokenstream exception in nextToken");
               System.err.println(e);
               e.printStackTrace();
           }
       }
       if(queue.length() > 0) {
           ret = queue.elementAt(0);
           queue.removeFirst();
           return ret;
       }
       System.out.println("no more queue, returning EOF");
       return new Token(Token.EOF_TYPE,"end of file");
   }
}

line: (emptyLine | contentLine | delim1stLine) ;

emptyLine: eol ;

delim1stLine: DELIM (FIELD | DELIM)+ eol ;

contentLine: firstTok:FIELD
   {
       String firstText = firstTok.getText();
       if(firstText.startsWith("Data Type") || 
firstText.startsWith("DataType")
           || firstText.equals("Count") || firstText.equals("Result")) {
           queue.append(new Token(STANZASEPARATOR,"stanza sep"));
       }
   }
   (FIELD | DELIM)* eol
   ;
  eol: (NEWLINE | EOF) ;