[antlr-interest] SUCCESS! (mostly) detecting transitions in
stanza-based files
Chris Black
chris at lotuscat.com
Thu May 12 08:02:45 PDT 2005
Just a follow up to let everyone know I've resolved the unexpected
token: null problem. Ric noted that I wasn't handling/matching EOFs in
my token filter (StanzaParser). I changed to using an eol rule and the
spurrious errors from the downline parser went away. I am still not
getting any prints from the catchall rule where I return a self-built
EOF token, but I think this is because the downstream parser must stop
as soon as a nextToken call gives and EOF (which came from a matching
rule). I also did a bit of cleaning up and extra prints. I'd like to
restate my call for a usable TokenQueue in the default ANTLR package
since I've found this token filtering approach to be quite helpful. See
http://www.codetransform.com/filterexample.html for Monty's article that
I based my filter on.
Thanks again to all!
Chris
Here is what I have now:
header {
package mypackage;
import antlr.*;
}
class StanzaParser extends Parser;
options {
importVocab=CSV;
k=2;
}
tokens {
STANZASEPARATOR;
}
{
MyTokenQueue queue = new MyTokenQueue(8);
public void consume() {
try {
queue.append(LT(1));
} catch(TokenStreamException e) {
System.err.println("error in consume");
System.err.println(e);
e.printStackTrace();
}
super.consume();
}
public Token nextToken() throws TokenStreamException {
Token ret;
if(queue.length() <= 0) {
try {
line();
} catch(RecognitionException e) {
System.err.println("recog exception in nextToken");
System.err.println(e);
e.printStackTrace();
}
catch(TokenStreamException e) {
System.err.println("tokenstream exception in nextToken");
System.err.println(e);
e.printStackTrace();
}
}
if(queue.length() > 0) {
ret = queue.elementAt(0);
queue.removeFirst();
return ret;
}
System.out.println("no more queue, returning EOF");
return new Token(Token.EOF_TYPE,"end of file");
}
}
line: (emptyLine | contentLine | delim1stLine) ;
emptyLine: eol ;
delim1stLine: DELIM (FIELD | DELIM)+ eol ;
contentLine: firstTok:FIELD
{
String firstText = firstTok.getText();
if(firstText.startsWith("Data Type") ||
firstText.startsWith("DataType")
|| firstText.equals("Count") || firstText.equals("Result")) {
queue.append(new Token(STANZASEPARATOR,"stanza sep"));
}
}
(FIELD | DELIM)* eol
;
eol: (NEWLINE | EOF) ;
More information about the antlr-interest
mailing list