[antlr-interest] Help with TokenStreamHiddenTokenFilter

[Artem Portnoy]

Hi,

I desperately need help with the TokenStreamHiddenTokenFilter. I'm using the java15.g grammar for Antlr v2. I managed to construct a tree using antlr.CommonHiddenStreamToken, but the tree appears to be incomplete. Some of the java comments are missing in the tree that gets constructed. For example, given the following text

/** class comment */
public class Test {
  public String field1; // field comment
}

the first comment is getting set into hiddenBefore for token "public", but the second comment is getting set into hiddenAfter for token ";". The problem is that the semicolon token is not present in the AST tree, so I can't get to the comment. The code I use is listed below.

    JavaLexer lexer = new JavaLexer(new FileInputStream(args[0]));
    // use the special token objects
    lexer.setTokenObjectClass("antlr.CommonHiddenStreamToken");
    // create the stream filter; hide WS and SL_COMMENT
    filter = new HFTokenStreamHiddenTokenFilter(lexer);
    filter.discard(JavaRecognizer.WS);
    filter.hide(JavaRecognizer.SL_COMMENT);
    filter.hide(JavaRecognizer.ML_COMMENT);
    // create the parser
    JavaRecognizer parser = new JavaRecognizer(filter);
    // create trees that copy hidden tokens into tree also
    parser.setASTNodeClass("antlr.CommonASTWithHiddenTokens");
    // Parse the input statements
    parser.compilationUnit();
    AST tree = parser.getAST();

Thank you,

Artem

_________________________________________________________________
Shed those extra pounds with MSN and The Biggest Loser!
http://biggestloser.msn.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080207/199f96dd/attachment.html