[antlr-interest] Problem using code from ANTLR 3 FAQ How do I implement include files?

[Rob Finneran]

Hello fellow ANTLR  fans!

Please forgive my newbie-ness.

The code from the FAQ doesn't seem to work for me:

@lexer::members {
    class SaveStruct {
      SaveStruct(CharStream input){
        this.input = input;
        this.marker = input.mark();
      }
      public CharStream input;
      public int marker;
     }

     Stack<SaveStruct> includes = new Stack<SaveStruct>();

    // We should override this method for handling EOF of included file
     public Token nextToken(){
       Token token = super.nextToken();

       if(token==Token.EOF_TOKEN && !includes.empty()){
        // We've got EOF and have non empty stack.
         SaveStruct ss = includes.pop();
         setCharStream(ss.input);
         input.rewind(ss.marker);
         token = super.nextToken();
       }

      // Skip first token after switching on another input.
       if(((CommonToken)token).getStartIndex() < 0)
         token = super.nextToken();

       return token;
     }
 }

// and lexer rule
 INCLUDE
     : 'include' (WS)? f=STRING {
       String name = f.getText();
       name = name.substring(1,name.length()-1);
       try {
        // save current lexer's state
         SaveStruct ss = new SaveStruct(input);
         includes.push(ss);

        // switch on new input stream
         setCharStream(new ANTLRFileStream(name));
         reset();

       } catch(Exception fnf) { throw new Error("Cannot open file " + name); }
     }
     ;

The lexer code appears to process both the included file, and the main
file, but the parser only seems to build the included file as a tree,
and does not appear to build a tree for the main file.

------------

Below is an email I sent to antlr-interest two days ago with a
slightly more sophisticated mechanism to process entire directory as
indicated by the namespace However, I don't see how this enhancement
has anything to do with the problem. Please forgive if I overlooked
something.

I have modified someone's code example to enhance the C-style include
file example to support reading all include files in a directory in a
similar fashion to C# "using" statements.


The lexer USING rule seems to recursively walk through the files using
a stack mechanism.
The problem is that the parser only SEEMS to see the tokens from the
very first include (or using statement) file. It never SEEMS to fully
processes the remainder of the original file. The lexer rule however
does actually process all of the files. The real problem is that the
AST tree that the PARSER builds only builds the tree for the first
included file.

I tried using input.mark() to save the state info but that version did
not work either.

Could the problem be caused because I need to separate the parser from
the lexer? I'm just  using the default combined lexer/parser but the
code here calls super.nextToken().

Is there a way for me to see the token stream? I have also tried input
= ss.input;

Thanks!

@lexer::members {
       class SaveStruct {
               public CharStream input;
               public String fileName;
       public int tokenStartCharIndex, tokenStartLine,
tokenStartCharPositionInLine;
    }

       Stack<SaveStruct> includes = new Stack<SaveStruct>();
       String currentFile = "default";
       List usingNamespaces = new ArrayList();
       // We should override this method for handling EOF of included file
       public Token nextToken() {
               Token token = super.nextToken();

               if(token==Token.EOF_TOKEN && !includes.empty()) {
                       // We've got EOF and have non empty stack.
                       SaveStruct ss = includes.pop();
                       System.err.println("pop: " + ss.fileName);
                       currentFile = ss.fileName;
                       setCharStream(ss.input);
                       tokenStartCharIndex = ss.tokenStartCharIndex;
           tokenStartLine = ss.tokenStartLine;
           tokenStartCharPositionInLine = ss.tokenStartCharPositionInLine;
                       token = super.nextToken();
               }

               // Skip first token after switching on another input.
               if(((CommonToken)token).getStartIndex() < 0) {
                       token = super.nextToken();
               }
               return token;
       }
}

USING
       : 'using' WS f+=ID ('.' f+=ID)* {
               StringBuffer sb = new StringBuffer();
               sb.append(Test.projectBaseDir);
               java.util.Iterator i = $f.iterator();
               while (i.hasNext()) {
                       Token t = (Token)i.next();
                       sb.append("\\");
                       sb.append(t.getText());
               }
               String name = sb.toString();
               if (!usingNamespaces.contains(name)) {
               usingNamespaces.add(name);
               try {
                       String fullName = null;
                       System.err.println("namespace: " + name);
                       java.io.File namespaceDir = new java.io.File(name);
                       String[] files = namespaceDir.list();
                               for (int j = 0; j < files.length; j++) {
                                       fullName = name + "\\" + files[j];
                                       // save current lexer's state
                                       SaveStruct ss = new SaveStruct();
                                       ss.input = input;
                                       ss.fileName = currentFile;
                                       ss.tokenStartCharIndex =
tokenStartCharIndex;
                                       ss.tokenStartLine = tokenStartLine;
                                       ss.tokenStartCharPositionInLine
= tokenStartCharPositionInLine;
                                       includes.push(ss);
                                       System.err.println("Push: " +
ss.fileName);

                                       // switch on new input stream
                                       setCharStream(new
ANTLRFileStream(fullName));
                                       currentFile = fullName;
                                       System.err.println("Current: "
+ currentFile);
                                       reset();
                               }
      } catch(Exception fnf) { throw new Error("Cannot open file " + name); }
          }
    }
    ;

Thanks!

Rob