[antlr-interest] C style include lexer/parser problem

Rob Finneran robfinneran at gmail.com
Thu Oct 4 14:41:45 PDT 2007


Hello fellow ANTLR  fans!

I have modified someone's code example to enhance the C-style include
file example to support reading all include files in a directory in a
similar fashion to C# "using" statements.
However, this enhancement probably has nothing to do with the problem.

The lexer USING rule seems to recursively walk through the files using
a stack mechanism.
The problem is that the parser only SEEMS to see the tokens from the
very first include (or using statement) file. It never SEEMS to fully
processes the remainder of the original file. The lexer rule however
does actually process all of the files. The real problem is that the
AST tree that the PARSER builds only builds the tree for the first
included file.

I tried using input.mark() to save the state info but that version did
not work either.

Could the problem be caused because I need to separate the parser from
the lexer? I'm just  using the default combined lexer/parser but the
code here calls super.nextToken().

Is there a way for me to see the token stream? I have also tried input
= ss.input;

Thanks!

@lexer::members {
	class SaveStruct {
		public CharStream input;
		public String fileName;
        public int tokenStartCharIndex, tokenStartLine,
tokenStartCharPositionInLine;
     }

	Stack<SaveStruct> includes = new Stack<SaveStruct>();
	String currentFile = "default";
	List usingNamespaces = new ArrayList();
	// We should override this method for handling EOF of included file
	public Token nextToken() {
		Token token = super.nextToken();

		if(token==Token.EOF_TOKEN && !includes.empty()) {
			// We've got EOF and have non empty stack.
			SaveStruct ss = includes.pop();
			System.err.println("pop: " + ss.fileName);
			currentFile = ss.fileName;
			setCharStream(ss.input);
			tokenStartCharIndex = ss.tokenStartCharIndex;
            tokenStartLine = ss.tokenStartLine;
            tokenStartCharPositionInLine = ss.tokenStartCharPositionInLine;
			token = super.nextToken();
		}

		// Skip first token after switching on another input.
		if(((CommonToken)token).getStartIndex() < 0) {
			token = super.nextToken();
		}
		return token;
	}
}

USING
	: 'using' WS f+=ID ('.' f+=ID)* {
		StringBuffer sb = new StringBuffer();
		sb.append(Test.projectBaseDir);
		java.util.Iterator i = $f.iterator();
		while (i.hasNext()) {
			Token t = (Token)i.next();
			sb.append("\\");
			sb.append(t.getText());	
		}
		String name = sb.toString();
		if (!usingNamespaces.contains(name)) {
		usingNamespaces.add(name);
		try {
			String fullName = null;
			System.err.println("namespace: " + name);
			java.io.File namespaceDir = new java.io.File(name);
			String[] files = namespaceDir.list();
				for (int j = 0; j < files.length; j++) {
					fullName = name + "\\" + files[j];
					// save current lexer's state
					SaveStruct ss = new SaveStruct();
					ss.input = input;
					ss.fileName = currentFile;
					ss.tokenStartCharIndex = tokenStartCharIndex;
					ss.tokenStartLine = tokenStartLine;
					ss.tokenStartCharPositionInLine = tokenStartCharPositionInLine; 					
					includes.push(ss);
					System.err.println("Push: " + ss.fileName);

					// switch on new input stream
					setCharStream(new ANTLRFileStream(fullName));
					currentFile = fullName;
					System.err.println("Current: " + currentFile);
					reset();
				}
       } catch(Exception fnf) { throw new Error("Cannot open file " + name); }
	   }
     }
     ;


More information about the antlr-interest mailing list