[antlr-interest] Rewinding the token stream between templates

[Terence Parr]

On Dec 12, 2006, at 1:15 PM, Kirk Woods wrote:

> I am in the process of creating a tool, with ANTLR 3.0b5 and  
> StringTemplate 3.0, which at this point has two template files and  
> will most likely have four in the future.  After parsing with the  
> first template group, I must rewind the token stream in order to  
> parse with the next template group.
>
>         parser.setTemplateLib(Main.createTemplates);
>         RuleReturnScope ruleScope = parser.program();
>         System.out.println(ruleScope.getTemplate().toString());
>
>         parser.getTokenStream().rewind(0);
>         parser.setTemplateLib(Main.alterTemplates);
>         ruleScope = parser.program();
>         System.out.println(ruleScope.getTemplate().toString());
>
> While I can do this by calling " parser.getTokenStream().rewind 
> (0)", this creates a dependency on knowledge of the internal  
> implementation of the parser.  I would like to suggest that the  
> implementation of the setTemplateLib method be modified to rewind  
> the token stream ( e.g. super.input.rewind(-1)).  Is there a valid  
> use case where one would not want the token stream reset whenever  
> the setTemplateLib method is called?

Hmm...possibly...might set it to default and then within option alter  
it. I would like to separate this functionality.

First, you want seek() not rewind(0)...that arg is a marker not an  
index.  I think that I wanted to use reset, but I know the following  
comment:

	/** reset the parser's state */
	public void reset() {
		// TODO: implement this!
		//following.setSize(0);
	}

[snicker].

  the following does not seem odious to me:

parser.getTokenStream().seek(0);

what do other people think?  I think reset() is better because the  
parser might want to reset error flags and so on.  Ok, I just built  
it BaseRecognizer.reset:

	/** reset the parser's state; subclasses must rewinds the input  
stream */
	public void reset() {
		// wack everything related to error recovery
		_fsp = -1;
		errorRecovery = false;
		lastErrorIndex = -1;
		failed = false;
		// wack everything related to backtracking and memoization
		backtracking = 0;
		for (int i = 0; ruleMemo!=null && i < ruleMemo.length; i++) { //  
wipe cache
			ruleMemo[i] = null;
		}
	}

Then in parser.reset

	public void reset() {
		super.reset(); // reset all recognizer state variables
		if ( input!=null ) {
			input.seek(0); // rewind the input
		}
	}

I'm pushing to the depot, so you should be able to get it from the  
repository browser within a few minutes if you don't want to cut and  
paste.

> p.s. These are great tools.  I have only been using these for two  
> weeks and the amount of code I am now able to generate is tremendous.

Thanks!

Ter