[antlr-interest] Newbie question

[Sinan]

Patrick LeBoutillier wrote:
> 
> Hi all,
> 
> I'm new to ANTLR and to the world of lexers/parsers in general.
> I read up on the documentation and I have a pretty good feel of what
> the lexer does and what the parser does. I have a question though:
> 
> Suppose I'm parsing Java source code. What I want to do is distinguish
> between white space that is used as indentation and whitespace that is in
> between other types of tokens. Indentation should be returned as a special
> token type, say INDENT, while other whitespace can be ignored.
> 
> Is it possible to do this (even though it might not be very useful)?

In other words you want new line followed wirh white space to be treated
as 
special. You just code it as such..... In lexer set a flag when you see
EOL
and in white space test it. Or have EOL return the white space string.
Or you can create a token stream selector class and have a nextToken
method 
that keeps track of all the tokens and automatically changes the first
white space after EOL to an INDENT token.

 One caveat is how to start the whole thing.

If you do

INDENT line EOL
INDENT line EOL


then first line is trouble. Since what you want is

EOL
INDENT line EOL
INDENT line EOL

This is where token stream selector comes in handy.

something like:

import antlr.*;
public class INDENTTokenStreamSelector extends TokenStreamSelector {

	protected boolean firstline;

	public F77TokenStreamSelector() {
		super();
		firstline = true;
	}

	public Token nextToken() throws TokenStreamException {
		for (;;) {
			try {


				if(firstline){
					wasEOL=true;
					return EOL;
				} 
				Token tok = super.nextToken();
				tok=super.nextToken();
				if(wasEOL){
					wasEOL=(tok.getType()==MyParser.EOL);
					if(tok.getType()==MyParser.WS){
							tok= MyParser.INDENT;// you should actually get/set the length
here
					}
				}
				return tok;	
			}
			catch (TokenStreamRetryException r) {
				// just retry "forever"
			}
		}
	}
}

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/