[antlr-interest] tab handling in ANTLR

[Terence Parr]

Folks,

Currently, you have to implement method tab() in your lexer to get it to 
track tabs as anything other than 1 char.  I did this for flexibility.  
You might want tabs to count in some weird way.  The common case isn't 
handled well though where you want it to count tabs as 4 or 8 or 
whatever columns.  Here is what I said in the 2.7.1 release notes:

>   added column tracking support; tabs are counted as 1 unless you 
> override
>   tab().  Called from consume(); bumps by one by default.  Overhead is
>   minimal; only called on tabs.  extra increment for all consume()s now
>   extra int in CommonToken now.
>
>     /** advance the current column number by an appropriate amount.
>      *  If you do not override this to specify how much to jump for
>      *  a tab, then tabs are counted as one char.  This method is
>      *  called from consume().
>      */
>     public void tab() {
> 	// update inputState.column as function of
> 	// inputState.column and tab stops.
> 	// For example, if tab stops are columns 1 and 5 etc...
> 	// and column is 3, then add 2 to column.
> 	inputState.column++;
>     }
>
>     added CharScanner.setColumn

So, the question is: "should I add typical case tab handling"?  I.e., 
should I add:

	protected int tabsize = 8;

	public void tab() {
		int c = getColumn();
		int nc = ( ((c-1)/tabsize) + 1) * tabsize + 1;      // 
calculate tab stop
		setColumn( nc );
	}

	public int setTabsize( int size ) {
		int oldsize = tabsize;
	  	tabsize = size;
		return oldsize;
	}

to the default Lexer stuff?  Speak now or forever hold your peace as 
they say...I'm rapidly trying to get thru my "fix it" list for 2.7.2. :)

Ter
--
Chief Scientist & Co-founder, http://www.jguru.com
Creator, ANTLR Parser Generator: http://www.antlr.org


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/