bug in 2.7.2a2 C++ lib

[aborschet]

In 2.7.1 you specified that the default value for tab is one.
However the C++ char scanner is instantiated with a tabsize of 8.

Module CharScanner.cpp contains:


CharScanner::CharScanner(InputBuffer& cb)
	: saveConsumedInput(true) //, caseSensitiveLiterals(true)
	, literals(CharScannerLiteralsLess(this))
	, inputState(new LexerInputState(cb))
	, commitToPath(false)
	, tabsize(8)
	, traceDepth(0)
{
...
}

I think "tabsize(8)" should be changed to "tabsize(1)" in all 
constructors.

Btw., to track the position of tokens it would be nice to have 
something like the offset of the token in the parsed buffer. I think 
this could be helpful to solve the problem with lines and columns 
which are dependend to tabs and newlines.

Regards
Alex


> 
> * made fields of CommonToken protected (open to subclasses), added 
col.
>   added column tracking support; tabs are counted as 1 unless you
> override
>   tab().  Called from consume(); bumps by one by default.  Overhead 
is
>   minimal; only called on tabs.  extra increment for all consume()s 
now
>   extra int in CommonToken now.
>  
>     /** advance the current column number by an appropriate amount.
>      *  If you do not override this to specify how much to jump for
>      *  a tab, then tabs are counted as one char.  This method is
>      *  called from consume().
>      */
>     public void tab() {
> 	// update inputState.column as function of
> 	// inputState.column and tab stops.
> 	// For example, if tab stops are columns 1 and 5 etc...
> 	// and column is 3, then add 2 to column.
> 	inputState.column++;
>     }
> 
>     added CharScanner.setColumn
> 
>     remove column info thing at website; say out of date, not needed
> 
> * added check for unterminated rules.  Labels in column 1 result in 
a
> warning.
> 
> * wasn't providing always exactly 4 digits for \u chars in
> JavaCharFormatter.escapeChar
> 
> -- 
> Chief Scientist, http://www.jGuru.com   -- Your Java Portal
> Co-Founder, http://www.NoWebPatents.org -- Stop Patent Stupidity!
> parrt at j...