[antlr-interest] tab handling in ANTLR

Matthew Ford Matthew.Ford at forward.com.au
Sat Dec 22 18:45:39 PST 2001


At the very least this code should be  in the comments/docs so people know
what to do.
matthew
----- Original Message -----
From: "Terence Parr" <parrt at jguru.com>
To: "ANTLR mailinglist" <antlr-interest at yahoogroups.com>
Sent: Sunday, December 23, 2001 12:32 PM
Subject: [antlr-interest] tab handling in ANTLR


> Folks,
>
> Currently, you have to implement method tab() in your lexer to get it to
> track tabs as anything other than 1 char.  I did this for flexibility.
> You might want tabs to count in some weird way.  The common case isn't
> handled well though where you want it to count tabs as 4 or 8 or
> whatever columns.  Here is what I said in the 2.7.1 release notes:
>
> >   added column tracking support; tabs are counted as 1 unless you
> > override
> >   tab().  Called from consume(); bumps by one by default.  Overhead is
> >   minimal; only called on tabs.  extra increment for all consume()s now
> >   extra int in CommonToken now.
> >
> >     /** advance the current column number by an appropriate amount.
> >      *  If you do not override this to specify how much to jump for
> >      *  a tab, then tabs are counted as one char.  This method is
> >      *  called from consume().
> >      */
> >     public void tab() {
> > // update inputState.column as function of
> > // inputState.column and tab stops.
> > // For example, if tab stops are columns 1 and 5 etc...
> > // and column is 3, then add 2 to column.
> > inputState.column++;
> >     }
> >
> >     added CharScanner.setColumn
>
> So, the question is: "should I add typical case tab handling"?  I.e.,
> should I add:
>
> protected int tabsize = 8;
>
> public void tab() {
> int c = getColumn();
> int nc = ( ((c-1)/tabsize) + 1) * tabsize + 1;      //
> calculate tab stop
> setColumn( nc );
> }
>
> public int setTabsize( int size ) {
> int oldsize = tabsize;
>   tabsize = size;
> return oldsize;
> }
>
> to the default Lexer stuff?  Speak now or forever hold your peace as
> they say...I'm rapidly trying to get thru my "fix it" list for 2.7.2. :)
>
> Ter
> --
> Chief Scientist & Co-founder, http://www.jguru.com
> Creator, ANTLR Parser Generator: http://www.antlr.org
>
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list