[antlr-interest] More information about token location

forumer at smartmobili.com forumer at smartmobili.com
Mon Jul 30 04:27:06 PDT 2012


Hi,

I am using antlr 3.4.1 with the c# backend to parse some java code and 
I would like to get more information when
using the lexer. For the moment the IToken interface is defined like 
that :

public interface IToken
     {
         int Channel { get; set; }
         int CharPositionInLine { get; set; }
         ICharStream InputStream { get; set; }
         int Line { get; set; }
         int StartIndex { get; set; }
         int StopIndex { get; set; }
         string Text { get; set; }
         int TokenIndex { get; set; }
         int Type { get; set; }
     }

But when I have a token I would like to know where is the line where 
the token starts but also where it ends (this
would be very handy for multiline comments for instance).
Would it be very difficult to modify the c# backend to get the 
following interface :



public interface IToken
     {
         int Channel { get; set; }
         int StartCharPositionInLine { get; set; }
         int EndCharPositionInLine { get; set; }
         ICharStream InputStream { get; set; }
         int StartLine { get; set; }
         int StopLine { get; set; }
         int StartIndex { get; set; }
         int StopIndex { get; set; }
         string Text { get; set; }
         int TokenIndex { get; set; }
         int Type { get; set; }
     }

Of course before using the lexer I could parse my entire buffer and 
build a map between index range and line number
but I find it stupid to do it because it could be done during the 
parsing.



More information about the antlr-interest mailing list