[antlr-interest] Visual Studio syntax highlighting for an Antlrgrammar

Don Caton dcaton at shorelinesoftware.com
Mon Dec 5 09:19:14 PST 2005


Pete:

I'm in the process of doing just that.  You have to take a slightly
different approach to your lexer.  Normally, you lex comments as a single
token which is ultimately discarded (e.g. $setType( Token::Skip )).

In a syntax highlighting parser, you want to parse the comment begin and end
markers separately and don't discard them.  Once you've seen a begin comment
token you need to remember that, which you can do by using the 'state'
parameter to ScanTokenAndProvideInfoAboutIt().  Once you're in a comment
"state", force the color for each successive token to be the comment color
until you see an ending comment token.

In the case of a single line comment, you would use your end-of-statement
token to turn that off.  Which also means that you don't want to discard
line feeds or carriage returns if that's an end-of-statement character in
the language you're parsing.

Your comment begin token can consume everything to the end comment token (or
the end of the text being parsed) but make sure you include all the consumed
text when you calculate the length of the token's text when you pass the
into back to VS in the tokeninfo structure.

I'm still working on this, but it seems to work ok.  I briefly considered
using the Babel interface but it's not well documented, the quality of the
sample code leaves something to be desired, and it seems to have fewer
capabilities than the managed language service interfaces.  And I really
didn't want to spend the time learning flex/bison when I already have an
Antlr grammar for my language.

Don
 

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org 
> [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Pete Gonzalez
> Sent: Monday, December 05, 2005 3:11 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Visual Studio syntax highlighting 
> for an Antlrgrammar
> 
> I am interested in implementing IDE syntax highlighting and 
> underlining (i.e. squiggly underlines marking syntax errors). 
>  The Microsoft Visual Studio SDK (called "VSIP" in previous 
> versions) has a really interesting template called "Babel".  
> Basically if you have a flex/bison grammar, an IDE wizard can 
> be used to automatically generate a DLL add-in implementing 
> syntax highlighting/underlining for that language.  It's 
> completely automated; the wizard even includes a GUI for 
> mapping flex token types to syntax color classes.
> 
> One disadvantage of Babel is that it's old-skool C++.  
> However, Visual Studio 2005 expands this API with managed 
> wrappers enabling an entire language service to be 
> implemented in C#.  (The C# classes lack flex/bison 
> integration however.)  Our Antlr grammars are in C#, so my 
> hope was to accomplish something similar to the Babel package 
> but in C# and with Antlr.
> 
> Currently I'm just focusing on syntax highlighting (which 
> uses the lexer but not the parser).  The IDE text editor is 
> optimized to prevent the entire file from being rescanned 
> whenever something changes.  The required C# interface looks 
> like this:
> 
>    void IScanner.SetSource(string source, int offset);
>    bool IScanner.ScanTokenAndProvideInfoAboutIt(TokenInfo tokenInfo,
>      ref int state);
> 
> The idea is that the editor passes a single line of text to 
> SetSource(), and then calls ScanTokenAndProvideInfoAboutIt() 
> repeatedly to obtain the colored tokens for that line.  In 
> this situation, the only context available to the lexer is a 
> single "state" integer (which for Babel stores flex's 
> "yy_start" global variable).  Unfortunately, since Antlr is a 
> recursive descent design, there isn't an obvious way to 
> restart the lexer e.g. in the middle of a multiline comment.  
> Has anyone else dealt with this problem before?
> 
> More generally, has anyone implemented a Visual Studio 
> language service using an Antlr grammar?  Or, is it difficult 
> to port an Antlr grammar to flex/bison?  :-)
> 
> Cheers,
> -Pete
> 
> ____
> 
> 
> 




More information about the antlr-interest mailing list