[antlr-interest] Visual Studio syntax highlighting for an Antlrgrammar

Don Caton dcaton at shorelinesoftware.com
Mon Dec 5 11:50:46 PST 2005


Pete:

> It sounds like your approach is to manually handle each 
> problem with custom modifications.  I think this might be 
> difficult in the general case of multiline strings, XML 
> comments, or nested languages (e.g. we have islands of SQL 
> expressions in one of our grammars).  The end result might be 
> a less readable grammar, and a fair amount of work compared 
> to the fully-automatic scenario with Flex.

Well, I was hoping to reuse the existing lexer for our compiler but that
didn't seem practical once I got into it.  Syntax highlighers want to know
about things that the compiler doesn't.  But so far, it has only required
fairly minor changes to our lexer.  This particular language doesn't have
multiline strings so the only token that can span multiple lines is a
multiline comment.  So the only difference between the compiler's lexer and
the syntax highlighter's lexer is its handling of comments.

> The impression I'm getting is that recursive descent lexers 
> are actually inferior with the kind of optimizations required 
> for responsive syntax highlighting.

Not having experience with other lexer/parser generators, I don't know.

> It is aesthetically pleasing that Antlr's lexer and parser 
> share a common algorithm.  However, this experience is 
> building a case that Antlr's approach is less versatile (and 
> possibly slower?) when it comes to the lexer.

I've been able to squeeze a lot of performance out of Antlr by cleaning up
the C++ code so that the compiler can inline most of it.  It made a huge
difference.

Don




More information about the antlr-interest mailing list