[antlr-interest] Building syntax highlighters with ANTLR

Sam Harwell sharwell at pixelminegames.com
Thu Apr 9 19:25:50 PDT 2009


I'm working to make it more generic (easier to apply to new languages).
The method has great potential, but the nuances for a triple-stacked
lexer like the one used for syntax highlighting in the following image
are harder to handle than it could be. That said, let's talk for a bit
about where this is headed. What *should* StringTemplate highlighting
look like? See the screenshot below for its current state.

 

The lexers in use are:

*         Group lexer

*         Template filter lexer that breaks up BIGSTRING

*         Template Keywords filter lexer that further breaks up a few of
the template tokens to identify keywords

 

I need to try out a few options - my former roommate and I discussed a
possible way to handle this that I'll try to implement tomorrow so I can
send another screenshot.

 

I know I promised implementation details, and I won't let y'all down,
but I need to do a few things first:

*         Identify common usages/demands of stacked lexers in syntax
highlighting

*         Design and implement a core API to hide some of the details,
since it will greatly improve performance, ease of use, and guarantee
IDE stability should any the lexers get into a bad state. I have some
ideas here that I think are going to be pretty cool.

 

Sam Harwell

 

 

 

 

-----Original Message-----
From: Terence Parr [mailto:parrt at cs.usfca.edu] 
Sent: Thursday, April 09, 2009 12:56 PM
To: Sam Harwell
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Building syntax highlighters with ANTLR

 

Let's hear the details, Sam! :)

Ter

On Apr 9, 2009, at 1:38 AM, Sam Harwell wrote:

 

> I've made a few posts about this in the past, and it looks like  

> another one is on the way. I designed a new, much easier, robust,  

> and general way to make a syntax highlighter from a grammar, and it  

> even allows clean "stacking" of lexers. As a quick example, my  

> primary grammar recognizes a block comment, but it doesn't recognize  

> Doxygen syntax within that comment. By stacking a Doxygen-style  

> fragment lexer in the colorizer, and telling it to post-process any  

> token of primary type "DOC_COMMENT", I ended up with this.

> 

> I'd give the gory details now, but it's 3:30am - just had to share  

> because figuring this out is what kept me up! I can't wait to see  

> multi-level lexer stacking for my StringTemplate highlighter. J

> 

> <image001.png>

> 

> List: http://www.antlr.org/mailman/listinfo/antlr-interest

> Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090409/6450573e/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 41969 bytes
Desc: image001.png
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20090409/6450573e/attachment.png 


More information about the antlr-interest mailing list