[antlr-interest] Building syntax highlighters with ANTLR
Sam Harwell
sharwell at pixelminegames.com
Thu Apr 9 19:25:50 PDT 2009
I'm working to make it more generic (easier to apply to new languages).
The method has great potential, but the nuances for a triple-stacked
lexer like the one used for syntax highlighting in the following image
are harder to handle than it could be. That said, let's talk for a bit
about where this is headed. What *should* StringTemplate highlighting
look like? See the screenshot below for its current state.
The lexers in use are:
* Group lexer
* Template filter lexer that breaks up BIGSTRING
* Template Keywords filter lexer that further breaks up a few of
the template tokens to identify keywords
I need to try out a few options - my former roommate and I discussed a
possible way to handle this that I'll try to implement tomorrow so I can
send another screenshot.
I know I promised implementation details, and I won't let y'all down,
but I need to do a few things first:
* Identify common usages/demands of stacked lexers in syntax
highlighting
* Design and implement a core API to hide some of the details,
since it will greatly improve performance, ease of use, and guarantee
IDE stability should any the lexers get into a bad state. I have some
ideas here that I think are going to be pretty cool.
Sam Harwell
-----Original Message-----
From: Terence Parr [mailto:parrt at cs.usfca.edu]
Sent: Thursday, April 09, 2009 12:56 PM
To: Sam Harwell
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Building syntax highlighters with ANTLR
Let's hear the details, Sam! :)
Ter
On Apr 9, 2009, at 1:38 AM, Sam Harwell wrote:
> I've made a few posts about this in the past, and it looks like
> another one is on the way. I designed a new, much easier, robust,
> and general way to make a syntax highlighter from a grammar, and it
> even allows clean "stacking" of lexers. As a quick example, my
> primary grammar recognizes a block comment, but it doesn't recognize
> Doxygen syntax within that comment. By stacking a Doxygen-style
> fragment lexer in the colorizer, and telling it to post-process any
> token of primary type "DOC_COMMENT", I ended up with this.
>
> I'd give the gory details now, but it's 3:30am - just had to share
> because figuring this out is what kept me up! I can't wait to see
> multi-level lexer stacking for my StringTemplate highlighter. J
>
> <image001.png>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090409/6450573e/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 41969 bytes
Desc: image001.png
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20090409/6450573e/attachment.png
More information about the antlr-interest
mailing list