[antlr-interest] Freemarker grammer w/ JavaScript target

[Gerald Rosenberg]

Interesting.  Not my experience at all.  And, that is even with Eclipse.

On 11/14/2012 1:21 PM, Sam Harwell wrote:
> This naïve approach is not scalable, and will introduce the following limitations:
>
> 1. Typing characters within a large token such as a block comment spanning many lines will be "laggy".
No reason that any particular token would take any more or less time to 
parse -- matching a .* is fast.  As long as the parser is kept warm, the 
incremental time required to parse an in-memory stream is quite small, 
particularly in comparison to keystrokes.  This is for source files of 
10s to 100s of KB.  Perhaps what you are seeing is particular to your IDE.
> 2. As the document grows in size, the editor will progressively slow down.
This is entirely dependent on the IDE implementation.  Highlighting and 
similar features should run in a separate thread and never affect 
keyboard performance.  A common strategy is that if the highlighting 
thread ever falls behind, just discard new highlighting changes.  Even 
in a heavyweight IDE like Eclipse, discards rarely if ever happen and, 
if they do, the effect is imperceptible.

A very useful (and common) strategy is to minimize UI updates.  Diff the 
results of the parse with an image of the UI content and apply only the 
changes.  For SWT and highlighting, the changes are just a series of 
attribute changes, typically just one or two, which are set without 
necessarily invoking a UI update.  Keystrokes do cause UI updates, so 
highlighting is synchronous.

For CodeMirror, it looks like highlighting is implemented by tweaking 
the DOM class of a span.  The time required to do DOM and UI updates 
will likely far outweigh the Antlr parse time.  Run the parser in a 
separate Worker thread and, with some attention to keeping the parser 
warm and managing the application of updates, I think you will be quite 
pleasantly surprised at the performance.

>
> For even medium sized documents, running *just* the lexer on the entire document in response to keystrokes will be noticeably slow. Not all editors treat syntax highlighting as a line-at-a-time problem. Even in those editors I use the line-at-a-time approach to greatly improve performance of my IDEs.
>
> The new lexer modes in ANTLR 4 make it much easier to break up tokens which would otherwise span multiple lines. It can be done in ANTLR 3 by manually tracking the mode information and using an override of nextToken that calls a fragment rule for the current mode instead of always calling mTokens. I haven't used the JavaScript target or worked with CodeMirror so I don't have any examples of how to implement this strategy in that environment.
>
> --
> Sam Harwell
> Owner, Lead Developer
> http://tunnelvisionlabs.com
>
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Gerald Rosenberg
> Sent: Wednesday, November 14, 2012 12:29 PM
> To: rodedb at gmail.com
> Cc: ANTLR-Interest Interest
> Subject: Re: [antlr-interest] Freemarker grammer w/ JavaScript target
>
> Although your editor's approach is line at a time, no reason to try and force Antlr to do the same.  Antlr is more than fast and light enough to re-parse the entire source file between each keystroke and walk the AST to provide highlighting info (and walk the AST to adjust error markers and to collect code assist hints and ... ).
>
> On 11/14/2012 12:26 AM, Roded wrote:
>> Hi list,
>> I'm planning on using ANTLR 3.3's JavaScript target for creating a
>> Freemarker <http://freemarker.sourceforge.net/> parser for the sake of
>> syntax highlighting (and auto-completion at a later stage) in a
>> web-based editor. Considering my lacking experience in ANTLR, I
>> thought I'd ask for any input or tips on accomplishing my goal.
>> As for highlighting, using a generated AST is simple enough, however
>> my editor component's (CodeMirror <http://codemirror.net/>) syntax
>> highlighting mechanism works on individual lines of the source. Is
>> there a way to use the ANTLR parser in an interruptible mode so it
>> could be called for every line separately while retaining its state?
>> and perhaps even remedying the last parsing error in view of the new
>> input (as not all source lines pass parsing by themselves)?
>> Any help and points in the right direction (whether in regards to the
>> JS target or ANTLR in general) would be much appreciated.
>> Many thanks,
>> Roded
>>
>> P.S. anyone encountered a Freemarker grammar?
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>