[antlr-interest] Can I restart lexing from definite positionindocument?

Sam Harwell sharwell at pixelminegames.com
Fri Apr 17 07:04:12 PDT 2009


You should tokenize on a per-line basis. Never allow a token to span
multiple lines, and never allow lookahead/back to cross a newline
boundary. I've documented this process in my blog followed by a email on
this list earlier this week with an improvement. Here's the original
article:
http://blog.280z28.org/archives/2008/10/21/

Sam

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of P.N.
Sent: Friday, April 17, 2009 8:58 AM
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Can I restart lexing from definite
positionindocument?

Sam Harwell schrieb:
> Is this part of a syntax highlighter?
>
Yes.

> Either way, you can always take an arbitrary token from your original
> token stream that falls before the location in the document where a
> change occurred, and substring your document text at that location.
Then
> create a token stream based on the original tokens up to the break
> followed by the tokens you just got back from the re-lexing of the
> document.
>

Sounds strange, and not useful. My point is, the document content may be

changed in a JEditorPane, and I don't want to do lexing/tokenizing
again.

Peter

> Sam
>
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org
> [mailto:antlr-interest-bounces at antlr.org] On Behalf Of P.N.
> Sent: Friday, April 17, 2009 4:51 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Can I restart lexing from definite position
> indocument?
>
>
> Hello!
>
> Just want to know, if it's possible to restart lexing in a big file.
> Say, e.g. the file might be of 20MB or more (okay, that's not good
> programming style, but's not the question here ;-) ), and I'd change
> sth. at the end of the file - would I need to do lexing from start
> again, or is there a chance to do it only for the last characters?
> Probably using org.antlr.runtime.RecognizerSharedState? How should I
use
> it?
>
> Kind regards
>
> Peter
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>



List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address


More information about the antlr-interest mailing list