[antlr-interest] Incremental Lexing in an Editor with Stateful Lexers

Martin Probst mail at martin-probst.com
Sun Nov 27 13:18:33 PST 2005


Hi,

I'm currently trying to create a nice GUI for our XQuery enabled XML
database. This of course includes a proper editor for XQuery, and I'm
going to do that in Eclipse. Eclipse standard TextEditors have some
really nifty features about incremental/partial parsing, e.g.
partitioning the document, detecting the 'damage' done to the view and
re-lexing only those parts that are damaged.

So far so good, but my language (XQuery) requires highly stateful
lexing. I have to maintain several different states throughout the
process, every input usually changes state, and to top it off I need a
stack of states.

So my problem is: even if I'm able to detect where exactly someone typed
something, I cannot start lexing for that region only again, as I need
the state stack the Lexer was in just before that region. This would be
fairly complex and brings a lot of overhead in turn (storing the lexing
state stack for each token).

I can only think of two solutions, either mark the places where the
lexer returns to the default state with an empty stack, or copy and
store the whole stack every n tokens. Another solution would be to
re-lex lazily after some 100ms of no keyboard activity, but that seems
less attractive (and is also not so easy, with background threads and
such...).

Has anyone done something like this before? Any experiences/tips?

Martin



More information about the antlr-interest mailing list