[antlr-interest] Island grammar in AntlrV3

[David Holroyd]

On Tue, Dec 05, 2006 at 01:31:33PM +0100, Emond Papegaaij wrote:
> On Tuesday 05 December 2006 13:06, David Holroyd wrote:
> > On Mon, Dec 04, 2006 at 11:20:05PM +0000, David Holroyd wrote:
> > > On Sat, Sep 02, 2006 at 11:01:43PM +0000, David Holroyd wrote:
> > > My first idea is to have an action at the point in the outer grammar
> > > where the island grammar's start-marker is recognised, which will...
> > >
> > >  1) take the unprocessed tail of the CommonTokenStream that the
> > >     outer parser currently has as input, and turn back into a string
> > >  2) create a new island lexer/TokenStream that reprocesses the tail
> > >     from 1)
> > >  3) create a parser for the island grammar, and parse the new token
> > >     stream from 2)
> > >  4) get the tail of the island grammar's token stream once the
> > >     end-marker was found, and convert back to the lexer for 'this'
> > >     grammar again
> > >  5) replace the original 'input' reference the parser was using, and get
> > >     going with the outer grammar again

> I've tried something similar, and it does work, but be prepared for some 
> difficulties. For example, the outer grammar MUST not look ahead past (or 
> into) the island grammar. As that will cause trouble with the lexer (as you 
> just mentioned). Interaction between the parser and the lexer is difficult to 
> get right, due to the LA. Writing a lazy token stream is not very difficult 
> with ANTLR v3 (just implement the interface).

I've sort-of got some stuff working along these lines, so I wrote up
some notes in the wiki:

http://www.antlr.org/wiki/display/ANTLR3/Island+Grammars+Under+Parser+Control

Hope they'll be of use to someone else.


ta,
dave

-- 
http://david.holroyd.me.uk/