[antlr-interest] Island grammars

[Martin Probst]

Hi,

> Well there is the example in examples-v3.  Remind me..were we  
> talking about something in addition?

I think we do. What I was referring to was basically stateful lexing  
a la Flex. There are situations where you just can't get around  
it ... I think we came up with a syntax like this:

island Main {
   FOO: "foo" { setState(OP); };
   BAR: "bar" "{"  { pushState(Main); setState(OP); };
}

island OP {
   NAME: "foo" { setState(Main); };
   RCURLY: "}" { popState(); };
}

This would be tremendously helpful for keyword free languages. It  
would need something like multiple Lexers within one Lexer (multiple  
DFAs). Your example with the action that creates a new lexer doesn't  
cut it - you can't create a new lexer for each token you're lexing,  
plus that way there is hardly any syntactical support.

Also your example is not quite valid - it uses a static field  
"nesting" in the simple.g grammar class. This might work for a simple  
compiler that always runs as a single threaded simple executable, but  
this will certainly wreck havoc on anything else (e.g. multi-threaded  
apps, editors with multiple instances, ...).

Please also see the discussion raised by Loring Craymer ("A proposal  
for keywords"). His ideas won't work for standalone Lexers though,  
which is highly desirable for IDEs and editors (so you don't need a  
parser/parse tree for syntax coloring).

Regards,
Martin