[antlr-interest] Island Parsing - a different way, seems to work

Mark Mandel mark.mandel at gmail.com
Mon Jun 18 00:23:18 PDT 2007


Hey all,

I've been playing around with Island Parsing, and I think I've come up
with a much simpler way of doing it other than the one that is in the
wiki
(http://www.antlr.org/wiki/display/ANTLR3/Island+Grammars+Under+Parser+Control)

I wanted to run it past you, in case there is something that I have missed.

I'll chop out a lot of the extraneous code around what I'm doing, so
hopefully I don't break it in the process.

I needed to be able to do some island parsing, simply where I just had
a string to parse, and  wanted to be able to insert the island grammar
tree into my current AST.

The code ended up looking pretty much like this:

startTag
	:
	(
	sto=START_TAG_OPEN stc=START_TAG_CLOSE	tc=tagContent
		(
		-> ^(CFTAG[$sto] START_TAG_CLOSE
						{
							parseScript(stc, tc)
						}
						  tagContent)
		)
	)
	;

The only issue I had here, was that I couldn't use $tc, because for
some reason ANTLR couldn't recognise it - so I just set it explicitly,
and everything seemed happy.

>From there, I was able to write my own parseScript function that
returns a CommonTree,

protected Tree parseCFScript(Token start, ParserRuleReturnScope stop)
	{
		
org.antlr.runtime.BitSet bit = new org.antlr.runtime.BitSet();
		bit.add(OTHER);
		List otherTokens =
((CommonTokenStream)input).getTokens(start.getTokenIndex(),
stop.stop.getTokenIndex(), bit);
		
		StringBuffer buffer = new StringBuffer();
		
		for(Object t : otherTokens)
		{
			buffer.append(((Token)t).getText());
		}

		CharStream input = new ANTLRNoCaseStringStream(buffer.toString());
        CFScriptLexer lexer = new CFScriptLexer(input);

        CommonTokenStream tokens = new CommonTokenStream(lexer);
        CFScriptParser parser = new CFScriptParser(tokens);

        try
        {
        	CFScriptParser.script_return root = parser.script();
        	Tree ast = (Tree)root.getTree();
        	return ast;
        }
        catch(RecognitionException exc)
        {
        	ErrorEvent event = new ErrorEvent(exc, "CFScript Error");
        	getObservable().notifyObservers(event);
        }
		
		return null;
	}	


-- 
E: mark.mandel at gmail.com
W: www.compoundtheory.com


More information about the antlr-interest mailing list