[antlr-interest] Novice-Question: Understanding $setType; how to use

Andre Moll amoll-mailer at libp.com
Tue Dec 18 00:10:20 PST 2001


I have difficults to understand how to change the TokenType within a rule.

If I do this the Parser never recognize it. I added a small example which
interprets HTML files and a small extract of the debug output.

The Lexer only recognize the '<title>' TAG. Any other TAG is going to the
FILTER Rule and I want to set the TokenType to GEN_OTAG or GEN_CTAG with
$setType(GEN_OTAG).

The Filter is working fine so far as I interpret my output, but the Parser
only recognize the TITLE and TEXT Tokens which aren't defined in the FILTER.

Seems I have not understood how to change the TokenType in general. Or do I
have to use $setToken for changing the TokenType if I want to have the
Parser recognize it?

I hope somebody can help.

Regradz Andre

---------
 [2001-12-18 16:04:10:409]: lexer debug: GEN_OTAG
 [2001-12-18 16:04:10:415]: lexer debug: TEXT
 [2001-12-18 16:04:10:422]: parser debug: TEXT
 [2001-12-18 16:04:10:427]: lexer debug: GEN_OTAG
 [2001-12-18 16:04:10:432]: lexer debug: TEXT
 [2001-12-18 16:04:10:437]: parser debug: TEXT
 [2001-12-18 16:04:10:443]: lexer debug: GEN_OTAG
 [2001-12-18 16:04:10:448]: lexer debug: TEXT
 [2001-12-18 16:04:10:452]: parser debug: TEXT
 [2001-12-18 16:04:10:458]: lexer debug: OTITLE
 [2001-12-18 16:04:10:462]: parser debug: OTITLE
 [2001-12-18 16:04:10:468]: lexer debug: TEXT
 [2001-12-18 16:04:10:473]: parser debug: TEXT
 [2001-12-18 16:04:10:479]: lexer debug: GEN_CTAG
 [2001-12-18 16:04:10:484]: lexer debug: TEXT
 [2001-12-18 16:04:10:489]: parser debug: TEXT
 [2001-12-18 16:04:10:499]: lexer debug: GEN_OTAG
 [2001-12-18 16:04:10:504]: lexer debug: TEXT
 [2001-12-18 16:04:10:509]: parser debug: TEXT
----------------------------------------------------------------------------
----

---------
header {
#include <iostream>
}


options {
	language="Cpp";
}

{
// Global additional Parser Stuff start


void HTMLParser::debug_out(const string & s){
	cout << s << endl;
};

// Global additional Parser Stuff end
}


class HTMLParser extends Parser;
options {
	exportVocab=HTML;
	k = 1;
	//defaultErrorHandler = false;
}

{
// internal additional Parser stuff start

protected: virtual void debug_out(const string & s);

//internal additional Parser stuff stop
}

document
		:(document_content)+ {debug_out("document");}
		;

document_content
		:TEXT		{debug_out("TEXT");}
		|GEN_OTAG	{debug_out("GEN_OTAG");}
		|GEN_CTAG	{debug_out("GEN_CTAG");}
		|OTITLE		{debug_out("OTITLE");}
		;


///////////////////////////////////
///                             ///
///     LEXER STARTS HERE       ///
///                             ///
///////////////////////////////////

{
// Global additional Lexer Stuff start

void HTMLLexer::error_out(const string & s){
	std::cerr << s << std::endl;
}

void HTMLLexer::debug_out(const string & s){
	std::cout << s << std::endl;
}

// Global additional Lexer Stuff end
}

class HTMLLexer extends Lexer;
options {
	k = 4;
	exportVocab=HTML;
	charVocabulary = '\3'..'\377';
	caseSensitive=false;
	filter=FILTER;
}


{
// internal additional Lexer stuff start

protected: virtual void error_out(const string & s);
protected: virtual void debug_out(const string & s);

//internal additional Lexer stuff stop
}

OTITLE	:"<title>" {debug_out("OTITLE");}
		;


TEXT	:((~('<')))+
		{
		debug_out("TEXT");
		}
		;

protected
FILTER	:'<' (~'/') ((~('<'|'>')))* '>'
			{
			$setType(GEN_OTAG);
			debug_out("GEN_OTAG");
			}
		|"</" ((~('<'|'>')))* '>'
			{
			$setType(GEN_CTAG);
			debug_out("GEN_CTAG");
			}
		;



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list