[antlr-interest] Novice-Question: Understanding $setType; how to use
Andre Moll
amoll-mailer at libp.com
Tue Dec 18 00:10:20 PST 2001
I have difficults to understand how to change the TokenType within a rule.
If I do this the Parser never recognize it. I added a small example which
interprets HTML files and a small extract of the debug output.
The Lexer only recognize the '<title>' TAG. Any other TAG is going to the
FILTER Rule and I want to set the TokenType to GEN_OTAG or GEN_CTAG with
$setType(GEN_OTAG).
The Filter is working fine so far as I interpret my output, but the Parser
only recognize the TITLE and TEXT Tokens which aren't defined in the FILTER.
Seems I have not understood how to change the TokenType in general. Or do I
have to use $setToken for changing the TokenType if I want to have the
Parser recognize it?
I hope somebody can help.
Regradz Andre
---------
[2001-12-18 16:04:10:409]: lexer debug: GEN_OTAG
[2001-12-18 16:04:10:415]: lexer debug: TEXT
[2001-12-18 16:04:10:422]: parser debug: TEXT
[2001-12-18 16:04:10:427]: lexer debug: GEN_OTAG
[2001-12-18 16:04:10:432]: lexer debug: TEXT
[2001-12-18 16:04:10:437]: parser debug: TEXT
[2001-12-18 16:04:10:443]: lexer debug: GEN_OTAG
[2001-12-18 16:04:10:448]: lexer debug: TEXT
[2001-12-18 16:04:10:452]: parser debug: TEXT
[2001-12-18 16:04:10:458]: lexer debug: OTITLE
[2001-12-18 16:04:10:462]: parser debug: OTITLE
[2001-12-18 16:04:10:468]: lexer debug: TEXT
[2001-12-18 16:04:10:473]: parser debug: TEXT
[2001-12-18 16:04:10:479]: lexer debug: GEN_CTAG
[2001-12-18 16:04:10:484]: lexer debug: TEXT
[2001-12-18 16:04:10:489]: parser debug: TEXT
[2001-12-18 16:04:10:499]: lexer debug: GEN_OTAG
[2001-12-18 16:04:10:504]: lexer debug: TEXT
[2001-12-18 16:04:10:509]: parser debug: TEXT
----------------------------------------------------------------------------
----
---------
header {
#include <iostream>
}
options {
language="Cpp";
}
{
// Global additional Parser Stuff start
void HTMLParser::debug_out(const string & s){
cout << s << endl;
};
// Global additional Parser Stuff end
}
class HTMLParser extends Parser;
options {
exportVocab=HTML;
k = 1;
//defaultErrorHandler = false;
}
{
// internal additional Parser stuff start
protected: virtual void debug_out(const string & s);
//internal additional Parser stuff stop
}
document
:(document_content)+ {debug_out("document");}
;
document_content
:TEXT {debug_out("TEXT");}
|GEN_OTAG {debug_out("GEN_OTAG");}
|GEN_CTAG {debug_out("GEN_CTAG");}
|OTITLE {debug_out("OTITLE");}
;
///////////////////////////////////
/// ///
/// LEXER STARTS HERE ///
/// ///
///////////////////////////////////
{
// Global additional Lexer Stuff start
void HTMLLexer::error_out(const string & s){
std::cerr << s << std::endl;
}
void HTMLLexer::debug_out(const string & s){
std::cout << s << std::endl;
}
// Global additional Lexer Stuff end
}
class HTMLLexer extends Lexer;
options {
k = 4;
exportVocab=HTML;
charVocabulary = '\3'..'\377';
caseSensitive=false;
filter=FILTER;
}
{
// internal additional Lexer stuff start
protected: virtual void error_out(const string & s);
protected: virtual void debug_out(const string & s);
//internal additional Lexer stuff stop
}
OTITLE :"<title>" {debug_out("OTITLE");}
;
TEXT :((~('<')))+
{
debug_out("TEXT");
}
;
protected
FILTER :'<' (~'/') ((~('<'|'>')))* '>'
{
$setType(GEN_OTAG);
debug_out("GEN_OTAG");
}
|"</" ((~('<'|'>')))* '>'
{
$setType(GEN_CTAG);
debug_out("GEN_CTAG");
}
;
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list