[antlr-interest] Dumping out lexer token stream?
Wincent Colaiuta
win at wincent.com
Sat Jun 23 01:28:10 PDT 2007
El 23/6/2007, a las 3:41, Cameron Esfahani escribió:
> To help with my debugging, I would like to see the tokenized output
> from the lexer. Before the parser gets a chance at, well, parsing it.
>
> I can't seem to find anything in ANTLRWorks which will do this.
> Does anyone have any suggestions?
>
> Cameron Esfahani
> dirty at apple.com
Normally the lexer is invoked automatically by the parser, which
repeatedly calls the "next token" method/function. So you can do the
same and watch the token stream that way. For example, in the C
target, something like the following (assuming you lexer is in the
variable "lexer"):
do
{
pANTLR3_COMMON_TOKEN token = lexer->pLexer->tokSource-
>nextToken(lexer->pLexer->tokSource);
if (token == NULL)
continue;
ANTLR3_UINT32 type = token->getType(token);
if (type == ANTLR3_TOKEN_EOF)
break;
// ...
} while (1);
And inside your loop you can look at the matched chars (token->getText
(token)->chars) or any other attributes that you are interested in.
Not sure if that's exactly what you're looking for, but hope it helps.
Cheers,
Wincent
More information about the antlr-interest
mailing list