[antlr-interest] Dumping out lexer token stream?

[Wincent Colaiuta]

El 23/6/2007, a las 3:41, Cameron Esfahani escribió:

> To help with my debugging, I would like to see the tokenized output  
> from the lexer.  Before the parser gets a chance at, well, parsing it.
>
> I can't seem to find anything in ANTLRWorks which will do this.   
> Does anyone have any suggestions?
>
> Cameron Esfahani
> dirty at apple.com

Normally the lexer is invoked automatically by the parser, which  
repeatedly calls the "next token" method/function. So you can do the  
same and watch the token stream that way. For example, in the C  
target, something like the following (assuming you lexer is in the  
variable "lexer"):

     do
     {
         pANTLR3_COMMON_TOKEN token = lexer->pLexer->tokSource- 
 >nextToken(lexer->pLexer->tokSource);
         if (token == NULL)
             continue;
         ANTLR3_UINT32 type = token->getType(token);
         if (type == ANTLR3_TOKEN_EOF)
             break;

     // ...

     } while (1);

And inside your loop you can look at the matched chars (token->getText 
(token)->chars) or any other attributes that you are interested in.

Not sure if that's exactly what you're looking for, but hope it helps.

Cheers,
Wincent