[antlr-interest] How to do a specific operation once on every token in the _parser_?

[Jan Harder]

Hello,

I wrote a parser for visual basic using ANTLR 2.7.6 and I want to 
compute the halstead metric for parts of the code. Basically, this means 
that I need to classify all tokens in two groups (identifiers, literals 
vs. everything else) and count the occurrences of tokens with the same 
text. The classification and the counting is very easy to do, but my 
problem is that I am not sure _where_ to do this. The main problem is 
that it must be ensured that each token is counted only once. This 
metric is of interest for each function in the code, so I need to this 
in the parser, where I know where functions start and where they end. 
Unfortunately I can't do the measurement on the AST tree because it does 
not contain all tokens. Things like brackets are not added to the tree. 
It is not an option to change the tree.

Of course I don't want to insert an counting action for every token that 
can appear in a function. What I need is a place where each token comes 
along exactly once and can be counted easily. I thought of using the 
parsers consume() function. My idea was to overwirte it and add the 
classification counting code for the metric there by inspecting LT(1) 
after the consume-implementation from LLKParser was called. I guess this 
way ensures that every token is inspected exactly once, but I'm not sure 
about that. Is it safe to do it that way or is the consume method 
affected be the rewind operation on the token stream so that the same 
token can be handeled by consume() more than once?

Thanks for your help.

Jan Harder

PS: I generate C++ code