[antlr-interest] "Tokens"

[Gavin Lambert]

At 04:49 19/11/2007, Steve Bennett wrote:
 >Incidentally, is there a way to visualise the tokenisation in
 >ANTLRWorks? I seem to have so many problems with tokens, it 
would
 >be great to just see the definitive list of how everything has
 >been broken up. I guess it shouldn't be too hard for me to just
 >add a dump from the TokenStream in my java test rig, too...

Sadly not.  I've often wished for this as well, but for now the 
best approach is to write a simple test harness in your target 
language that takes a given input file, runs it through the lexer, 
and spits out the list of tokens thus generated.

My normal strategy is to concentrate on the lexing part first 
(especially since ANTLR has some weird quirks in that area), and 
only start working on the parser once I'm certain that the lexer 
is doing the right things.

It's also a good idea to hook this in to a test harness so that 
you can write unit tests that guarantee your token output for 
given input text; this way you won't accidentally break things 
later on.