[antlr-interest] "Tokens"

[Steve Bennett]

Hi all,
  I was wondering if someone could explain exactly what the "tokens"
keyword does. The book (4.4)  doesn't really explain how it differs
from a series of lexer rules.

For example, what's the difference between this:

tokens {
  FOO='FOO';
}

and just this:

FOO: 'FOO';

The book says that this construct "allows you to provide an alias for
a string literal" - but how is that different to just defining a
normal lexer rule?

Also, how exactly do normal string literals operate? Is there a
difference between this:

r: foo '!';

and:

r: foo EXCLAMATION;
EXCLAMATION: '!';

?

Thanks all,
Steve (frustrated by lack of fine detail in the doc atm)