[antlr-interest] V3 token vocabulary questions

[Alan Rooks]

I'm moving a parser (only - hand-implemented lexer) grammar from v2 to v3,
and I'm having some problems with token vocabularies.

In the v2 grammar, I used the importVocab=X option to import a token types
file named XTokenTypes.txt.  That file contained token definitions that were
automatically generated during build, and were used by both the lexer and
the parser.  Some of the token definitions used a "paraphrase" string.  Here
are some of the token definitions:
    CONSTANT ("a constant") = 10
    BREAKPOINT              = 11
    PLUS ("+")              = 12

I'm having some trouble getting equivalent functionality in v3.  The book's
description of the tokenVocab option says "The value associated with this
option is the name of a grammar, not the name of a file".  What does that
actually mean?  Since it's a hand-generated lexer, I don't have a grammar
that defines the tokens.  When I say tokenVocab=X, it appears to be just
looking for X.tokens and not X.g or XLexer.java or anything.  That's good,
but can I count on it continuing to work?

The "name" of a grammar is a bit variable anyway: a combined grammar has one
name for two grammars (or there are two names, XLexer and XParser, so X
names neither of them), and is a "lexer grammar" named X, or XLexer (similar
for parser)?

I haven't been able to figure out how to get the paraphrases (e.g. "a
constant, or "+") into the X.tokens file.  Is it possible?  The book doesn't
say anything about what must be there and in what format, when you do
importVocab=X.  The v2 syntax doesn't work.  Has anyone else figured this
out?

Thanks in advance,

Alan

-------------------------
Alan Rooks
AMI Semiconductor Canada
http://www.amis.com/
-------------------------
AMI Semiconductor - "Silicon Solutions for the Real World"
NOTICE: 
This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.