[antlr-interest] vocabulary (token) import and export

[bob mcwhirter]

If there's a grammar that includes all tokens (in this case,
possibly your tree-walker), let that one be the exporter.

Though, I think the typical case is that the parser includes
all relevant tokens from the lexer, no?  You seem to indicate
that's not true, which confuses me, personally.

I find that my parser has it's own tokens, uses the 'public' 
lexer tokens, and constructs the tree using possibly more tree
tokens.

I don't see how a tree-walker can deal with lexer-based tokens
that the parser knows nothing about.  How'd the tree get a lexer
token without the parser seeing it?

Anyhow, there's really no distinction between parser, lexer and
tree-walker tokens.  The whole point is of course to unify them
into a single token set, and each parser may only use a subset
of the entire set.

So, if by some chance, there is no lexer/parser/tree-walker that
contains all tokens, simply add them in the tokens{} block, and
consider that file to be the master which does the exportVocab
and is antlr'd first.  All others importVocab, and are compile
subsequently.

btw, I've never worked with antlr/C++, so things may be different
from my java-based world view.

	-bob



On Wed, 22 May 2002, jsrs701 wrote:

> Hi gang,
> 
> I am confused.  I recently decided to bust up my mammoth .g file into 
> three different files, one each for lexer, parser, and tree walker.
> 
> Well, I'm running into vocab hell from this.
> 
> What I want to do is have the lexer define some tokens that it uses 
> and exports.
> 
> Then I want to have the parser define some tokens that IT uses and 
> exports.
> 
> Then I want the tree walker to use all of those.
> 
> Unfortunately, it looks like by the time the parser is compiled, it 
> has only exported the tokens that it uses, and left out all the 
> tokens that only the lexer uses.  This is unfortunate, since then all 
> the C++ files are compiled, and the Lexer.cpp file can't find a bunch 
> of symbols that it needs!
> 
> Ack!
> 
> So anyone else have this problem?  What's the solution?  By the way, 
> take it as a given that I cannot compile the C++ files right after 
> compiling the .g file.
> 
> Thanks,
> JSRS
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 
> 
> 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/