[antlr-interest] Fwd: A couple of questions for lexing strategy.

Gavin Lambert antlr at mirality.co.nz
Tue Mar 4 01:55:14 PST 2008

At 07:34 4/03/2008, Darien Hager wrote:
>At least when using ANTLRworks a file called grammarname.tokens 
>is created with each line containing a token-text to ID mapping. 
>This file is AFAIK only for your convenience, nothing relies on 

It's ANTLR itself that produces it, and it's not quite true that 
nothing relies on it.  If you have a separated lexer and parser 
then the .tokens file is an output of the lexer and an input of 
the parser (via the tokenVocab option), since both lexer and 
parser need to agree on a common set of tokens.  (I think it's 
used internally in a similar fashion even when building a combined 

I believe it can also be an output of the parser and input for a 
tree parser, since the parser can introduce new imaginary tokens 
for use in the AST.  But I've never actually used a tree parser 
myself, so I'm not completely sure about that :)

>As a non-lawyer, I don't think that's a major problem--you don't 
>need to be of legal age to begin making a runtime port or other 
>software project... you just have to be of a legal age or have 
>help before you make any contractual agreement stuff. (You know, 
>like licensing your work or something.)

Well, there *is* licensing involved in contributing to ANTLR; I'm 
simplifying but basically you have to agree that the code you've 
written can be freely distributed and modified by other people. 

More information about the antlr-interest mailing list