[antlr-interest] Imported parser tokens mismatch

[Luigi Iannone]

Jim,

what about the imaginary tokens I create in the parser? From what you write it seems I have to include them in my hand-crafted .tokens files, but which .token file?

Thanks,

Luigi

On 15 Apr 2010, at 21:51, Jim Idle wrote:

> Your best bet is to pre-create the .tokens file and use the tokenVocab option in the lexer. Just make sure that you keep token numbers contiguous and don't leave gaps. The token numbers will then not only be shared, but known values. You can have more tokens than you actually generate or deal with in a particular parser. I have used this concept across SQL parsers for instance so that a target enging can be origin agnostic.
> 
> Without a vocab option, the lexer will parse and produce the .tokens file. This will be imported by the parsers, which will produce their own .tokens files, which can be imported by tree walkers and so on but when they add tokens (even if they are the same names), then they will not be shared by different parsers.
> 
> You could probably get the outcome you want by carefully controlling the build order and setting the tokenVocab and so on, but it is trivial to maintain your own.
> 
> Jim
> 
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Luigi Iannone
>> Sent: Thursday, April 15, 2010 6:45 AM
>> To: ANTLR
>> Subject: [antlr-interest] Imported parser tokens mismatch
>> 
>> Hi,
>> 
>> I am struggling to re-use tokens across grammars. I have two combined
>> grammars, namely
>> 
>> ManchesterOWLSyntax.g
>> OPPLScript.g
>> 
>> They both import a parser grammar called MOWLParser.g,
>> 
>> ManchesterOWLSyntax.g  imports it directly, whereas OPPLScript.g
>> imports OPPLParser.g which imports MOWLParser.g
>> 
>> I would have thought that all the .tokens generated files would overlap
>> on all the tokens defined inside MOWLParser.g
>> 
>> Maybe I am missing something but if I try to generate the code I end up
>> with different MOWLParser.tokens which results in different token type
>> codes for the same token.
>> 
>> Just as example the token CONJUNCTION gets code 77 when generating
>> OPPLScript.g and 56 when generating ManchesterSyntax.g
>> 
>> I put all the relevant files at
>> http://www.cs.man.ac.uk/~iannonel/antlrexperiments/ (both the loose
>> files and their zipped archive: import.zip)
>> 
>> Thanks in advance for any help,
>> 
>> Luigi
>> 
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
> 
> 
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address