[antlr-interest] Help - ANTLR and localized keywords

Jim Idle jimi at temporal-wave.com
Sun Sep 5 14:32:54 PDT 2010

Your options are:

1) Write a custom lexer returning the same token numbers but recognizing
locale for the sequence;
2) Write a separate ANTLR lexer for each locale, sharing the toke
definitions, invoke the lexer that corresponds to the locale;

Either way, you will need to create a list of tokens to import as a vocab to
either the ANTLR lexers or the parser. 

Changing the definition of tokens according to the locale is not a very good
idea I think. If I send you my program and your locale is different, how
does the compiler know what locale the source is in unless it is flagged in
the source? Also, this implies that the language we are using is likely to
be shared amongst different language speakers, but now instead of having at
least a common token set to feast our eyes on, there might be 30; and unless
I know Czech, then I am lost. So, this makes the implied purpose impossible.

Hence the only use is where the code is in fact only in use in a single
language domain (or ones where it is common to speak two, such as say French
and English...) and the intent is to make local programmers more efficient.
Anyway, as you did not invent the language, that is hardly relevant ;-) but
I would write a custom lexer I think (though either is a good solution


> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Milos Silhanek
> Sent: Sunday, September 05, 2010 1:28 PM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Help - ANTLR and localized keywords
> Hi,
> I have implemented Karel language support in NetBeans Platform and I will
> use ANTLR to generate lexer, parser and AST.
> Karel language is described on
> http://karelnb.sourceforge.net/introduction.html
> But Karel's keywords are nationale specific (and case insensitive) - the
> command in English is KROK in Czech. I tried several ways. Using
> generates code into parser but not in lexer. Advices in mailing list
didn't help
> me.
> How can I test keywords in different languages in lexer?
> Can I translate localized keywords into basic (English) in AntlrCharStream
> I suppose to load localized keywords in each lexer/parser. Program will be
> able to translate Karel sources from one language to others.
> Thanks for help.
> Milos Silhanek
> silhanek.m at seznam.cz
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address

More information about the antlr-interest mailing list