[antlr-interest] newbie question about nondeterminism betweenkeywords and identifiers

[Kurt Rayner]

David,
 
You don't say which version you are using.  I am using 2.7.6 and generating
C++.
 
I have found that I have to declare keywords in the tokens section to avoid
ambiguities with identifiers.  I also set testLiterals to false.  
 
In cases where a keyword is also a valid identifier, I add the keyword to
the grammar's identifier rule.  I would be happy to hear about a simpler
approach.
 
Enjoy!
 
Kurt
 
class SqlLexer extends Lexer;
 
options {
            testLiterals = false;
            k = 12;
            caseSensitive = false;
            caseSensitiveLiterals = false;
            exportVocab = Sql;
            charVocabulary = '\3'..'\377'; 
}
 
tokens {
            A5SQLTOKEN_ALL = "all" ;
            A5SQLTOKEN_AND = "and" ;
            A5SQLTOKEN_APPEND = "append" ;
            A5SQLTOKEN_ANY = "any" ;
            A5SQLTOKEN_AS = "as" ;
 
  _____  

From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of David Guy
Sent: Thursday, February 01, 2007 11:58 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] newbie question about nondeterminism
betweenkeywords and identifiers
 
I have a typical lexer IDENT rule:
IDENT
options {testLiterals=true;}
   : ('_'|'a'..'z')('_'|'a'..'z'|'0'..'9')*
  ;
The language has some built in types. For example (from lexer):
 
TYPE_DATE   :"date";
// declares type
COLON        : ':';
 
In my parser, if I have a rule like:
 
decl:
IDENT COLON TYPE_DATE
;
 
I cannot parse "mydate : date" or "date_foo : date". The first example gets
IDENT than unexpected TYPE_DATE and the second case gets unexpected
TYPE_DATE. 
 
I know this is very basic stuff, but I have looked at sample Java grammars
and don't see anything different and of course in Java you can say
int myint; int int_xxx;
 
_______________________________________________________________________

Notice:  This email message, together with any attachments, may contain

information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated

entities,  that may be confidential,  proprietary,  copyrighted  and/or

legally privileged, and is intended solely for the use of the individual

or entity named in this message. If you are not the intended recipient,

and have received this message in error, please immediately return this

by email and then delete it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20070202/38094815/attachment-0001.html