[antlr-interest] Java1.5 grammar and enum as identifier and not as a keyword

[Jörg Rech]

Hi everyone,
  here is an update and short summary regarding the enum problem.

I traced the lexer and parser and found that the following statement in 
the lexer:
        _ttype = testLiteralsTable(_ttype);
sets _ttype to "109" that translates to LITERAL_enum from the TokenTypes 
(my Java15CodeTokenTypes.java file). This token seems to come from the 
"enumDefinition"-rule from the parser.

The "enumDefinition"-rule fires as "enum" is stated as a string. I guess 
that the parser sees the "enum" string, finds no modifiers, and 
interprets the previous IDENT ("Enumeration") as an error for an enum 
definition.

Thus the enableEnum(false) tactic does not work as it only handles the 
enum-identifier in the lexer.


David Ewing schrieb:

> It's called on demand. Another possibility would be to change 'IDENT'  
> to '(IDENT | "enum")' in a few key places elsewhere in the grammar  
> (e.g. variable declarators, expressions). That might be a tad more  
> compatible, but it also might add ambiguities to the grammar. In any  
> case, I haven't tried either, but I'd be interested to hear if they  
> work.


Michael Studman schrieb:

> If you're not in a position to be able to change the source code  
> you're parsing and you can guarantee "enum" is not used as a  keyword  
> then you can call JavaLexer.enableEnum(false) on the  lexer before you 
> pass it into the recogniser.

Strangely, this seems not to work in my system. I use the latest java.g 
with minor changes (e.g., LineNumberAST) and tried to set the 
enumEnabled field after constructing the Lexer:
            Java15CodeLexer javaLexer = new Java15CodeLexer(readerOne);
            javaLexer.setFilename(sourceName);
            
javaLexer.setTokenObjectClass("core.extractor.parser.LineNumberToken");
            javaLexer.enableEnum(false);
Alternatively, I initialized the field with false but that did not work 
either.
    private boolean enumEnabled = false;

Btw: I use antlr-2.7.5.exe on windowsXP to generate the lexer and parser 
and use a filter to hide comments:
            TokenStreamHiddenTokenFilter javaFilter  = new 
TokenStreamHiddenTokenFilter(javaLexer);
            javaFilter.discard(Java15CodeTokenTypes.WS);
            javaFilter.hide(Java15CodeTokenTypes.ML_COMMENT);
            javaFilter.hide(Java15CodeTokenTypes.SL_COMMENT);
            javaFilter.hide(Java15CodeTokenTypes.JAVADOC_COMMENT);
            Java15CodeParser javaParser = new Java15CodeParser(javaFilter);
            javaParser.setFilename(sourceName);
            javaParser.compilationUnit();


Thank you all,
Joerg

-- 
Joerg Rech
Computer Scientist        Web:   http://www.joerg-rech.de
Tel.: (+49|0) 631/10653   email: joerg.rech at gmail.com