[antlr-interest] inversion problems: unreachable statements & lex failures

[mzukowski at yci.com]

You really should match something in a non-protected lexer rule

TEXT : (~',')* ;

Should be 

TEXT : (~',')+ ;

Give that a try.  It may still fail, if so let us know.

Also check that you have the charVocabulary option set (no, I guess you
don't looking below.)  If it is not set then your entire lexical alphabet is
whatever is referenced from lexer and parser.  In these examples that is
just ',' and nothing else.  ANTLR does not match all ASCII characters by
default.  

Monty

-----Original Message-----
From: girard_chandler [mailto:girard_chandler at yahoo.com] 
Sent: Tuesday, September 09, 2003 1:43 PM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] inversion problems: unreachable statements & lex
failures


I've tried to scan the archives and web for similar problems but I haven't
found any solutions.  Apologies if these are known problems.

1st, the following produces code which can't compile:

        class P extends Parser;
        startRule :  TEXT ( COMMA TEXT )* ;

        class L extends Lexer;
        TEXT : (~',')* ;
        COMMA : ",";

Gives:

+ java antlr.Tool -trace t.g
ANTLR Parser Generator   Version 2.7.2   1989-2003 jGuru.com
warning: public lexical rule TEXT is optional (can match "nothing")
+ javac L.java Main.java P.java PTokenTypes.java
L.java:100: unreachable statement
                        if ( _createToken && _token==null &&
_ttype!=Token.SKIP ) {
                        ^
1 error


This error can be eliminated by lexing a bogus keyword which overlaps the
TEXT token.  However, the result fails to run:

        class P extends Parser;
        options { buildAST=true; }
        startRule :  a:TEXT
                { System.out.println("a:" + a.getText()); }
                ( COMMA b:TEXT
                { System.out.println("b:" + b.getText()); }
                )*
                ;

        class L extends Lexer;
        options { k=2; }
        TEXT : (~',')+ ;
        COMMA : ",";
        HACK: ",==";

When run, with input "a,b,c", I get:

java Main
 > startRule; a,b,c
exception: line 1:1: unexpected char: 'a'
line 1:1: unexpected char: 'a'
        at L.nextToken(L.java:78)
        at antlr.TokenBuffer.fill(TokenBuffer.java:69)
        at antlr.TokenBuffer.LT(TokenBuffer.java:86)
        at antlr.LLkParser.LT(LLkParser.java:56)
        at antlr.LLkParser.trace(LLkParser.java:66)
        at antlr.LLkParser.traceIn(LLkParser.java:78)
        at P.startRule(P.java:56)
        at Main.main(Main.java:8)

This can be partially fixed by not using an inverse rule, and substituting
the following for TEXT:

TEXT : ('a'..'z'|'A'..'Z')+;

Then the above example works fine.  However, I want to read all possible
characters, except for ','.

Can this be done with Antlr?

Thanks,

Girard

p.s.
Partial java output for the 1st example:
L.java:
        public final void mTEXT(boolean _createToken) throws
RecognitionException, CharStreamException, TokenStreamException {
                int _ttype; Token _token=null; int _begin=text.length();
                traceIn("mTEXT");
                _ttype = TEXT;
                int _saveIndex;
                try { // debugging

                        {
                        _loop6:
                        do {
                                {
                                        matchNot(',');
                                }

                        } while (true);
                        }
                        if ( _createToken && _token==null &&
_ttype!=Token.SKIP ) {
                                _token = makeToken(_ttype);
                                _token.setText(new String(text.getBuffer(),
_begin, text.length()-_begin));
                        }
                        _returnToken = _token;
                } finally { // debugging
                        traceOut("mTEXT");
                }
        }

p.p.s:
My Main.java:
import java.io.*;
class Main {
    public static void main(String[] args) {
        try {
            L lexer = new L(new DataInputStream(System.in));
            P parser = new P(lexer);
            parser.startRule();
        } catch(Exception e) {
            System.err.println("exception: "+e);
            e.printStackTrace();
        }
    }
}


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/