[antlr-interest] [v2] Separating Parser and Lexer - was: Parser seems to confuse strings and literals

[John B. Brodie]

Charles :-

>Now this time I did real clean build; I removed every .class and all the
>generated .java files before running antlr on the two .g files. Then I
>generated the lexer, compiled it, and checked that it still works. Finally,
 ^^^^^^^^^^^^^^^^^^^                          
>I generated the parser and compiled it but it doesn't work anymore.  So, can
   ^^^^^^^^^^^^^^^^^^^
>you tell what else I have to do when the lexer/parser grammars are in
>seperate files?

It is my understanding that you should exportVocab from the Parser;
importVocab into the Lexer; *AND* generate the Parser *BEFORE* the Lexer.

Here is a short 4 file example which works for me. Does it work for you?

file 1) make.sh - a linux bash script to build the example
// ------------------------------ Begin ------------------------------
#!/bin/bash

java antlr.Tool SepParser.g
java antlr.Tool SepLexer.g

javac *.java

java Main
// ------------------------------  End  ------------------------------

file 2) SepParser.g
// ------------------------------ Begin ------------------------------
class SepParser extends Parser;

options {
    exportVocab = Separate;
}

program
    : "begin"
        ( id:ID EQUALS n:NUMBER
            { System.out.println(id.getText() + " is number " + n.getText()); }
        )+
        "end"
        EOF
    ;
// ------------------------------  End  ------------------------------

file 3) SepLexer.g
// ------------------------------ Begin ------------------------------
class SepLexer extends Lexer;

options {
    testLiterals = false;
    importVocab = Separate;
}

ID options{ testLiterals = true; } : LETTER ( LETTER | DIGIT | '_' )* ;

NUMBER : ( DIGIT )+ ;

EQUALS : '=' ;

protected LETTER : 'a' .. 'z' | 'A' .. 'Z' ;
protected DIGIT  : '0' .. '9' ; 

WS : ( ' ' | ( '\t' { tab(); } ) | '\f' )+ { $setType(Token.SKIP); } ;
NEWLINE : (('\r' ('\n')?) | '\n' ) { newline(); $setType(Token.SKIP); } ;
// ------------------------------  End  ------------------------------

file 4) Main.java
// ------------------------------ Begin ------------------------------
import java.io.*;

class Main {
   public static void main(String[] args) {
      try {
         SepLexer lexer =
            new SepLexer(new StringReader("begin ANTLR = 1 end"));
         SepParser parser = new SepParser(lexer);
         parser.program();
      } catch(Exception e) {
         System.err.println("exception: "+e);
      }
   }
}
// ------------------------------  End  ------------------------------