[antlr-interest] Re: [v2] Separating Parser and Lexer - was: Parser seems to confuse strings and literals

[Charles Felstead]

John, you're the man!!

Your suggestion works perfectly! I applied your suggestion to my
lexer/parser and it worked just fine. I now have two different .g files.
However, I now realize that I can't easily get a generic lexer as it has to
inport the vocabulary for each and every grammar I come up with.

But that problem has more to do with the way I wrote my lexer/grammars. If I
was not using string literals in the grammar, than I would not need to
import them in the lexer. But assuming I need those literals somewhere, than
I would have to put them in the lexer which defeats the prupose of trying to
make a generic lexer?

So in the end, I'm going to stay with one combined lexer/parser .g file for
every file type I need to parser because at least this way, the lexer is
clean; all the specifics are in the grammar.


Thanks very much to all of you for having been very responsive and useful!!

Charles.


On 8/15/06, John B. Brodie <jbb at acm.org> wrote:
>
> Charles :-
>
> >Now this time I did real clean build; I removed every .class and all the
> >generated .java files before running antlr on the two .g files. Then I
> >generated the lexer, compiled it, and checked that it still works.
> Finally,
> ^^^^^^^^^^^^^^^^^^^
> >I generated the parser and compiled it but it doesn't work anymore.  So,
> can
>    ^^^^^^^^^^^^^^^^^^^
> >you tell what else I have to do when the lexer/parser grammars are in
> >seperate files?
>
> It is my understanding that you should exportVocab from the Parser;
> importVocab into the Lexer; *AND* generate the Parser *BEFORE* the Lexer.
>
> Here is a short 4 file example which works for me. Does it work for you?
>
> file 1) make.sh - a linux bash script to build the example
> // ------------------------------ Begin ------------------------------
> #!/bin/bash
>
> java antlr.Tool SepParser.g
> java antlr.Tool SepLexer.g
>
> javac *.java
>
> java Main
> // ------------------------------  End  ------------------------------
>
> file 2) SepParser.g
> // ------------------------------ Begin ------------------------------
> class SepParser extends Parser;
>
> options {
>     exportVocab = Separate;
> }
>
> program
>     : "begin"
>         ( id:ID EQUALS n:NUMBER
>             { System.out.println(id.getText() + " is number " + n.getText());
> }
>         )+
>         "end"
>         EOF
>     ;
> // ------------------------------  End  ------------------------------
>
> file 3) SepLexer.g
> // ------------------------------ Begin ------------------------------
> class SepLexer extends Lexer;
>
> options {
>     testLiterals = false;
>     importVocab = Separate;
> }
>
> ID options{ testLiterals = true; } : LETTER ( LETTER | DIGIT | '_' )* ;
>
> NUMBER : ( DIGIT )+ ;
>
> EQUALS : '=' ;
>
> protected LETTER : 'a' .. 'z' | 'A' .. 'Z' ;
> protected DIGIT  : '0' .. '9' ;
>
> WS : ( ' ' | ( '\t' { tab(); } ) | '\f' )+ { $setType(Token.SKIP); } ;
> NEWLINE : (('\r' ('\n')?) | '\n' ) { newline(); $setType(Token.SKIP); } ;
> // ------------------------------  End  ------------------------------
>
> file 4) Main.java
> // ------------------------------ Begin ------------------------------
> import java.io.*;
>
> class Main {
>    public static void main(String[] args) {
>       try {
>          SepLexer lexer =
>             new SepLexer(new StringReader("begin ANTLR = 1 end"));
>          SepParser parser = new SepParser(lexer);
>          parser.program();
>       } catch(Exception e) {
>          System.err.println("exception: "+e);
>       }
>    }
> }
> // ------------------------------  End  ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20060818/59ddcfc9/attachment-0001.html