[antlr-interest] Re: Questions about learning ANTLR

Hensley, Richard richard.hensley at mckesson.com
Fri Oct 4 16:25:05 PDT 2002

The use of token {} section is entirely optional. I use them in my
grammars because I like to organize my grammars using tokens that I can
identify. Also, if you use "keyword" construct in your grammar, then
your token type will be LITERAL_keyword. If you use KEYWORD = "keyword";
in a tokens section construct, then your token type will be KEYWORD. I
would rather use KEYWORD than LITERAL_keyword in my java code when
testing types in an AST.

I use junit by creating a number of files that have both valid and
invalid input. For the invalid input ones, I examine the returned
exception to make sure the right one happened, and that it happened in
the right place. For the valid ones, I always have a big file that
contains most cases from my grammar. If the big file throws an
exception, I print out the antlr exception, and fail the test. I also
have some smaller files, after they parse I programmatically examine the
AST to make sure that it is structured the way I expect.

One thing I do for my test data files is that I use the following Java

        InputStream in =

This allows me to place my test data in the class path, instead of some
absolute or relative file path. Just my preferece, but it works for me.
By the way, this only works if the directory where testdata/ is located
is part of the classpath.

Here is the testing target from my ant file: 

    <target name="test-java" depends="env,check-resources,build-java"> 
                <fileset dir="${test.output}" includes="*" /> 
        <junit printsummary="yes"> 
                <path refid="classpath" /> 
                <pathelement path="${testdata.classpath}" />

            <formatter type="plain" /> 
            <batchtest todir="${test.output}"> 
                <fileset dir="${source.dir}"> 
                    <include name="**/test/*Test.java" /> 

Notice that I always package my tests as a subpackage to the one being
tested. I also name my tests according to the standard of <class being
tested>Test.java. I also added the testdata/ to my classpath so the
getResourceAsStream() routines work.

-----Original Message----- 
From: micheal_jor [ mailto:open.zone at virgin.net
<mailto:open.zone at virgin.net> ] 
Sent: Friday, October 04, 2002 4:14 PM 
To: antlr-interest at yahoogroups.com 
Subject: [antlr-interest] Re: Questions about learning ANTLR 

Sorry to butt in Richard but I had a few questions once I read your 

> In general, the tokens { } section is for literal 
> strings that occur in the stream exactly as specified, or for 
> tokens required to construct a abstract syntax tree (AST). 

What about keywords?. I wondered why the java.g for example didn't 
define keywords like "class", "final" etc as tokens. 

> When I was developing my parser, I would make one simple change at a 
> time, build the parser, execute a my junit test suite. I created an 
> build.xml script that made this simple. 

How do you use junit to test a parser?. Care to share more details on 
this please?. 




Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/ <http://docs.yahoo.com/info/terms/>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20021004/73eeb572/attachment.html

More information about the antlr-interest mailing list