[antlr-interest] Antlr ANT task: Seemless builds using the Antlr -Problems and solutions. Help appreciated!

[Scott Stanchfield]

I'm working on updating the ANTLR ANT task ;)

*However*, order of .g compilation can matter, as one grammar can output
token types that another grammar needs, or there could be grammar
inheritance.

javac doesn't have this problem b/c the compiler reads "all" the java files
and compiles them as a group. ANTLR only processes a single file at a time. 

I was planning on checking mod times and adding the new options (-smap and
-plugin [for xml support]).

For each of your requests:
* mutiple .g files -- ANTLR itself doesn't support it
  I could probably do this, with the cavaet that order can bite you

* mapping source->dest paths - hmm... the task could do that...
  ANTLR itself doesn't "know" the package; it's just part of the header spec
  I'll look into this... I'd really like this meself...

* timestamps -- I want to add this to the task

* invoking twice for docs/java -- ANTLR runs with separate
  generators for each. We should be able to write a combo generator
  that we delegate to both java and docbook or html generators...
  Hmmm... If I have time, I'll take a quick poke at this, but
  it's not likely I'll have time...

I'd also like to set up some way to make "clean" easy, so we'll know which
files were generated by antlr. I'm doing this in the eclipse plugin (I added
"clean" support), but to do it for an ant build would require tracking
generated files in another file, like .antlr-generated or something. We can
then have a <antlr clean="yes"> option that removes any generated files. (We
would have to always append to the .antlr-generated file to capture all
renames and such until a clean)
 
I'm working on the ant task today, btw, so good timing ;) Then I've got to
figure out how to contribute the changes... too much fun...

Thoughts?
-- Scott


> -----Original Message-----
> From: antlr-interest-bounces at antlr.org 
> [mailto:antlr-interest-bounces at antlr.org] On Behalf Of M C
> Sent: Sunday, April 17, 2005 7:48 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Antlr ANT task: Seemless builds 
> using the Antlr -Problems and solutions. Help appreciated!
> 
> Hi all,
> 
> This mail is about my "hard time" integrating the Antlr ANT 
> task in a seemless build enviroment. It detils some 
> solutions, some unsolved problems and a request for changes 
> in the next Antlr version.
> 
> For seemless integration I wanted to use the ANTLR task to:
> 1) Automatically pick up all *.g files in whatever
> src\** directories and generate *.java files and place them 
> in corresponding build\generatedsrc\** directories (this 
> assumes that the *.g files are placed in a sub directory 
> under the main source directory that corresponds to the 
> package the resulting java files will have when put in the 
> generatedsrc build directory. ).
> 
> 2) Generate grammar doocumentation and place it in doc directory.
> 
> 3) Only generate *.java and documentation files if the *.g 
> files have changed (timestamp of *.g files are newer than the 
> generated *.java files).
> 
> Unfortunately, unlike the javac task the antlr ant task has 
> key limitations:
> a) Does not allow for multiple *.g files.
> b) Does not provide direct support for mapping source to 
> destination paths.
> c) Does not check timestamps.
> d) Needs to be invoked twice to generate both *.java files 
> and documentation.
> 
> Despite the above limitation, I tried to use various ANT 
> extensions and complicated techniques to accomplish my goals. 
> See code below. The code is VERY complex but it does conform 
> to goals 1 and 2. However, I could not get the timestamps to 
> work. Consequently, every time I build, the ANT ANTLR task 
> generate new java code, resulting in a long cascade of 
> subsequent build actions that are uncessary when nothing has changed.
> 
> It would be really nice if the next version of Antlr would 
> provide a better ANT task that would allow for integration 
> into a seemless build enviroment.
> Meanwhile, I would welcome if anyone have ideas for improving 
> the ANT workarounds I have detailed below:
> 
> Cheers,
> Morten Christensen
> 
> ----
> 
> <target name="generateparsers" depends="init">
> 	 <echo message="Generating source code parser(s) for 
> ${ant.project.name}"/>
> 
> 	<!-- Need to generate parsers for ALL *.g files from whatever 
> 	     location. Unfortunately, the ANTLR task does not allow
> 	     for multiple *.g files so we need to iterate though
> 	     all *.g files -->
> 	<foreach target="generateparsers-antlr-subtask"
> param="antlrGrammarFile">
> 		<path>
> 			<fileset dir="${mainsrc}">
> 				<include name="**/*.g"/>	
> 			</fileset>		
> 		</path>
> 	</foreach>
> </target>
> 	
> <target name="generateparsers-antlr-subtask"
> if="antlrGrammarFile" description="internal helper target">
> 		
> 	<!-- Need to calculate location in generatedmainsrc 
> that correspond to
> 	     location in src. I.e. outputDir =
> ${antlrGrammarFile} 
> 	     with substring ${srcmain] replaced with 
> ${generatedmainsrc}.
> 	     Tricky using ANT - wish it could be easier.
> 	--> 
> 
> 	<dirname property="inputDir"
> file="${antlrGrammarFile}"/>
> 	<path id="mainSrcDirPath"
> location="${basedir}${file.separator}${mainsrc}"/>
> 	 <pathconvert dirSep="${file.separator}"
> pathsep="${file.separator}"
> property="normalizedMainSrcDir"
> refid="mainSrcDirPath"/>
> 	<path id="inputDirPath" location="${inputDir}"/>
> 	 <pathconvert dirSep="${file.separator}"
> pathsep="${file.separator}"
> property="normalizedOutputDir" refid="inputDirPath"/>
> 	<path id="normalizedOutputDirPath"
> location="${normalizedOutputDir}"/>
> 	 <pathconvert dirSep="${file.separator}"
> pathsep="${file.separator}" property="outputDir"
> refid="normalizedOutputDirPath">
> 	    <map from="${normalizedMainSrcDir}"
> to="${basedir}${file.separator}${generatedmainsrc}"/>
> 	 </pathconvert>
> 		
> 	<!--
> 	Would like to only run antlr task on *.g files that
> 	are newer than the generated java files.
> Unfortunately,
> 	I can't make the code update property initialize
> 	itself correctly.
> 		
>             <uptodate
> property="checkAntlrGrammarFileChanged">
> 	      <srcfiles dir= "/"
> includes="${antlrGrammarFile}"/>
> 	      <mapper type="merge" to="${outputDir}"/>
> 	</uptodate>
> 		
> 	<echo message="Uptodate? :
> ${checkAntlrGrammarFileChanged}"/>
> 	-->
> 		
> 	<!-- Create output dir on the fly and invoke antlr 
> (need to invoke antlr twice. Once for code, once for docs). -->
> 	<echo message="Invoking antlr on grammar 
> ${antlrGrammarFile} with code output to directory 
> ${outputDir} and documentation output to ${referencedocs}"/>
> 	<mkdir dir="${outputDir}" />
> 	<antlr dir="."
> 	    target="${antlrGrammarFile}"
> 	    outputdirectory="${outputDir}"/>
> 	<antlr dir="." html="true"
> 	    target="${antlrGrammarFile}"
> 	    outputdirectory="${referencedocs}"/>
> </target>
> 	
>