Page History

Grouper currently has no facility to manage third party library dependencies. Various JARs exist in the codebase itself that are pulled together via build scripts to execute compilation, packaging and deployment tasks. This approach has a number of drawbacks:

• Difficult to upgrade a JAR, should a new version be required
• Difficult to add new capabilities to Grouper which may depend on new JARs
• Grouper codebase size is dramatically increased

We would want to devise a strategy by which such third party libraries can be pulled from public artifact repositories and integrated with the build. The following dependency management solutions may be considered.

Apache Ivy

The Grouper's build is managed via Apache Ant. Apache Ivy simply adds dependency management to Ant without replacing the existing build. Dependencies can be pulled from public repositories, conflicts managed and profiles created, etc. The advantage of using this option would be strong and tight integration with existing Ant scripts with relatively minimal effort. 

Notes to consider, if Ivy is chosen:

1. Existing ant build scripts can be kept. Ivy has excellent integration with Ant. 
2. Consider that Ant is somewhat of a deprecated tool and is largely being replaced by the likes of Maven, and specially Gradle.
3. All configuration is via XML. May or may not be desirable. 
4. Ant installation, and configuration is required of a grouper developer.
5. IDE integration is decent

Apache Maven

Apache Maven is a build AND dependency management tool. While Ant is a bit more task-based and can provide script-like tasks and targets to execute builds without requiring any particular project structure or layout, Maven introduces a new standard layout for projects' resources and additionally allows support for managing and pulling in dependencies. Maven is also, unlike Ant, lifecycle based and so various phases in the overall build cycle are introduced by default that take care of compilation, packaging and deployment of artifacts without requiring much custom work. 

Notes to consider, if Maven is chosen:

1. Existing ant build scripts need to be rewritten for Maven. Both are XML, but the syntax and format is totally different and it's not possible to use Maven's dependency management features without using its build features.
2. Of course, it's all going to be XML. Configuration of builds, dependencies, profiles, plugins and such can be very verbose.
3. Maven is not particularly good at running custom build scripts. All tasks need to fit into its own phases and the lifecycle cannot easily be intercepted or modified without writing maven plugins.
4. A Maven installation and configuration is required of a grouper developer.
5. Plenty of plugins exist for Maven that provide integration with all sorts of systems and capabilities.
6. Grouper can take advantage of Maven's release plugin, by publishing artifacts to Maven central, etc when needed.
7. JARs that are not found in any public repositories can be added to a maven build. While Maven has a concept of System dependencies, it would however be best if such JARs are put into a repository that is perhaps owned by Internet2. 
8. Maven has a concept for Profiles, which allow specific settings to be activated per environment (dev, QA, etc)
9. IDE integration is decent. Eclipse may be problematic though. 
10. Custom artifact repositories can be declared with no problem at all. 
11. Maven's dependency management can result into a "Dependency Hell" type of problem for very large and complicated projects. Managing transitive dependencies and resolving conflicts may not be immediately trivial. 

Gradle/Grape

Gradle is also an entirely new build tool with a dependency management feature called Grape. Gradle is the best of Ant and Maven combined. It has support for Maven project layouts and Maven repositories for managing dependencies. Build scripts are entirely based on Groovy and so can be heavily customized to provide Ant-like features. Gradle also introduces a standard set of phases and goals, and additional tasks can be added to the build script. The dependency management of Gradle, Grape, also has the advantage of declaring dependencies directly in the code via annotations, can work with Maven and Gradle repositories:

Notes to consider, if Gradle is chosen:

1. Existing ant build scripts need to be rewritten for Gradle. It's not possible to use Gradle's dependency management features without using its build features.
2. Of course, the build script is written in Groovy. May or may not be desirable.
3. Gradle by far is the most modern and sought after build tool, today. 
4. A Gradle installation is NOT required of a grouper developer. Gradle can download and configure itself.
5. Since the build script is written in Groovy, all manners of custom tasks, profiles, configurations and such can be added.
6. Gradle's concurrent and parallel builds with caching and daemons can be a HUGE benefit when running builds for large projects.
7. Gradle supports all Maven-like artifact repositories.
8. Plenty of plugins exist for Gradle that allow integration and customization of various build features.

Considerations for Grouper

1. Build/Dependency tools must be configured to retrieve the source artifact for JARs.
2. Build/Dependency tools must be easily able to create and update javadocs.
3. Build/Dependency tools must have the ability to support environment profiles.
4. Build/Dependency tools must have support for local dependencies.
5. Consider what may be the most natural and easiest option for a Grouper contributor to get started.
6. There needs to be a way for Grouper to determine the version of its JARs used:
   1. Can be use/reference the Maven build maintained in parallel with ant.
   2. Can use the JAR manifest itself.
7. IDE integration is important. While IntelliJ/NetBeans have no issues with either of these tools, Eclipse may be problematic for Maven or Gradle.

Notes about how it previously worked

1. Jars contained the source of the jar so it could be easy and low risk to debug/patch
2. Jars had names of the jar without the version in the name so that version control would show a history on the file
3. Dev target in ant would copy jars to webapp folder so the application could be debugged without a warfile.  Note, source is in jar so it can be easily debugged
4. The hibernate jar was patched, not sure which others
5. All versions are in the manifest of the jar.  If the jar doesnt put it in there by default, it was added by grouper developer
6. Grouper will check the size and version of jars on startup and warn which ones arent right
7. Grouper installer will try to look for jar conflicts

Notes of call 11/3/2015 (Misagh, JJ, Jasper, Dima, Chris)

1. Dima took first pass at gradle
2. Converts maven to gradle
3. Each project has its own gradle file
4. grouper-parent will compile subprojects
5. If there is a non standard structure we can specify in gradle script
6. gradle branch at github in unicon
7. Parent script is: https://github.com/Unicon/grouper/blob/gradle/grouper-parent/settings.gradle
8. Issue with misc stuff that is not common, do we gradlize it?  move to different branch?
9. Get rid of grouper-parent, move stuff to parent dir
10. grouperw is the wrapper to download gradle and run it
11. gradle replace installer?  Need to discuss this since installer has features
12. need to externalize dependency versions so the gradle properties dont have repetition on log4j for example
13. gradle plugin to create projects for IDE's
14. can you download all jars and sources, two line script, lets get this in
15. system jars, find dependency for custom jars (hibernate)
16. manipulate jar manifest for implementation-version?  do checksums of jars?
17. offline builds?  would tarballs have jars in it?  might be able to
18. grouper ui not in maven central.  grouper ui java in different project than jsp?
19. artifactory is proxy for dependencies

Overview

A functional gradle build environment exists in the Github repository in branch "gradle". The build is orchestrated by a parent build.gradle file for the root grouper project and a number of individual build.gradle file for every module that Grouper presently houses. Each build.gradle file inherits from the parent build.gradle file and in turn is able to specify its own requirements for dependency management and source tree structure. The list of Grouper modules that are recognized by the Gradle build is specified via a settings.gradle in a specific order to account for the build dependencies of the modules. There is also one gradle.properties file which hosts specific build settings for the project, such as groupid, java version and dependency versions used throughout the project. 

Build

To run the full build, issue the following command at the root:

./gradlew build -x test -x javadoc --parallel --stacktrace

The same command can be used inside a specific module directory to build that module only. 

The above command will use the Gradle wrapper to download an appropriate Gradle version once to be used for the build. This removes the requirement for a developer to explicitly have Gradle downloaded and installed, but if a Gradle installation is already available, the same command can be used via invoking the native Gradle instance installed. Build artifacts are produced for each module inside the build/libs directory. JAR files are produced for main, tests and javadoc. 

./gradle build -x test -x javadoc --parallel --stacktrace

The following gradle commands may also prove useful during development:

# Show all the project dependencies
./gradle dependencies 
 
# Show where a specific dependency for a gradle configuration comes from
./gradle dependencyInsight --dependency <name> --configuration <compile/runtime/testCompile/...>
 
# Prepare an Intellij IDEA workspace for Grouper
./gradle idea
 
# Prepare an Eclipse project workspace for Grouper
./gradle eclipse

Gradle Wrapper Updates

The Gradle wrapper need only exist at the root of the project. The wrapper can be updated to note a specific newer version of a Gradle via the following command:

./gradle wrapper

Javadocs

To produce the Javadoc for the entire project, run the following command at the root:

./gradlew build alljavadoc -x test -x javadoc --parallel --stacktrace

The above command aggregates the javadocs for all grouper modules inside the root build/javadoc directory. 

Travis CI Builds

Grouper is configured against Travis CI to execute the full build on every commit. Travis builds all grouper branches provided there is a .travis.yml inside the branch at the root of the repository. This is a YAML configuration file that tells Travis what version of Java should be available, which commands to use for the build lifecycle and which scripts to execute after a successful build. 

Publishing Javadocs

As part of the Travis CI build, Javadocs are published to the gh-pages of the repository, available on the web here, once a successful builds. This is handled via the javadocs-ghpages.sh script that runs after the build. The script will attempt to collect all project javadocs and push them to the appropriate branch. In order for the push to succeed, an OAUTH token is made available to the script to do the job. The OAUTH token must be generated by a developer with push rights to the repo, and must be encrypted and placed inside the .travis.yml file under a "secure" heading. 

OAUTH token can be encrypted for Travis via the following command:

gem install travis
travis login --github-token <TOKEN>
travis encrypt GH_TOKEN=<GH-TOKEN>-r Internet2/grouper

• The TOKEN must be generated by you inside your Github profile settings. This is used to allow you to log into Travis.
• The GH-TOKEN must be generated by you inside your Github profile settings. This is used to allow Travis to push to the gh-pages branch. 
• When the values are encrypted, they are printed back on the screen. Copy/paste them to the .travis.yml file under env/secure entry. 

Publishing SNAPSHOTs

As part of the Travis CI build, SNAPSHOT artifacts for all grouper modules listed in the settings.gradle file are published to the Sonatype snapshot repositories for Grouper once a successful builds. This is handled via the deploy-to-sonatype.sh script that runs after the build. The script will attempt will collect all JAR artifacts produced by the build and will attempt to publish them. The command that used by the script is:

./gradlew uploadArchives -DpublishSnapshots=true -DsonatypeUsername=$SONATYPE_USER -DsonatypePassword=$SONATYPE_PWD

The above command invokes the "uploadArchives" task of Gradle to start publishing. The configuration of this task is specified in the root build.gradle file.

In order for the command to succeed, a developer must provide his/her own Sonatype credentials, and must also be authorized to publish artifacts to the edu.internet2.grouper namespace. Credentials that are required for the command to succeed are encrypted and placed inside the .travis.yml file under a "secure" heading. The developer must also be authorized to have access to the "Settings" area of the Grouper github repository. 

Sonatype credentials can be encrypted for Travis via the following command:

gem install travis
travis login --github-token <TOKEN>
travis encrypt SONATYPE_USER=yourUsername -r Internet2/grouper
travis encrypt SONATYPE_PWD=yourPassword -r Internet2/Grouper

• The TOKEN must be generated by you inside your Github profile settings. 
• When the values are encrypted, they are printed back on the screen. Copy/paste them to the .travis.yml file under env/secure entry. 

Development environment

• Open morphstring in eclipse
• Delete .classpath and .project

• Set java home and path
  
  

  Code Block
  C:\Users\mchyzer\Documents\GitHub\grouper\grouper>set JAVA_HOME=C:\dev_inst\java7 C:\Users\mchyzer\Documents\GitHub\grouper\grouper>set PATH=%JAVA_HOME%\bin;%PATH%

  
  

• sdf

...