Skip to end of metadata
Go to start of metadata

Steps needed to make a Grouper release

Final todo's

  • Shilen WS tests
  • Chris redo messaging tarball for Vivek
  • Chris look at jars not supposed to be there (from Chad)
  • Bert putting pspng changes in master

  • Bert 2 tarballs

  • Once release ready: Chad pom.xml dance
  • Once release ready: Tell packaging team to package it up
  • Chris and Shilen upgrade
    • This week
    • Shilen (08/08/2018) - Tested upgrade using Oracle - Installed 2.3 API/UI/WS using installer with all patches.  Upgraded each of the 3 components using the 2.4 installer without any issues.
    • Shilen (08/08/2018) - Tested upgrade using Oracle - Installed 2.3 API using installer with only API patch 1.  Upgraded API without any issues.  Confirmed ehcache migration.
  • Once release out: Chris to patch grouperClient checkconfig file size
  • Chris and Emily to announce and update wiki
  • Emily check wiki make sure things are up to date
    • This week

Verifying cherry-picks from branch are in master

This is a somewhat tedious step, as not all commits are official cherry picks (they could be two similar commits), and some cherry picks don't get detected as identical changes (e.g. if there is a merge conflict).

As a first step, run `git checkout <branch>; git cherry master`. This will list all the commits in the branch. Commits with a minus (-) sign are already cherry picked. and commits with a plus (+) sign need further investigation. This will narrow down the investigation a lot. Alternatively, you can use `git log --left-right --graph --cherry-mark --oneline GROUPER_2_3_BRANCH...master` which will show the commit graph for both branches, with '<' for the commits only in the branch and '=' for cherry-picked commits.

For the commits needing attention, there is often no quick way to verify they have been applied to master. Probably the easiest thing to do is look at what changed using `git show <commit>`, see what changed, and then `git blame <filename>` in the master branch to see if the changes somehow made it to master. Github also has a git blame function, so you can compare changes in two different windows. Another way to compare, if you can find a master commit with the same commit message, is to do a `git show` in both commits and compare the differences (maybe using the diff command). If the only differences are in line numbers and spacing, the commits are a good match.

For all the commits that can't be accounted for, ask the original committers to look into why they never got cherry-picked.

Changes in git

Check git to make sure all commits in 2.3 are in master branch (verify cherry-picks as above)

Edit all the README.txt files (done for 2.4) 

Edit all misc/version.properties (done for 2.4)

Update Grouper specsheet software requirements page (done for 2.4)

Grouper client (done for 2.4)

  • conf/grouper.client*.properties  grouperClient.webService.client.version = v2_4_000

Look for other occurrences of grouper.client*.properties and refresh them (should only be grouper API, wsSourceAdapter, subject) (done for 2.4)

Look for grouperClient.jar in SVN, and update it after build and copy to release server (should only be grouper API) (done for 2.4)

Grouper API

  • GrouperVersion.java (done for 2.4)
  • Make sure the subject jar up to date in api lib
  • Run the command line utility to code generate which jars/sizes/etc are expected: GrouperCheckconfig.main() (did this but do it again in final version) (done for 2.4)
  • Tag, build, get grouper client jar, but back in API, run GrouperCheckConfig, commit and push, tag again, build, and only take the API and put to release server (done for 2.4)

Grouper installer

  • grouper.installer.example.properties (done for 2.4)
  • GiGrouperVersion.java (done for 2.4)

Grouper UI

  • translate the Grouper UI text to French (our French partners have agreed to do this, as of discussion with Chris in Fall 2012)

Grouper WS

  • grouper-ws.base.properties (done for 2.4)
  • If there is a new minor version with changes to the wsdl, add a new source folder for that version, copy the coresoap *.javas over to it, refactor to change the package, change the build.xml to build a WSDL for that version.  Change GrouperService to be like a previous version but add new methods or change methods in it (done for 2.4)
  • Make sure that there is no coresoap (package) in use in src/grouper_ws_v1_6, src/grouper_ws_v2_0, etc.  Take out all src/grouper_ws_vx_x except one, make sure they dont depend on each other (done for 2.4)
  • Generate the wssec aar's and commit them (done for 2.4)
  • Generate WSDLs and commit them (done for 2.4)
  • Add src/grouper-ws_v2_4 to the source list of the Maven build-help plugin in grouper-ws/grouper-ws/pom.xml

Search for the old version (e.g. 2.2.2 or 2_2_2 or 2_2_002) in all the files (done for 2.4)

See which config files / jar files / ddl changed since last release, make sure there are entries in the change log. (do we do this anymore? Grouper changes v2.2)

If its a minor release, change the release version policy page (done for 2.4)

Make sure the crontab on webprod3 is installed for this version to make the "grouper.all*.tar.gz tarball" (done for 2.4)

Git Release Commit

This is the single commit that will be the official release. All code work should be finished at this point.

Create a new branch: git checkout -b GROUPER_2_4_BRANCH

Edit all the pom.xmls <version> tags

  • FROM <version>2.4.0-SNAPSHOT</version>
  • TO <version>2.4.0</version>


mvn versions:set -DnewVersion=2.4.0

# if the git diff looks ok:
mvn versions:commit

# otherwise
# mvn versions:revert


Look for other snapshot references that needs to change: `find . -name pom.xml | xargs grep SNAPSHOT`

Compile grouperClient and morphString with the 2.4.0 tag, in case the jar size changes

Update checkConfig with the grouperClient file size

Update grouperClient and morphString jars in the grouper lib (anywhere else?)

Commit in the new branch. Note, if this is in the master branch, or any other branch that Travis will act on, include "[ci skip]" in the commit message, so that it won't build and publish to the Sonatype snapshot repository.

Push the new branch (TODO needs an extra git command to track the remote?)

For the commit just pushed, tag as GROUPER_2_4_0 and push


git tag GROUPER_2_4_0
git push --tags

Or, if there is already a GROUPER_2_4_0 tag, these steps will override it in the remote repository:

# Retag:
# https://stackoverflow.com/questions/8044583/how-can-i-move-a-tag-on-a-git-branch-to-a-different-commit
# https://hackernoon.com/under-the-sink-removing-remote-git-tags-3d5644d67ea0

# (this will force delete the tag in the remote tag, since there is no source before the colon)
git push upstream :refs/tags/GROUPER_2_4_0

    # > To https://github.com/Internet2/grouper.git
    # >  - [deleted]                   GROUPER_2_4_0

# Force update the local tag to the current commit
git tag -fa GROUPER_2_4_0

# Dry run - note in the output that it will push the commit plus the tag
git push --dry-run upstream master --tags

git push upstream master --tags

In the branch, change pom 2.4.0->2.4.1-SNAPSHOT, update .travis.yml to include the 2.4 branch, commit, test build, commit, and push

In master, change pom 2.4.0-SNAPSHOT->2.5.0-SNAPSHOT, test build, commit, and push


Handle Copyright According to Policy

Run the copyright wizard on the entire branch for java files.  Or you can diff in version control and go through the new files.  Note, only new files should have diffs...  this is the custom copyright.  You can update the year for new files, don't update the year for existing files.  The year should be used and not a range.  That is our policy on copyrights. (dont want to change all files)

Copyright 2015 Internet2

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Make sure the subjects.sql and quickstart.xml file are in the release directory (copy from 2.3.0) (done for 2.4)

Misc Checks

  • Make sure all tables, views, and cols (of tables and views) have comments in the DB (oracle or postgres). (TO DO on a rainy day)
  • Make sure the grouper-ws and gruoper-ui web.xml file has the security basic auth in there (didn't get accidentally overridden in commit) (done for 2.4)

Security Review

Review UI / WS code changes* from the previous version with respect to Open Web Application Security Project (OWASP) top 10 web application security vulnerabilities 

If possible run a security scan against a test instance - University of Pennsylvania via Chris?

*We have not carried out a thorough security review of the existing code base for any version of Grouper. We should do that in order for the incremental reviews to be adequate

Testing

Run API (SuiteDefault) JUnit tests (set true in all JUnit test includes in grouper.properties).  Also run the Grouper Installer, it should end in success for the client connecting to the WS and you should be able to use the UI.

API JUnit Tests

Database

Linux

OS X (10.6)

Solaris

Windows

HSQLDB





MySQL (with utf/bin collation table types)





Oracle





PostgreSQL





MsSQL





  • MySQL windows
  • MySQL unix with case sensitive table names
  • Postgres
  • Oracle

    create role GrouperRole;
    GRANT connect, resource, CREATE view TO GrouperRole;
    CREATE USER groupertest IDENTIFIED BY <pass>;
    GRANT connect, resource, GrouperRole TO groupertest;
    
  • Hsql
ps -ef | grep java   --- kill hsql
wget http://www.internet2.edu/grouper/release/x.y.z/grouper.apiBinary-x.y.z.tar.gz
cd grouper.apiBinary-2.0.0
cd bin
java -cp ../lib/jdbcSamples/hsqldb.jar org.hsqldb.Server -port 9001 -database.0 file:grouper -dbname.0 grouper &
./gsh.sh -registry -runscript
./gsh.sh -test -all

Try the UI with a few different languages in the browser request (en without US, french, something not common)

Try checkConfig tests.

Upgrade from x.y-1 to x.y (e.g. 2.3 to 2.4).

  •  
  • Open the UI, browse around

Install the grouper installer

  • Download grouper installer
  • Run: java -jar grouperInstaller.jar
  • Try the UI that was installed
  • Make sure gsh.sh and gsh works in api, ws, ui for 2.4.0

Database

Linux

OS X (10.6)

Solaris

Windows

HSQLDB

CH




MySQL





Oracle





PostgreSQL





MsSQL





Web Service JUnit Tests (Grouper WS and Grouper Client) (Shilen work on for 2.4.0, has errors in client tests)

WS samples (DONE for 2.4.0)

WS javadoc (generate, commit, test).  Make sure new operations / args / etc are documented in the WS doc page (TODO?)

UI internationalization tests (TODO?)

Test an API patch

# try to see if someone else is testing, look when process started, wait if so or email list...
[appadmin@i2midev6 ~]$ ps -ef | grep gsh | grep test | grep all



# see if hsql is running, and stop if so
[appadmin@i2midev6 ~]$ ps -ef | grep hsql
[appadmin@i2midev6 ~]$ kill <the_pid>
[appadmin@i2midev6 ~]$ testGrouper2_3.sh > /tmp/grouperTestResults.txt & 2>&1
[appadmin@i2midev6 ~]$ tail -f /tmp/grouperTestResults.txt

Compile test on server

/**
 * Test {@link Stem}.
 * <p />
 * @author  blair christensen.
 * @version $Id: TestStem.java,v 1.34 2009-12-07 07:31:09 mchyzer Exp $
 */
public class Test extends GrouperTest {

    // Private Class Constants
    private static final Log LOG = GrouperUtil.getLog(Test.class);

    /**
     * Method main.
     * @param args String[]
     */
    public static void main(String[] args) {
        TestRunner.run(new Test("testCache"));
        //TestRunner.run(TestStem.class);
    }

    public Test(String name) {
        super(name);
    }


javac -classpath .:../lib/grouper/*:../lib/test/*:../dist/lib/* -sourcepath . Test.java


 java -classpath ../conf:.:../lib/grouper/*:../lib/test/*:../dist/lib/grouper.jar:../lib/jdbcSamples/hsqldb.jar Test


Packaging and releasing

  • ssh to i2mibuild.internet2.edu
  • cd to ~mchyzer
  • Build all with: [mchyzer@i2mibuild mchyzer]$ bin/buildGrouperAll.sh GROUPER_x_y_z
  • This is the same as these individual commands:
    • bin/buildGrouper.sh GROUPER_x_y_z
    • bin/buildGrouperWs.sh GROUPER_x_y_z
    • bin/buildGrouperClient.sh GROUPER_x_y_z
    • bin/buildGrouperUi.sh GROUPER_x_y_z
    • bin/buildGrouperQs.sh GROUPER_x_y_z
  • Note: sometimes the grouperWs doesnt build correctly due to a bug with javadoc, just try building the ws again...

Resulting .tar.gz's are in ~mchyzer/tmp/grouperAll/build_<username>. There are both source and binary tarballs for the API and Client packages.

  • scp packages built above, to:
sftp login.internet2.edu    (then sftp to webprod3)
lcd /home/mchyzer/tmp/grouperAll/build_username
mkdir /home/htdocs/home/htdocs/software.internet2.edu/grouper/release/1.2.12
cd /home/htdocs/software.internet2.edu/grouper/release/1.2.12
mput *.tar.gz
  •  Build to demo server, after the release is tagged, and built on the build server in Chris' directory... (TODO after release)

[appadmin@i2midev1 2.0.0]$ pwd
/opt/grouper/2.0.0
[appadmin@i2midev1 2.0.0]$ ./upgrade_2.0.0.sh

Packaging and releasing : provisioning

After tagging, publish to maven central by following the instructions on v2.1.0 Grouper Development Environment Using Maven.

Publish to maven central here

Copy grouper.psp-version.tar.gz to the release URL.

Archive the current release

  • Create PDF of current documentation for the release archive
    • Browse -> Advanced -> Export Space
    • select PDF, uncheck save comments, clear all
    • select pages to export. try to omit non-product pages, defunct pages, and pages for future releases.
    • export, save to desktop, rename and attach to GrouperWG/Archives
  • update Archives page with info from the current vN.N+Release+Notes page and links from the Software+Download page.

Update vN.N+Release+Notes page

Generally roll notes from the oldest release off and add notes for the new release. Keep previous release notes in there, so that we always have current + previous.

Update Software+Download page

https://spaces.at.internet2.edu/display/Grouper/Grouper+Downloads

Review product pages

  • Ensure that each page identifies which version it's current for, as in "Building the Grouper API as of v1.4.1" as an h2 at the top.
  • Review incoming and outgoing links info for each page.
  • If there's new features with new pages, update the Grouper+Product page to appropriately reference the new pages.
  • Is it all there?
  • Are there pages in the Working Group or Community areas that have graduated to being core product doc? Negotiate with originator to move them over to the product pages.

Update GrouperWG/Home

Just update the "NEW!!" message on this page. Maybe review the Background section to see if it could use some freshening.  Maybe add a news item

Review Training Videos

Review Impact, if any, on TIER Grouper Deployment Guide

  • Consider if the new release should lead to any changes/updates in the TIER Grouper Deployment Guide  
  •  if yes, discuss with Bill Thompson, Lafayette College

Update Grouper Wiki Documentation

  • Update wiki pages for all changed features. 
  • Add a new selection to the Grouper Admin Guides TOC if needed. See here  https://spaces.at.internet2.edu/x/UAfw
    • Be sure that Grouper wiki doc pages for new features are moved from the   Development Items are and area and instead linked from the Admin Guides area. 
  • Review the Glossary, remembering this could be a first place new people go to learn important terms. It seems rusty to Emily but not exactly sure how to improve it. HELP appreciated.
  • Be sure that wiki pages on new Features are linked correctly from the Admin Guides TOC page

Release Numbering and Testing Period

As of Feb. 2012, these procedures were adopted regarding release numbering and testing:

- The X.X.0 release will be considered a pre-release, like a Release Candidate
- That will be the testing release
- The X.X.1 release will be the actual release, presumably after any initial bugs identified have been fixed.

Packaging team

Talk to the TIER packaging team to see what their plan is

Notify about the release

  • Compose email to grouper-dev@internet2.edu and grouper-users@internet2.edui and grouper-announce@internet2.edu with highlights of the new release and link to the Grouper Downloads page
  • email to mw-announce [at] internet2.edu email list
  • Highlights should resemble those on the vN.N Release Notes page and maybe even be identical.
  • If it's especially enlightening, also include a direct link to the changelog (vN.N+Release+Notes#vN.NReleaseNotes-changelog) or to an appropriate Jira report.
  • Change jira admin to make that version released, and if the next build or version isnt there, add it

Other Outreach

  • update Grouper website features page to highlight newest Grouper features
  • Social Media: Announce the new Grouper release on the InCommon and Internet2 Facebook and Twitter accounts as appropriate
  • work with Todd Sedmak, Internet2 PR & Media Relations Manager, around other possible publicity/press releases etc.
  • A Grouper Team member (DaveL for the Grouper 2.2 release) will create a Youtube video (using same tools as the Grouper Training videos) highlighting new features
    • promote this video in coordination with the Internet2 Marketing and Communications Team 
  • schedule a webinar to promote the new release


Publishing Maven jars to Sonatype (Maven Central)

See also.: v2.1.0 Grouper Development Environment Using Maven#Maven+Central+Deployment

Make sure you are at the official release commit, as it's the only commit without a -SNAPSHOT version number

git checkout GROUPER_2_4_0 (warnings about detached HEAD are ok, just remember to checkout an actual branch when finished)

cd ~/bin
./getGrouperSource.sh GROUPER_2_4_0

    #> --2018-08-29 16:19:42--  https://github.com/Internet2/grouper/archive/GROUPER_2_4_0.zip
    #...
    #> 2018-08-29 16:24:45 (992 KB/s) - ‘GROUPER_2_4_0.zip’ saved [307471992]
    #> 
    #> mv: cannot stat ‘GROUPER_2_4_0’: No such file or directory
    #> unzipping GROUPER_2_4_0.zip...
    #> Grouper unzipped to /home/mchyzer/tmp/grouperDownload/build_mchyzer/grouper-GROUPER_2_4_0

cd ~/tmp/grouperDownload/build_mchyzer/grouper-GROUPER_2_4_0
cd grouper-parent

export PATH=~/software/apache-maven-3.3.3/bin:$PATH
export JAVA_HOME=/opt/java8
LOGFILE=~/tmp/grouperDownload/build_mchyzer/grouper-GROUPER_2_4_0.mvn.log

mvn -Dlicense.skip=true -DskipTests=true clean deploy -Prelease -Dadditionalparam=-Xdoclint:none | tee $LOGFILE 2>&1


The Maven step will package all the artifacts, gpg sign them (creating *.asc for all the objects) then upload them to the Sonatype staging repository

Go to https://oss.sonatype.org/#stagingRepositories and there should be a newly created grouper repository with the artifacts inside. The login and password for Sonatype can be found in ~/.m2/settings.xml.

If the objects are all there and look ok, click the close icon to freeze the repository. At that point, you should test the objects, by trying to building something, using the staging repository as if it were the release repository. In your .m2/settings.xml, add:


  <profiles>

    <profile>
      <id>grouper-stage</id>
      <repositories>
        <repository>
          <id>grouper-stage</id>
          <url>https://oss.sonatype.org/service/local/repositories/eduinternet2middlewaregrouper-1014/content</url>
          <releases>
            <enabled>true</enabled>
          </releases>
        </repository>
        <!-- ... -->

  <activeProfiles>
    <activeProfile>grouper-stage</activeProfile>
  </activeProfiles>


If it builds ok, click the release icon for the repository, and then it will be moved to the official release repository. Within a few hours, it may be synchronized to other Maven repositories.

If something isn't right with the staging repository, just drop it and the Maven build will create a new one with a different increment number.

Grouper Failing Tests

TestOwnerStatus and dateReason it was failing
testIndexesGroupVivekFixedNow we are creating more folders at the startup time due to deprovisioning and they are using all the reserveIds.
testProcessMessagesHappyPathVivekFixedfake http server was being created at the wrong context.
testFindAttrDefAttributeAssignmentsChrisFixedbad security was being called
testFindAttrDefAttributeAssignmentsByValueChrisFixedbad security was being called
testFindByAttributeAssignOnAssignValuesAndPrivilegeVivekFixedNeeded to assign correct privileges to test subject
testGrouperAttestationPrivilegesVivekFixedNeeded to assign correct privileges to test subject