This is an exercise to see how an Amazon EC2 cloud server can host Grouper. Amazon has inexpensive (~$500/year) dedicated virtual servers that can be provisioned quickly. The ability to run Grouper on one of these servers could help save money for institutions or even make it feasible for other institutions that might not have the server space or resources to manage their own Grouper server. There are high availability options and clustering, though this study assumes the default server configuration.

There are issues with running Grouper on a cloud server. Grouper might contain sensitive (HIPAA or FERPA protected) data, and it might be against your institution's policy to outsource hardware for such data. Also, if the server is located offsite (in Amazon's network), or if you need secure connections (e.g., for your person source), you will need to figure out how to make that happen. E.g. with mysql over SSL or LDAP over SSL or secure oracle connections.

In any event, if you decide you want to run Grouper on cloud infrastructure, here are instructions to make it happen.

Note: you should google all the software to get the latest and greatest since the versions here will be stale soon.

Note: this documentation was posted by a developer, not a sysadmin, there are better ways to secure this software, this is one way to do it, but not necessarily the recommended way. (smile)

Provision your server

# useradd appadmin

Install Apache

# cd /opt
# mkdir software
# cd software/
# wget http://mirror.candidhosting.com/pub/apache/httpd/httpd-2.2.14.tar.gz

[httpd-2.2.14]# yum install gcc
[httpd-2.2.14]# yum install *openldap* -y
[httpd-2.2.14]# yum install *zlib* -y                 [note: this installs extraneous packages I think...]
[httpd-2.2.14]# yum install openssl-devel -y
[httpd-2.2.14]# ./configure --prefix=/opt/apache --enable-ssl --enable-cgi --enable-so --enable-shared --enable-mods-shared=all --with-ldap --enable-authnz-ldap --enable-ldap
[httpd-2.2.14]# make
[httpd-2.2.14]# make install
[opt]# chown -R appadmin.appadmin /opt/apache
Go to godaddy.com
Register the SSL and domain name: ec2testhyzer.info (make sure it is not auto renew if this is temporary)
Go to godaddy total dns control, and set the a record to the ec2 ip address
Manage SSL certificates, use the credit you just bought
Manage certificate, request certificate
   (go back to server, make a csr)
[opt]# mkdir cert
[opt]# cd cert
[cert]# openssl genrsa -out ec2testhyzer_info.key 2048
[cert]# openssl req -new -key ec2testhyzer_info.key -out ec2testhyzer_info.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:PA
Locality Name (eg, city) [Newbury]:Media
Organization Name (eg, company) [My Company Ltd]:ec2testhyzer.info
Organizational Unit Name (eg, section) []:ec2testhyzer.info
Common Name (eg, your name or your server's hostname) []:ec2testhyzer.info
Email Address []:dont@email.me

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[cert]#

copy the contents of the CSR file and paste to godaddy
generate the certificate
download the certificate
upload to ec2 with psftp
unzip it

psftp> cd /opt/cert
psftp> put c:\temp\ec2testhyzer.info.zip

[cert]# unzip ec2testhyzer.info.zip


[apache]# cd /opt/apache/conf
[conf]# yum install emacs
[conf]# emacs httpd.conf

User appadmin
Group appadmin

Include conf/extra/httpd-ssl.conf

[conf]# emacs extra/httpd-ssl.conf

ServerName ec2testhyzer.info

SSLCertificateFile "/opt/cert/ec2testhyzer.info.crt"

SSLCertificateKeyFile "/opt/cert/ec2testhyzer_info.key"

SSLCertificateChainFile "/opt/cert/gd_bundle.crt"

[init.d]# emacs /etc/init.d/httpd

#!/bin/sh
#
# Startup script for Apache
#
# chkconfig: - 85 15
# description: Apache
# processname:
# pidfile:
# config:
# Tomcat
# description: Starts and stops Apache
# See how we were called.

apachectl=/opt/apache/bin/apachectl
RETVAL=0


case "$1" in
  start)
        $apachectl start
        ;;
  stop|restart|graceful|help|configtest|fullstatus|start)
        $apachectl $@
        RETVAL=$?
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|configtest|help|fullstatus|start|graceful}"
        exit 1
esac

[init.d]# chkconfig --add httpd
[init.d]# chkconfig --levels 345 httpd on
[init.d]# chmod +x httpd
[init.d]# service httpd start

Test in a browser: https://ec2testhyzer.info/



Java

Download:
[init.d]# cd /opt/software/

NOTE: this next link wont work since it is session specific.  Google the java jdk and get the link

[software]# wget http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-6u18-linux-i586.bin?BundledLineItemUUID=qcRIBe.lWGwAAAEmb2kRH41T&OrderID=mL9IBe.lgiQAAAEmZGkRH41T&ProductID=6XdIBe.pudAAAAElYStRSbJV&FileName=/jdk-6u18-linux-i586.bin
[software]# mv jdk-6u18-linux-i586.bin\?AuthParam\=1265091604_6c594bd3bb701613a5c20291632bd64a\&TicketId\=B%2Fw4lRWBSFNCTRNFM19TlQ%2Fm\&GroupName\=CDS\&FilePath\=%2FESD6%2FJSCDL%2Fjdk%2F6u18-b07%2Fjdk-6u18-linux-i586.bin\&File\=jdk-6u18-linux-i586.bin jdk-6u18-linux-i586.bin
[software]# chmod +x jdk-6u18-linux-i586.bin
[software]# ./jdk-6u18-linux-i586.bin
[software]# mv jdk1.6.0_18 ..
[software]# cd ..
[opt]# ln -s jdk1.6.0_18 java
[opt]# chown -R appadmin.appadmin jdk1.6.0_18


Ant

 [opt]# cd software/
[software]# wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.gz
[software]# tar xzvf apache-ant-1.7.1-bin.tar.gz
[software]# mv apache-ant-1.7.1 ..
[software]# cd ..
[opt]# ln -s apache-ant-1.7.1 ant
[opt]# chown -R appadmin.appadmin apache-ant-1.7.1

[opt]# emacs /etc/profile


export JAVA_HOME=/opt/java
export ANT_HOME=/opt/ant

PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH

Tomcat

 [opt]# cd software/
[software]# wget http://ftp.wayne.edu/apache/tomcat/tomcat-6/v6.0.24/bin/apache-tomcat-6.0.24.tar.gz
[software]# tar xzvf apache-tomcat-6.0.24.tar.gz
[software]# mv apache-tomcat-6.0.24 ..
[software]# cd ..
[opt]# ln -s apache-tomcat-6.0.24 tomcat6
[opt]# chown -R appadmin.appadmin apache-tomcat-6.0.24
[opt]# ln -s java javaTomcat6
[opt]# cd /etc/init.d/
[init.d]# emacs tomcat6

#!/bin/sh
#
# Startup script for the Tomcat Server
#
# chkconfig: - 86 14
# description: Tomcat
# processname:
# pidfile:
# config:
# Tomcat
# description: Starts and stops the Tomcat
# See how we were called.

export CATALINA_BASE="/opt/tomcat6"
export TOMCAT_NAME="tomcat6"
export JAVA_HOME="/opt/javaTomcat6"
export JAVA_OPTS="-server -Xms5M -Xmx400M -XX:MaxPermSize=120M"

export TOMCAT_HOME="/opt/tomcat6"
export CATALINA_HOME="/opt/tomcat6"

export PATH="$PATH:$JAVA_HOME/bin:$GROOVY_HOME/bin"
export TOMCAT_USER="appadmin"

echo

cd $CATALINA_BASE

case "$1" in
  start)
    echo -n "Starting $TOMCAT_NAME Tomcat services: "
    echo
    if [ "$UID" -ne "0" ]; then
      $TOMCAT_HOME/bin/startup.sh
    else
      sudo -u $TOMCAT_USER $TOMCAT_HOME/bin/startup.sh
    fi
    echo
    ;;
  stop)
    echo -n "Shutting down $TOMCAT_NAME Tomcat services: "
    echo
    if [ "$UID" -ne "0" ]; then
      $TOMCAT_HOME/bin/shutdown.sh
    else
      sudo -u $TOMCAT_USER $TOMCAT_HOME/bin/shutdown.sh
    fi

    #loop until the program is dead
    waitingForExit=true
    iterator=1

    while [ "$waitingForExit" == "true" ]; do
      processId=`ps -ef | grep "$JAVA_HOME/bin/java" | grep -v "grep" | cut -c10-15`
      if [ -n "$processId" ]; then
        echo Waiting for exit...
      else
        waitingForExit=false
      fi

      if [ "$iterator" -gt "10" ]; then
        waitingForExit=false
      fi

      let "iterator += 1"
      sleep 1
    done


    #doesnt work since it kills are javas
    #killall -e $JAVA_HOME/bin/java
    killAgain=true
    while [ "$killAgain" == "true" ]; do
      processId=`ps -ef | grep "$JAVA_HOME/bin/java" | grep -v "grep" | cut -c10-15`
      if [ -n "$processId" ]; then
        echo "Killing the tomcat abruptly"

        if [ "$UID" -ne "0" ]; then
          kill -9 $processId
        else
          sudo -u $TOMCAT_USER kill -9 $processId
        fi

        sleep 1
      else
        killAgain=false
      fi
    done

    echo
    ;;
  status)
    echo | ps -ef | grep $JAVA_HOME | grep -v "grep" | grep $TOMCAT_USER
    ;;
  restart)
    echo -n "Restarting Tomcat $TOMCAT_NAME services: "
    $0 stop
    sleep 5
    $0 start
    echo "done."
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
esac

[init.d]# chkconfig --add tomcat6
[init.d]# chkconfig --levels 345 tomcat6 on
[init.d]# chmod +x tomcat6
Test starting
\[appadmin tomcat6\]$ /sbin/service tomcat6 start
\[appadmin tomcat6\]$ netstat \-an \| grep 8080
\[appadmin tomcat6\]$ /sbin/service tomcat6 stop
\[appadmin tomcat6\]$ cat /opt/tomcat6/logs/catalina.out

Edit the <Connectors to have uri encoding of utf8
<Connector  URIEncoding="UTF-8"

MySQL

[software]# wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-5.1.43-1.glibc23.i386.rpm/from/http://mysql.he.net/
[software]# rpm -Uvh MySQL-server-5.1.43-1.glibc23.i386.rpm
[software]# wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-5.1.43-1.glibc23.i386.rpm/from/http://mysql.llarian.net/
[software]# rpm -Uvh MySQL-client-5.1.43-1.glibc23.i386.rpm
[software]# /usr/bin/mysqladmin -u root password 'somesecret'
[software]# /usr/bin/mysqladmin -u root -h domU-12-31-39-03-CC-16 password 'somesecret'
[software]# netstat -an | grep 3306
[software]# /sbin/service mysql stop
[software]# /sbin/service mysql start
[software]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.43 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database grouper;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'grouper'@'localhost' identified by 'somesecret';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on grouper.* to 'grouper'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql>


Note: you can tunnel the 3306 port from ec2 to local through putty, and connect with sqlyog or another mysql gui UI


Grouper API

[software]# wget http://www.internet2.edu/grouper/release/1.5.1/grouper.binary.1.5.1.tar.gz
[opt]# cd ..
[software]# mkdir grouper
[software]# cd grouper
[grouper]# mkdir 1.5.1
[grouper]# cd 1.5.1
[1.5.1]# tar xzvf /opt/software/grouper.binary.1.5.1.tar.gz
[1.5.1]# cd /opt/grouper/1.5.1/grouper.binary.1.5.1/conf
[conf]# emacs grouper.hibernate.properties

hibernate.dialect               = org.hibernate.dialect.MySQL5Dialect

hibernate.connection.driver_class = com.mysql.jdbc.Driver

hibernate.connection.url = jdbc:mysql://localhost:3306/grouper

hibernate.connection.username         = grouper

hibernate.connection.password         = someSecret

[conf]# cd ../bin
[bin]# ./gsh.sh -registry -runscript
[bin]# ./gsh.sh

[opt]# chown -R appadmin.appadmin /opt/grouper

Mod JK (or mod proxy ajp below)

[software]# wget http://download.nextag.com/apache/tomcat/tomcat-connectors/jk/source/jk-1.2.28/tomcat-connectors-1.2.28-src.tar.gz
[software]# tar xzvf tomcat-connectors-1.2.28-src.tar.gz
[software]# cd tomcat-connectors-1.2.28-src/native
[native]# ./configure --with-apxs=/opt/apache/bin/apxs
[native]# make
[native]# make install

[appadmin modules]$ emacs /opt/apache/conf/httpd.conf

LoadModule jk_module modules/mod_jk.so

Include /opt/apache/conf/extra/httpd-jk.conf

[appadmin modules]$ emacs /opt/apache/conf/extra/httpd-jk.conf

JkWorkersFile /opt/apache/conf/workers.properties
JkLogFile /opt/apache/logs/mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"

[appadmin modules]$ emacs /opt/apache/conf/extra/httpd-grouper.conf

JkMount /grouper/* tomcat6

[appadmin modules]$ emacs /opt/apache/conf/extra/httpd-ssl.conf

  Include /opt/apache/conf/extra/httpd-grouper.conf

</VirtualHost>

[appadmin modules]# /sbin/service httpd restart
[appadmin modules]# /sbin/service tomcat6 restart



Mod Proxy AJP (or mod jk above)

Note, this was already install in apache
[appadmin@i2midev1 logs]$ more /etc/httpd/conf.d/proxy_ajp.conf

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

ProxyPass /grouper_v1_6/ ajp://localhost:8009/grouper_v1_6/

#############################
Make sure request.tomcatAuthentication="false" is in the tomcat server.xml

   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" request.tomcatAuthentication="false" />


Shibboleth

GrouperUI

[bin]# cd software
[software]# wget http://www.internet2.edu/grouper/release/1.5.1/grouper-ui-1.5.1.tar.gz
[software]# cd /opt/grouper/1.5.1/
[1.5.1]# tar xzvf /opt/software/grouper-ui-1.5.1.tar.gz
[software]# cd grouper-ui-1.5.1/
[grouper-ui-1.5.1]# ant
exit
[grouper-ui-1.5.1]# emacs build.properties

grouper.folder=/opt/grouper/1.5.1/grouper.binary.1.5.1

[grouper-ui-1.5.1]# ant
war

[grouper-ui-1.5.1]# sudo su - appadmin
[appadmin]$ cd /opt/tomcat6/webapps/
[webapps]$ cp /opt/grouper/1.5.1/grouper-ui-1.5.1/dist/grouper.war /opt/tomcat6/webapps/
[webapps]$ emacs /opt/tomcat6/conf/tomcat-users.xml

  <role rolename="grouper_user"/>
  <user username="GrouperSystem" password="GrouperSystem1" roles="grouper_user"/>

Test with a browser: https://ec2testhyzer.info/grouper/, login as GrouperSystem / someSecret

[opt]# chown -R appadmin.appadmin /opt/grouper

Grouper WS

[software]# wget http://www.internet2.edu/grouper/release/1.5.1/grouper-ws-1.5.1.tar.gz
[software]# cd ../grouper/1.5.1/
[1.5.1]# tar xzvf /opt/software/grouper-ws-1.5.1.tar.gz
[1.5.1]# cd grouper-ws-1.5.1/grouper-ws
[grouper-ws]# ant
  will fail
[grouper-ws]# emacs build.properties

grouper.dir=/opt/grouper/1.5.1/grouper.binary.1.5.1

[grouper-ws]# ant
[grouper-ws]# su - appadmin
[appadmin opt]$ cp /opt/grouper/1.5.1/grouper-ws-1.5.1/grouper-ws/build/dist/grouper-ws.war /opt/tomcat6/webapps/

[appadmin opt]$ emacs /opt/apache/conf/extra/httpd-grouper.conf

JkMount /grouper-ws/* tomcat6

[grouper-ws]# /sbin/service httpd graceful

Add a test group

[grouper-ws]# cd /opt/grouper/1.5.1/grouper.binary.1.5.1/bin
[bin]# ./gsh.sh
Using GROUPER_HOME: /opt/grouper/1.5.1/grouper.binary.1.5.1/bin/..
Using GROUPER_CONF: /opt/grouper/1.5.1/grouper.binary.1.5.1/bin/../conf
Using JAVA: /opt/java/bin/java
using MEMORY: 64m-512m
Grouper starting up: version: 1.5.1, build date: 2010/01/22 09:58:41, env: <no label configured>
grouper.properties read from: /opt/grouper/1.5.1/grouper.binary.1.5.1/conf/grouper.properties
Grouper current directory is: /opt/grouper/1.5.1/grouper.binary.1.5.1/bin
log4j.properties read from:   /opt/grouper/1.5.1/grouper.binary.1.5.1/conf/log4j.properties
Grouper is logging to file:   /opt/grouper/1.5.1/grouper.binary.1.5.1/bin/../logs/grouper_error.log, at min level WARN for package: edu.internet2.middleware.grouper, based on log4j.properties
grouper.hibernate.properties: /opt/grouper/1.5.1/grouper.binary.1.5.1/conf/grouper.hibernate.properties
grouper.hibernate.properties: grouper@jdbc:mysql://localhost:3306/grouper
sources.xml read from:        /opt/grouper/1.5.1/grouper.binary.1.5.1/conf/sources.xml
sources.xml groupersource id: g:gsa
sources.xml jdbc source id:   jdbc: GrouperJdbcConnectionProvider
Type help() for instructions
gsh 0% addRootStem("test", "test");
stem: name='test' displayName='test' uuid='80e89fc568504d09adc99087f6d44e5f'
gsh 1% addGroup("test", "testGroup", "testGroup");
group: name='test:testGroup' displayName='test:testGroup' uuid='eb79ba0fe6dd4f849782b2823fae1bb1'
gsh 2% quit

[bin]#

Test the web services:

https://ec2testhyzer.info/grouper-ws/servicesRest/v1_5_000/groups?wsLiteObjectType=WsRestFindGroupsLiteRequest&groupName=test:testGroup&queryFilterType=FIND_BY_GROUP_NAME_APPROXIMATE

Login as: GrouperSystem / someSecret

[opt]# chown -R appadmin.appadmin /opt/grouper

Grouper client

[software]# wget http://www.internet2.edu/grouper/release/1.5.1/grouperClient.binary-1.5.1.tar.gz
[software]# cd /opt/grouper/1.5.1/
[1.5.1]# tar xzvf /opt/software/grouperClient.binary-1.5.1.tar.gz
[1.5.1]# cd grouperClient.binary-1.5.1/
[grouperClient.binary-1.5.1]# emacs grouper.client.properties


grouperClient.webService.url = https://ec2testhyzer.info/grouper-ws/servicesRest

grouperClient.webService.login = GrouperSystem

grouperClient.webService.password = someSecret

[grouperClient.binary-1.5.1]# java -jar grouperClient.jar --operation=findGroupsWs --queryFilterType=FIND_BY_GROUP_NAME_APPROXIMATE --groupName=test:testGroup
Index 0: name: test:testGroup, displayName: test:testGroup
[grouperClient.binary-1.5.1]#
[opt]# chown -R appadmin.appadmin /opt/grouper

Grouper loader

[conf]# emacs /opt/grouper/1.5.1/grouper.binary.1.5.1/conf/grouper-loader.properties

loader.autoadd.typesAttributes = true

[conf]# cd /opt/grouper/1.5.1/grouper.binary.1.5.1/bin
[bin]# gsh.sh
gsh 0% exit                (that loaded the types and attributes)

run this mysql:

[software]# mysql -uroot -p grouper
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.43 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE TABLE loader_example ( subject_id VARCHAR(100), source_id varchar(100) );

INSERT INTO loader_example (subject_id, source_id) VALUES ('GrouperSystem', 'g:isa');
COMMIT;

Open the UI: https://ec2testhyzer.info/grouper/

Go to group: test:testGroup

Edit it, add type: grouperLoader, edit attributes

grouperLoaderDbName: grouper
grouperLoaderScheduleType: CRON

(every minute, you wouldnt normally do this, just for testing)
grouperLoaderQuartzCron: 0 * * * * ?

grouperLoaderQuery: select subject_id, source_id as subject_source_id from loader_example
grouperLoaderType: SQL_SIMPLE

[opt]# chown -R appadmin.appadmin /opt/grouper

Kick off the loader as a background job:

[appadmin bin]$ cd /opt/grouper/1.5.1/grouper.binary.1.5.1/bin
[appadmin bin]$ nohup ./gsh.sh -loader &

Now, you can look in grouper_loader_log table, and see log entries every minute.
Also, you can remove the record from loader_example table, see the membership get removed
(in log table and UI, at the top of the minute), and you can add it back,
and see it get added back in at top of minute.

Configure OpenLDAP

LDAPPC

<?xml version="1.0" encoding="utf-8"?>

<ldappc>
  <grouper>
    <group-queries>

      <attribute-matching-queries>
        <attribute-list>
          <attribute name="name" value="test:testGroup2" />
        </attribute-list>
      </attribute-matching-queries>

    </group-queries>

    <groups
      structure="flat"
      root-dn="ou=groups,${edu.vt.middleware.ldap.base}"
      ldap-object-class="groupOfNames"
      ldap-rdn-attribute="cn"
      grouper-attribute="name">

      <group-members-dn-list list-object-class="groupOfNames" list-attribute="member" list-empty-value="" />

      <group-members-name-list list-object-class="eduMember" list-attribute="hasMember">
        <source-subject-name-mapping>
          <source-subject-name-map source="example" subject-attribute="cn" />
          <source-subject-name-map source="g:gsa" subject-attribute="name" />
        </source-subject-name-mapping>
      </group-members-name-list>

      <group-attribute-mapping ldap-object-class="groupOfNames">
        <group-attribute-map group-attribute="description" ldap-attribute="description" />
      </group-attribute-mapping>

    </groups>

    <memberships>
      <member-groups-list list-object-class="eduMember" list-attribute="isMemberOf" naming-attribute="name" />
    </memberships>

  </grouper>

  <source-subject-identifiers>
    <source-subject-identifier source="example" subject-attribute="cn">
      <ldap-search
        base="ou=people,${edu.vt.middleware.ldap.base}"
        scope="subtree_scope"
        filter="(cn={0})" />
    </source-subject-identifier>
  </source-subject-identifiers>

</ldappc>
[root]# cat /etc/rsyslog.conf
...
local4.*  /var/log/ldap.log

[root]# ps -ef | grep rsyslog
[root]# kill -HUP <pid of rsyslog>
[root]# cat /etc/openldap/ldap.conf
BASE    dc=grouper,dc=edu
URI     ldap://127.0.0.1:389
BINDDN  cn=Manager,dc=grouper,dc=edu

sdf