{warning} This document is a work in progress. {warning} h3. Getting Started with Real-Time Provisioning Real-time provisioning is the incremental provisioning of groups, stems, and memberships triggered from the Grouper change log. Incremental provisioning is distinguished from full provisioning in that only a single or subset of an attribute's values are provisioned. Real-time provisioning is available from the provisioning service provider (psp, formerly known as ldappcng) as of version 2.1.0. It should be possible to install and configure the psp for Grouper versions 1.6 and up. These instructions assume that Grouper subjects are already provisioned to your ldap directory, and makes use of the [vt-ldap|http://code.google.com/p/vt-middleware/wiki/vtldap] based ldap source adapter. These instructions were written on a Mac, other platforms should be similar. {toc} h4. Requirements * Grouper API (requires Java 6 and a database, details are [here|Grouper:Prerequisites]) * provisioning service provider (psp) * ldap directory h4. Install Install the provisioning service provider by copying jar files and example configuration files from the psp distribution to your Grouper API installation. Many jars are dependencies of the Shibboleth attribute resolver and may not be necessary in your deployment. h5. Install : Grouper Installer To install Grouper including the API, UI, WS, grouperClient, psp, etc., download and run the [Grouper:Grouper Installer]. As of version 2.1.0, the psp may be installed but not configured via the Grouper Installer. {noformat} curl http://www.internet2.edu/grouper/release/2.1.0/grouperInstaller.jar -O java -jar grouperInstaller.jar {noformat} h5. Install : Manual To install manually, download and unpack the psp, then copy jars and configuration files to your Grouper API installation. h6. Download and Unpack the PSP Download the PSP [here|http://www.internet2.edu/grouper/software.html] and unpack. The distribution name is of the form {{grouper.psp-${version}.tar.gz}} {noformat} curl http://www.internet2.edu/grouper/release/2.1.0/grouper.psp-2.1.0.tar.gz -O tar xzf grouper.psp-2.1.0.tar.gz {noformat} h6. Copy Jars Copy jars located in {{lib/custom}} from the psp distribution to the Grouper API installation. {noformat} cp -vR grouper.psp-2.1.0/lib/custom/ grouper.apiBinary-2.1.0/lib/custom/ {noformat} h6. Copy Example Configuration Files Copy example configuration files located in {{conf}} from the psp distribution to the Grouper API installation. Example psp configuration files are in directories named {{psp-example-\*}}. {noformat} cp -vR grouper.psp-2.1.0/conf/ grouper.apiBinary-2.1.0/conf/ {noformat} h5. Configuration Example : Grouper to LDAP This configuration example should apply to any ldap directory server. | examples | [psp-example-grouper-to-ldap|http://anonsvn.internet2.edu/viewvc/viewvc.py/i2mi/java-provisioning-provider/trunk/psp-example-grouper-to-ldap/src/test/resources/] | | DN structure | bushy | | {{member}} | member DNs | h5. Configuration Example : Grouper to Tivoli This configuration example targets an IBM Tivoli Directory Server with requirements from Penn State. | examples | [psp-example-grouper-to-tivoli|http://anonsvn.internet2.edu/viewvc/viewvc.py/i2mi/java-provisioning-provider/trunk/psp-example-grouper-to-tivoli/src/test/resources/] | | DN structure | flat | | {{member}} | member subject ids | | {{memberOf}} | group DNs | | {{hasMember}} | member names | | {{isMemberOf}} | group names | h5. Configuration Example : Grouper to OpenLDAP This configuration example applies to OpenLDAP. | examples | [psp-example-grouper-to-openldap|http://anonsvn.internet2.edu/viewvc/viewvc.py/i2mi/java-provisioning-provider/trunk/psp-example-grouper-to-openldap/src/test/resources/] | | DN structure | bushy | | {{member}} | member DNs | | {{hasMember}} | member names | | {{isMemberOf}} | group names | h5. Configuration Example : Grouper to Active Directory h5. Configuration Example : LDAP to Grouper h4. Configuration : SPMLv2 h5. Provisioned Objects Provisioned objects, in SPMLv2 terms, consist of identifiers, attributes (probably), and references (maybe) to the identifiers of other objects, which are most likely located on the same provisioning target. h5. Identifiers Identifiers consist of a string ID, a target ID, and possibly a container ID. We consider a container ID to be similar to an ldap base dn. A container ID is itself an identifier, recursing potentially indefinitely. h5. Attributes Name value pairs. Probably multi-valued. Case sensitive names and values. We return values in the same order as they were given to us. h5. References A reference refers to the identifier of another object. It consists of two identifiers, the "from object" and the "to object". A node in a directed graph. Directional. h4. Configuration : Grouper h5. Configure LDAP Connection in {{ldap.properties}} Configure the default search base DN to match your directory : {noformat} edu.vt.middleware.ldap.baseDn = dc=example,dc=edu {noformat} Configure authentication and encryption : {noformat} edu.vt.middleware.ldap.serviceUser=cn=Manager,dc=example,dc=edu edu.vt.middleware.ldap.serviceCredential=secret {noformat} Configure the default base DN (container) for people and groups : {noformat} # The base DN for groups. edu.internet2.middleware.psp.groupsBaseDn = ou=groups,dc=example,dc=edu # The base DN for people. edu.internet2.middleware.psp.peopleBaseDn = ou=people,dc=example,dc=edu {noformat} h5. Configure Grouper Stem to be Provisioned in {{ldap.properties}} Configure the name of the Grouper stem to be provisioned, by default this is the root stem, which is the empty string. {noformat} # The base Grouper stem to be provisioned. edu.internet2.middleware.psp.baseStem= {noformat} h5. Configure LDAP Connection in {{sources.xml}} Configure Grouper to look for subjects in your LDAP directory by using the ldap source adapter. Connection information is defined in {{ldap.properties}}. {noformat} <source adapterClass="edu.internet2.middleware.subject.provider.LdapSourceAdapter"> <id>ldap</id> <name>LdapSourceAdapter</name> <type>person</type> <init-param> <param-name>ldapProperties_file</param-name> <param-value>ldap.properties</param-value> </init-param> {noformat} Configure the base DN to match your directory in the various search configuration elements : {noformat} <param-name>base</param-name> <param-value>ou=people,dc=example,dc=edu</param-value> {noformat} h5. Configure Ldap Connection in the Grouper UI Copy {{sources.xml}} and {{ldap.properties}} from the Grouper API to the Grouper UI. {noformat} cp grouper.apiBinary-2.1.0/conf/ldap.properties grouper.ui-2.1.0/dist/grouper/WEB-INF/classes/ cp grouper.apiBinary-2.1.0/conf/sources.xml grouper.ui-2.1.0/dist/grouper/WEB-INF/classes/ {noformat} Copy {{vt-ldap.jar}} from the Grouper API to the Grouper UI : {noformat} cp grouper.apiBinary-2.1.0/lib/custom/vt-ldap-3.3.4.jar grouper.ui-2.1.0/dist/grouper/WEB-INF/lib {noformat} {note} The Grouper UI will not load unless you edit {{ldap.properties}} in your Grouper UI installation or copy {{psp-ldap-target-2.1.0-SNAPSHOT.jar}} to your Grouper UI installation. {note} Comment out or remove the psp specific search result handlers in {{ldap.properties}} in your Grouper UI installation : {noformat} # edu.vt.middleware.ldap.searchResultHandlers=edu.internet2.middleware.psp.ldap.QuotedDnResultHandler,... {noformat} h5. Configure LDAP Structure in {{ldap.properties}} The LDAP structure and corresponding group RDN source attribute ID are configurable via macro replacement in {{ldap.properties}}. The {{flat}} or {{bushy}} structure is important when calculating ldap DNs and RDNs in the attribute resolver configuration {{psp-resolver.xml}}. {code:xml} <!-- Group identifier and attributes. --> <!-- The LDAP DN of a group. For example, "cn=groupExtension,ou=stem,ou=groups,dc=edu". --> <resolver:AttributeDefinition id="groupDn" xsi:type="psp-grouper-ldap:LdapDnFromGrouperNamePSOIdentifier" structure="${edu.internet2.middleware.psp.structure}" sourceAttributeID="name" rdnAttributeName="cn" base="${edu.internet2.middleware.psp.groupsBaseDn}"> <resolver:Dependency ref="GroupDataConnector" /> <resolver:Dependency ref="DeleteGroupChangeLogDataConnector" /> <resolver:Dependency ref="UpdateGroupChangeLogDataConnector" /> </resolver:AttributeDefinition> <!-- The value of the group "cn" attribute is the group extension. --> <!-- If the group DN structure is "bushy" the sourceAttributeID should be "extension". --> <!-- If the group DN structure is "flat" the sourceAttributeID should be "name". --> <resolver:AttributeDefinition id="cn" xsi:type="ad:Simple" sourceAttributeID="${edu.internet2.middleware.psp.cnSourceAttributeID}"> <resolver:Dependency ref="GroupDataConnector" /> </resolver:AttributeDefinition> {code} h6. Flat In a {{flat}} structure all groups are provisioned under a single base DN (container ID). A {{flat}} group's ldap RDN is its Grouper name or displayName. !grouper-rtp-flat.png! Configure the {{flat}} LDAP structure and {{name}} group RDN source attribute ID in {{ldap.properties}} : {noformat} edu.internet2.middleware.psp.structure=flat edu.internet2.middleware.psp.cnSourceAttributeID=name {noformat} h6. Bushy In a {{bushy}} structure groups are provisioned hierarchically, with stems as branches (ldap organizationalUnits) in the tree. A {{bushy}} group's RDN is its Grouper extension or displayExtension. !grouper-rtp-bushy.png! Configure the {{bushy}} LDAP structure and {{extension}} group RDN source attribute ID in {{ldap.properties}} : {noformat} edu.internet2.middleware.psp.structure=bushy edu.internet2.middleware.psp.cnSourceAttributeID=extension {noformat} h5. Configure Grouper Change Log To enable change log provisioning : {noformat} changeLog.consumer.psp.class = edu.internet2.middleware.psp.grouper.PspChangeLogConsumer {noformat} To schedule when the change log is processed : {noformat} changeLog.consumer.psp.quartzCron = 0 * * * * ? {noformat} To run full synchronizations periodically (by default every day at 5am) : {noformat} changeLog.psp.fullSync.class = edu.internet2.middleware.psp.grouper.PspChangeLogConsumer changeLog.psp.fullSync.quartzCron = 0 0 5 * * ? {noformat} To run a full synchronization job at loader startup : {noformat} changeLog.psp.fullSync.runAtStartup = true {noformat} The Quartz cron string documentation is [here|http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger]. h5. Tune Subject API Cache Tune Subject API caching in {{grouper.ehcache.xml}}. Tune {{maxElementsInMemory}} to be greater than or equal to the number of subjects. Tune {{timeToIdleSeconds}} and {{timeToLiveSeconds}} ... As of Grouper version 2.1.0, which uses Ehcache 2.4, {{statistics}} needs to be {{"true"}}. {code:xml} <!-- Subject resolving caching --> <!-- @see CachingResolver#find(...) --> <cache name="edu.internet2.middleware.grouper.subj.CachingResolver.Find" maxElementsInMemory="5000" eternal="false" timeToIdleSeconds="30" timeToLiveSeconds="120" overflowToDisk="false" statistics="true" /> <!-- @see CachingResolver#findAll(...) --> <cache name="edu.internet2.middleware.grouper.subj.CachingResolver.FindAll" maxElementsInMemory="5000" eternal="false" timeToIdleSeconds="30" timeToLiveSeconds="120" overflowToDisk="false" statistics="true" /> <!-- @see CachingResolver#findByIdentifier(...) --> <cache name="edu.internet2.middleware.grouper.subj.CachingResolver.FindByIdentifier" maxElementsInMemory="5000" eternal="false" timeToIdleSeconds="30" timeToLiveSeconds="120" overflowToDisk="false" statistics="true" /> <!-- @see CachingResolver#findByIdOrIdentifier(...) --> <cache name="edu.internet2.middleware.grouper.subj.CachingResolver.FindByIdOrIdentifier" maxElementsInMemory="5000" eternal="false" timeToIdleSeconds="30" timeToLiveSeconds="120" overflowToDisk="false" statistics="true" /> {code} h5. Configure Grouper Logging You may want to change the Grouper log file appenders in {{grouper.apiBinary-2.1.0/conf/log4j.properties}}. {noformat} log4j.appender.grouper_error = org.apache.log4j.DailyRollingFileAppender log4j.appender.grouper_error.File = ${grouper.home}logs/grouper_error.log log4j.appender.grouper_error.DatePattern = '.'yyyy-MM-dd log4j.appender.grouper_event = org.apache.log4j.DailyRollingFileAppender log4j.appender.grouper_event.File = ${grouper.home}logs/grouper_event.log log4j.appender.grouper_event.DatePattern = '.'yyyy-MM-dd {noformat} h5. Grouper Versions Prior to 2.1.0 {note} The following changes are necessary to support provisioning Grouper API versions prior to 2.1.0 with the psp. {note} 1. For versions prior to 2.1.0, there is a bug which will throw a NullPointerException if the following is not present in {{sources.xml}} : {noformat} <search> <searchType>searchSubjectByIdentifierAttributes</searchType> <param> <param-name>filter</param-name> <param-value> (&(uid=%TERM%)(objectclass=person)) </param-value> </param> <param> <param-name>scope</param-name> <param-value>SUBTREE_SCOPE</param-value> </param> <param> <param-name>base</param-name> <param-value>ou=people,dc=example,dc=edu</param-value> </param> </search> {noformat} 2. For versions prior to 2.1.0, the location of {{ldap.properties}} specified in {{sources.xml}} must be an absolute path. For versions 2.1.0 or later, the location of {{ldap.properties}} may be an absolute path or in your Java classpath. For example, Grouper API version 2.0.3 requires an absolute path to ldap.properties in {{sources.xml}} : {noformat} <init-param> <param-name>ldapProperties_file</param-name> <param-value>/opt/grouper/2.0.3/grouper.apiBinary-2.0.3/conf/ldap.properties</param-value> </init-param> {noformat} For Grouper UI versions prior to 2.1.0, the path to {{ldap.properties}} specified in {{sources.xml}} will be different than in the Grouper API since the psp specific search result handlers must be commented out or removed in the Grouper UI : {noformat} <init-param> <param-name>ldapProperties_file</param-name> <param-value>/opt/grouper/2.0.3/grouper.ui-2.0.3/dist/grouper/WEB-INF/classes/ldap.properties</param-value> </init-param> {noformat} 3. For Grouper API versions prior to 2.1.0, the ldap source adapter in {{subject.jar}} does not provide the method which allows the psp to re-use the same ldap connection as the subject source. You will need to copy {{lib/grouper/subject.jar}} from the Grouper 2.1.0 API distribution to your pre-2.1.0 Grouper API installation. h4. Configure : Provisioning Service Provider The psp configuration files are : | {{psp.xml}} | Maps source attributes returned by a Shibboleth attribute resolver to target objects. | | {{psp-resolver.xml}} | The Shibboleth attribute resolver configuration file. | | {{psp-services.xml}} | Configures Shibboleth attribute resolver services, including provisioning targets. | | {{psp-internal.xml}} | Bootstraps the Shibboleth attribute resolver. | The configuration files for Grouper ldap subjects are : | {{sources.xml}} | Configures Grouper subjects. | | {{ldap.properties}} | The vt-ldap configuration. Also used for macro replacement in the psp configuration files. | h5. Configure PSP : Provisioned Objects, Identifiers, Attributes and References h5. Configure PSP : Attribute Resolver h4. Provision Grouper Before you can provision anything from Grouper to ldap or anywhere else, you will need to create the corresponding objects in Grouper using the UI, API, GSH, WS, loader, import, etc. h5. Provision Grouper : GSH To calculate how a group should be provisioned : {noformat} bin/gsh.sh -psp -calc edu:group {noformat} To diff the current and correct provisioning of a group : {noformat} bin/gsh.sh -psp -diff edu:group {noformat} To provision or synchronize a group : {noformat} bin/gsh.sh -psp -sync edu:group {noformat} h5. Provision Grouper : Grouper Change Log To provision in real-time triggered by the Grouper change log, enable the psp consumer in {{grouper-loader.properties}} and run the loader via {noformat} bin/gsh.sh -loader {noformat} h3. Real-Time Provisioning Beta-Testing h4. Grouper Subject Sources || Institution || Subject Source || Number of Subjects || Subject ID || | LIGO | LDAP | 1,000 | dn: +employeeNumber=882+,ou=people,dc=ligo,dc=org \\ | | Penn State | LDAP | 165,000 | dn:+uid=xyx123+,dc=psu,dc=edu | | UCLA | LDAP | 40,000 | | | UMontreal | LDAP | 120,000 | sAMAccountName (value same as cn) | | UVienna | Undecided | 155,000 | cn, uid | | UWMadison | | | | h4. Provisioning Targets || Institution || Target || Implementation || | LIGO | LDAP | OpenLDAP 2.4.x | | Penn State | LDAP | IBM Tivoli Directory Server | | UCLA | LDAP | Sun Java System Directory Server Enterprise Edition 6.3.1 | | UMontreal | LDAP | Active Directory | | UVienna | LDAP | Active Directory, OpenLDAP | | UWMadison | | | h4. Provisioning memberOf The groups that a member _is a member of_ may be provisioned to the memberOf attribute. Some LDAP implementations, such as Active Directory, automatically maintain the memberOf attribute. OpenLDAP maintains the memberOf attribute automatically via the memberOf overlay. The value of the memberOf attribute is typically a group DN. || Institution || memberOf for members (people) || memberOf for groups || | LIGO | + | | | Penn State | + | | | UCLA | \- | | | UMontreal | automatic (Active Directory) | automatic (Active Directory) | | UVienna | automatic (Active Directory), OpenLDAP+memberOf | automatic (Active Directory), OpenLDAP+memberOf | | UWMadison | + | | h4. Provisioning eduMember The [eduMember|http://middleware.internet2.edu/dir/docs/internet2-mace-dir-ldap-group-membership-200507.html] objectClass defines the isMemberOf and hasMember attributes, whose values are identifiers which are not DNs. || Institution || isMemberOf || hasMember || | LIGO | + | + | | Penn State | + | + | | UCLA | uclaIsMemberOf | uclaHasMember | | UMontreal | \- | \- | | UVienna | \- | \- | | UWMadison | + | + | h4. Provisioning eduCourse The [eduCourse|http://middleware.internet2.edu/courseid/docs/internet2-mace-dir-courseid-educourse-ldap-200507.html] objectClass defines course related attributes. || Institution || eduCourse || | LIGO | \- | | Penn State | \- | | UCLA | \- | | UMontreal | \- | | UVienna | \- | | UWMadison | + | h4. Provisioning Structure The group provisioning structure may be either {{flat}} or {{bushy}}. A {{flat}} structure provisions all groups into a single container. A {{bushy}} structure provisions groups hierarchically. For example, the DN of a group with name 'edu:stem:group' in a {{flat}} structure looks like : dn: cn=edu:stem:group,ou=groups,dc=example,dc=edu while the DN of a group with name 'edu:stem:group' in a {{bushy}} structure looks like : dn: cn=group,ou=stem,ou=edu,ou=groups,dc=example,dc=edu || Institution || Structure (flat or bushy) || | LIGO | bushy | | Penn State | flat | | UCLA | flat | | UMontreal | bushy | | UVienna | ? | | UWMadison | flat | h4. Membership Structure Given groupA with memberA and groupB with memberB : {code} dn : cn=groupA,ou=groups member: cn=memberA,ou=people dn: cn=groupB,ou=groups member: cn=memberB,ou=people {code} If groupB is added as a member to groupA, how do you want groupA to be provisioned : everything : {code} dn : cn=groupA,ou=groups member: cn=memberA,ou=people member: cn=memberB,ou=people member: cn=groupB,ou=people {code} immediate : {code} dn : cn=groupA,ou=groups member: cn=memberA,ou=people member: cn=groupB,ou=people {code} The everything membership structure handles applications which may not support nested groups and represents the nested structure of the group memberships. The same membership structure applies to memberOf : everything : {code} dn: cn=memberB,ou=people memberOf: cn=groupB,ou=groups memberOf: cn=groupA,ou=groups {code} immediate : {code} dn: cn=memberB,ou=people memberOf: cn=groupB,ou=groups {code} {code} {code} || Institution || member || memberOf || | LIGO | everything | everything | | Penn State | | | | UCLA | | | | UMontreal | immediate | immediate | | UVienna | everything | everything | | UWMadison | | | |