See Also Grouper Provisioning Strategy
See Also Grouper SQL database provisioning
We will make a SQL provisioner in Grouper v2.5.
Configuration
Common config attributes for LDAP are below.
Config | Example | Description | Notes |
---|---|---|---|
class | edu.internet2.middleware.grouper.app.sqlProvisioning.SqlMembershipProvisioner | Class extends the base provisioner class | This class informs configuration decisions. Required. Read-only. |
hasSubjectLink | true false | If the subject API is needed to resolve attribute on subject | required, drives requirements of other configurations. defaults to false. |
hasTargetUserLink | true false | If subjects need to be resolved in the target before provisioning | defaults to false. required. |
hasTargetGroupLink | true false | If groups need to be resolved in the target before provisioning | defaults to false. required. |
subjectSourcesToProvision | pennperson | subject sources to provision | required. defaults to all except g:gsa, grouperExternal, g:isa, localEntities. comma separated list. checkboxes. |
userSearchTableName | users | table to query to lookup users | required if hasTargetUserLink |
userSearchAttributeName | employee_id | column to filter on | required if hasTargetUserLink |
userSearchAttributeValueFormat | ${subject.id} | value for the user search attribute name | required if hasTargetUserLink |
membershipTableName | memberships | table where memberships go | required |
syncMemberToId2AttributeValueFormat | ${targetEntity.attributes['user_id']} | main identifier of the user on the target side | show = false |
syncMemberToId3AttributeValueFormat | ${targetEntity.attributes['uid']} | identifier of the user as referred to by the membership | show = false |
syncMemberFromId2AttributeValueFormat | ${targetEntity.attributes['netId']} | target attribute value that helps look up user | show = false |
syncMemberFromId3AttributeValueFormat | ${subject.attributes['myLdapId']} | subject attribute value that helps look up user | show = false |
syncGroupToId2AttributeValueFormat | ${targetGroup.attributes['group_id']} | main identifier of the group on the target side | show = false |
syncGroupToId3AttributeValueFormat | ${targetEntity.attributes['gid']} | identifier of the group as referred to by the membership | show = false |
syncGroupFromId2AttributeValueFormat | ${targetEntity.attributes['groupName']} | target attribute value that helps look up group | show = false |
syncGroupFromId3AttributeValueFormat | show = false | ||
userSearchAttributes | user_id, name, email | columns to search when getting users | optional. show if hasTargetUserLink. |
createMissingUsers | true or false | defaults false, optional. show if hasTargetUserLink | |
createMissingGroups | true or false | defaults to true. show if hasTargetGroupLink | |
groupSearchAttributeName | gidNumber | attribute name to filter on | show if hasTargetGroupLink required |
groupSearchAttributeValueFormat | ${syncGroup.groupIdIndex} | value to filter group on | show if hasTargetGroupLink required |
groupSearchAttributes | cn,gidNumber,samAccountName,objectclass | attributes to get if searching for groups | optional show if hasTargetGroupLink |
groupAttributesMultivalued | someAttr | everything is assumed to be single valued except object class. List attributes in the groupSearchAttribute which are multivalued | optional. show if hasTargetGroupLink |
deleteInTargetIfInTargetAndNotGrouper | true or false | if groups in full sync should be deleted if in group all filter and not in grouper or for attributes delete other attribute not provisioned by grouper | default to false |
deleteInTargetIfDeletedInGrouper | true or false | if groups that were created in grouper were deleted should it be deleted in ldap? or for attributes, delete attribute value if deleted in grouper | default to true |
membershipFields | members read,admin update,admin admin | if provisioning normal memberships or privileges | default to "members" for normal memberships |
configName must match the provisioner config name. Example config in grouper-loader.properties is provisioner.<configName>.ldapConfigId. Note you can also configure generic provisioning configuration
LDAP provisioner config
Type | Config | Example | Description | Notes |
---|---|---|---|---|
All | ldapExternalSystemConfigId | mySchoolAd | links to ldap external system config in grouper-loader.properties | required |
All | ldapProvisioningType | groupMemberships: group ldap object has attribute to hold memberships userAttributes: user ldap object has attribute to hold memberships | required, drives requirements of other configurations | |
All | isActiveDirectory | true or false | if this is active directory, it affects the defaults and logic | optional |
User | userSearchBaseDn | ou=users | where users are | optional. show if hasTargetUserLink |
User | userSearchFilter | employeeID=$userSearchAttributeValueFormat$ | how to find a user | optional. show if hasTargetUserLink |
User | userSearchAllFilter | employeeID=* | filter users when searching all (needed?) | optional. show if hasTargetUserLink |
User | userObjectClass | person | if want to specify person object class | optional. show if hasTargetUserLink |
User | userCreationParentDn | ou=users | baseDn to create users (is this appended to ldap base dn if not included?) | required. show if createMissingUsers |
User | userCreationNumberOfAttributes | integer between 1 and 10 | required. show if createMissingUsers | |
User | userCreationLdifTemplate_attr_0 | cn | the 0th attribute name | required if createMissingUsers |
User | userCreationLdifTemplate_val_0 | ${syncMember.memberToId2} | the 0th attribute value | required if createMissingUsers |
User | userCreationLdifTemplate_attr_1 | objectclass | the 1st attribute name | required if createMissingUsers |
User | userCreationLdifTemplate_val_1 | person | the 1st attribute value | required if createMissingUsers |
Group | groupSearchBaseDn | ou=groups | optional. show if hasTargetGroupLink | |
Group | groupSearchFilter | (&(objectclass=group)(gidNumber=${syncGroup.groupIdIndex})) -or- dn=${syncGroup.groupToId2} | find a single group (other than by DN) | optional. show if hasTargetGroupLink |
Group | groupsSearchAllFilter | (&(objectclass=group)(gidNumber=*)) | find all groups | optional. show if hasTargetGroupLink |
Group | groupDnType | bushy or flat | bushy makes an ou for each folder and cn is extension. flat is cn is group name | required flat dn will be: |
Group | groupObjectClass | groupOfUniqueNames | default object class | optional |
Group | groupCreationNumberOfAttributes | integer between 1 and 10 | required. show if createMissingGroups | |
Group | groupCreationLdifTemplate_attr_[0 to 9] | cn | the 0th attribute name | required if createMissingGroups |
Group | groupCreationLdifTemplate_val_[0 to 9] | ${syncGroup.groupToId2} | the 0th attribute value | required if createMissingGroups |
Group | groupCreationLdifTemplate_maxLength_[0 to 9] | 900 | max length of this attribute, abbreviate after that (ellipses) | helpful e.g. for description in active directory |
Group User Attribute | provisionedAttributeName | attribute to provision to | attribute of person or group that has memberships | |
Group User Attribute | provisionedAttributeValueFormat | ${syncGroup.groupName} or ${syncMember.memberToId2} | how to get the value for the attribute | |
Attribute | allProvisionedValuesPrefix | prefix of attributes to manage. e.g. delete if not in grouper |
LDAP provisioning types
Discussion topics
Should we use DAO for LDAP so we can unit test everything but the LDAP stuff? Also for dry run?
Yes- User cache / group cache / subject cache
- Need a provisioner col for last retrieved
Update cache with nightly full sync, on error, refresh subject and ldap user from target, and try again - How daemon works
syncing and the cache updating. Note: it can use the subject source data from the nonblocking state inside the blocking state
1. Nonblocking, unless over a threshold. Few changes, start blocking after know changes, get the state from both sides for those changes, make the changes, unblock.
2. If over a certain threshold (need algorithm for number of objects: groups or users), block, and recalculate and sync - How cached fields stored in db? JSON? cols?
Keep a col for the lookups to query on, and a col for json representation of group and user - Store subject attributes? use member table? add a couple cols? identifier1, identifier2..5? update periodically?
Lookup attributes will be in the sync tables
- Need a provisioner col for last retrieved
- Pulling data from grouper, only certain columns?
Dont pull from hibernate, just use simple query like grouper_memberships_lw_v but without distinct (maybe groupId and memberId) - User sync?
Yes, we should plan for this - Folder sync
Do individual groups if its inside a threshold or full refresh from outside the threshold. If group deletes in ldap are needed, a full sync is needed - Selecting subjects
- Use the subject cache, and tune that as needed to fully prime
- If errors resolve subjects without cache
- Batch size, should be at ldap or provisioner level? do we need different for users and groups? e.g. userSearch_batchSize
Default batch size at LDAP level, override for provisioner. If over threshold, get all, otherwise, cycle through. Could leave "all" filter blank if known not to provision a lot of groups- Need a batch size for updates? How do batches work?
Theres no batch size for objects, one at a time. Attributes have batch size with default at ldap level - Can this be in the LDAP framework (and it will batch things up, thats what we do for jdbc) yes
- Need a batch size for updates? How do batches work?
- Supports empty groups, at ldap or provisioner level?
Could have null member, could delete group if no members. Default at ldap level. - Should errors be stored in the sync objects so they can be absorbed? yes
- Message for processing errors? yes and timestamp
- Group, user, membership errors? yes, list the error and skip (based on configuration)
- Search result paging is at ldap or provisioner level? defaults at ldap level
- Do we need an isActiveDirectory flag? keep this, TODO add back in
- Support dry run?
For grouper admins to test what will happen if configuration changes.
For a single group, do a "pretend is provisionable" in ui jvm...
For single group or user, this would just happen synchronously in ui, or have the ajax updating screen thing (upcoming) - Assume gid will not change? name? dn? How do we track groups that move? i.e. in the grouper_sync_group table?
The cached DN needs to not change until the provisioner can make sure theres a way to reference the old object to do updates and not delete/insert - Ldif templates, lets change the config of this. maybe json? or at least user $newline$? Need for jexl things? Escaping?
Yes, do this, and use $newline$ - Replacing grouperIsAuthoritative to be more explicit
Yes do this - Logging like change log consumer or database provisioning
Yes - Does memberOf and hasMember always use the DN of the user or group? generally yes
- Do we need to support updating group and user or will updating the group handle it? this is an attribute provisioner
- Convert to some simple object structure from ldap? or use the ldap objects for jexl? do we need these with the sync groups? We have a low-memory object structure
- Single valued attributes strings unless configured attributes are multivalued (use arrays since lightweight) Yes, single valued objects or array of strings
- Last updated columns in ldap? numbers: number of millis from epoch Not used
- User
- Bean with single and multivalued attribute Similar to pspng, bean with dn and name/value pair attributes
- Can we support getting memberships or privileges (e.g. ADMIN/READ) Yes, interface to get memberships
- Option to constrain to subject source? Yes
- Do we know the types of ldap attributes? Is there a metadata schema query? Shilen says generally strings
- Group DNs, can it be dynamic? Can it be handled upstream in provisioning framework? We would like it to happen in the provisioning framework. Note, complicates the all group filter
- Is memberOf or hasMember the only multi-valued attributes? do types of attributes matter? e.g. integer vs string? objectclass is multivalued, and memberships, other attributes are arrays of strings
LDAP operations
If batches fail, do each individually. This is the DAO implemented three ways: read/write to LDAP, mock for testing, and readonly dry run
Operation | Description |
---|---|
Create object | With DN and list of attributes |
Delete object | By DN |
Modify attribute batch | For one object by DN, modify an object |
Modify attribute indiv | If error on batch of attribute modifies, do individually |
Move object | From one DN to another |
Read | By DN with attributes |
Search | By filter with attributes |
Caching
Sync objects can cache information in LDAP. Synced from full sync (if doesnt exist or if errors), incremental (if doesnt exist or if errors), and the nightly (scheduled) subject resolution daemon (full refresh)
Object | Field | Cached data |
---|---|---|
gcGrouperSyncGroup | groupToId2 | group DN |
gcGrouperSyncGroup | groupToId3 | whatever attribute value the user attribute refers to |
gcGrouperSyncGroup | groupFromId2 | ldap group object attribute value that looks up group |
gcGrouperSyncMember | memberToId2 | user DN |
gcGrouperSyncMember | memberToId3 | whatever attribute value the group attribute refers to users as |
gcGrouperSyncMember | memberFromId2 | ldap person object attribute value that looks up user |
gcGrouperSyncMember | memberFromId3 | subject attribute value that helps look up user |
Group of unique names example
ldapProvisioner.myOpenLdap.ldapPoolName = myOpenLdapServer ldapProvisioner.myOpenLdap.ldapProvisioningType = groupMembershipsLdapLink ldapProvisioner.myOpenLdap.subjectSourcesToProvision = mySubjectSource ldapProvisioner.myOpenLdap.userSearchBaseDn = cn=users,dc=example,dc=edu ldapProvisioner.myOpenLdap.userSearchAttributeName = uid ldapProvisioner.myOpenLdap.userSearchAttributeValueFormat = ${syncMember.getSubjectId()} ldapProvisioner.myOpenLdap.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu ldapProvisioner.myOpenLdap.groupSearchAttributeName = gidNumber ldapProvisioner.myOpenLdap.groupSearchAttributeValueFormat = ${syncGroup.getGroupIdIndex()} ldapProvisioner.myOpenLdap.groupObjectClass = groupOfUniqueNames ldapProvisioner.myOpenLdap.provisionedAttributeName = uniqueMember ldapProvisioner.myOpenLdap.provisionedAttributeValueFormat = ${ldapUser.getDn()} ldapProvisioner.myOpenLdap.groupDnType = flat
This will assume the following:
Item | Description |
---|---|
userSearchFilter | (uid=${syncMember.getSubjectId()}) |
userSearchAllFilter | (uid=*) |
userSearchAttributes | uid,dn |
groupSearchAttributes | gidNumber,objectclass,cn,dn |
groupSearchFilter | (&(objectclass=groupOfUniqueNames)(cn=${syncGroup.getGroupName()}) |
groupsSearchAllFilter | (&(objectclass=groupOfUniqueNames)(cn=*)) |
groupCreationLdifTemplate | dn=${groupName},ou=grouper,ou=groups,dc=example,dc=edu cn=${groupName} objectclass=groupOfUniqueNames |
Active Directory groups example
ldapProvisioner.myActiveDirectory.ldapPoolName = myActiveDirectoryServer ldapProvisioner.myActiveDirectory.isActiveDirectory = true ldapProvisioner.myActiveDirectory.ldapProvisioningType = groupMembershipsLdapLink ldapProvisioner.myActiveDirectory.subjectSourcesToProvision = mySubjectSource ldapProvisioner.myActiveDirectory.userSearchBaseDn = cn=users,dc=example,dc=edu ldapProvisioner.myActiveDirectory.userSearchAttributeName = samAccountName ldapProvisioner.myActiveDirectory.userSearchAttributeValueFormat = ${syncMember.getSubjectIdentifier()} ldapProvisioner.myActiveDirectory.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu ldapProvisioner.myActiveDirectory.groupDnType = bushy ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_attr_0 = displayname ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_val_0 = ${group.getDisplayExtension()} ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_attr_1 = description ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_val_1 = ${group.getDescription()} ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_maxLength_1 = 900
This will assume the following:
Item | Description |
---|---|
userSearchFilter | (samAccountName=${syncMember.getSubjectIdentifier()}) |
userSearchAllFilter | (samAccountName=*) |
userSearchAttributes | samAccountName,dn |
groupObjectClass | group |
groupSearchAttributeName | gidNumber |
groupSearchAttributeValueFormat | ${syncGroup.getGroupIdIndex()} |
groupSearchAttributes | gidNumber,objectclass,cn,dn,displayname,description |
groupSearchFilter | (&(objectclass=groupObjectClass )(gidNumber=${syncGroup.getGroupIdIndex()}) |
groupsSearchAllFilter | (&(objectclass=groupObjectClass )(gidNumber=*)) |
groupCreationLdifTemplate | dn=${grouperUtil.convertNameToCnAndOus(groupName)},ou=grouper,ou=groups,dc=example,dc=edu |
provisionedAttributeName | member |
provisionedAttributeValueFormat | ${syncMember.getMemberToId2()} |
User attributes example
ldapProvisioner.openLdapUserAttributes.ldapPoolName = myOpenLdapServer ldapProvisioner.openLdapUserAttributes.ldapProvisioningType = userAttributesSimple ldapProvisioner.openLdapUserAttributes.subjectSourcesToProvision = mySubjectSource ldapProvisioner.openLdapUserAttributes.userSearchBaseDn = cn=users,dc=example,dc=edu ldapProvisioner.openLdapUserAttributes.userSearchAttributeName = uid ldapProvisioner.openLdapUserAttributes.userSearchAttributeValueFormat = ${syncMember.getSubjectId()} ldapProvisioner.openLdapUserAttributes.provisionedAttributeName = eduPersonEntitlement ldapProvisioner.openLdapUserAttributes.provisionedAttributeValueFormat = g:${group.name} ldapProvisioner.openLdapUserAttributes.allProvisionedValuesPrefix = g:
This will assume the following:
Item | Description |
---|---|
userSearchFilter | (uid=${syncMember.getSubjectId()}) |
userSearchAllFilter | (uid=*) |
userSearchAttributes | uid,eduPersonEntitlement |