Grouper membership SQL provisioner in v2.5

We will make a SQL provisioner in Grouper v2.5.

Configuration

Common config attributes for LDAP are below.

LDAP provisioning types

Discussion topics

• Should we use DAO for LDAP so we can unit test everything but the LDAP stuff?  Also for dry run?  
  Yes
  
  

• User cache / group cache / subject cache
  • Need a provisioner col for last retrieved
    Update cache with nightly full sync, on error, refresh subject and ldap user from target, and try again
  • How daemon works
    syncing and the cache updating.  Note: it can use the subject source data from the nonblocking state inside the blocking state
    1. Nonblocking, unless over a threshold.  Few changes, start blocking after know changes, get the state from both sides for those changes, make the changes, unblock.
    2. If over a certain threshold (need algorithm for number of objects: groups or users), block, and recalculate and sync
  • How cached fields stored in db?  JSON?  cols?
    Keep a col for the lookups to query on, and a col for json representation of group and user
  • Store subject attributes?  use member table?  add a couple cols?  identifier1, identifier2..5?  update periodically?
    Lookup attributes will be in the sync tables
• Pulling data from grouper, only certain columns?
  Dont pull from hibernate, just use simple query like grouper_memberships_lw_v but without distinct (maybe groupId and memberId)
• User sync?
  Yes, we should plan for this
• Folder sync
  Do individual groups if its inside a threshold or full refresh from outside the threshold.  If group deletes in ldap are needed, a full sync is needed
• Selecting subjects
  • Use the subject cache, and tune that as needed to fully prime
  • If errors resolve subjects without cache
• Batch size, should be at ldap or provisioner level?  do we need different for users and groups? e.g. userSearch_batchSize
  Default batch size at LDAP level, override for provisioner.  If over threshold, get all, otherwise, cycle through.  Could leave "all" filter blank if known not to provision a lot of groups
  
  • Need a batch size for updates?  How do batches work?
    Theres no batch size for objects, one at a time.  Attributes have batch size with default at ldap level
  • Can this be in the LDAP framework (and it will batch things up, thats what we do for jdbc)    yes
• Supports empty groups, at ldap or provisioner level?
  Could have null member, could delete group if no members.  Default at ldap level.
• Should errors be stored in the sync objects so they can be absorbed?   yes
  
  • Message for processing errors?  yes and timestamp
  • Group, user, membership errors?  yes, list the error and skip (based on configuration)
• Search result paging is at ldap or provisioner level?  defaults at ldap level
• Do we need an isActiveDirectory flag?  keep this, TODO add back in
• Support dry run? 
  For grouper admins to test what will happen if configuration changes.
  For a single group, do a "pretend is provisionable" in ui jvm...
  For single group or user, this would just happen synchronously in ui, or have the ajax updating screen thing (upcoming)
• Assume gid will not change?  name?  dn?  How do we track groups that move?  i.e. in the grouper_sync_group table?
  The cached DN needs to not change until the provisioner can make sure theres a way to reference the old object to do updates and not delete/insert
• Ldif templates, lets change the config of this.  maybe json?  or at least user $newline$?  Need for jexl things?   Escaping?
  Yes, do this, and use $newline$
• Replacing grouperIsAuthoritative to be more explicit
  Yes do this
• Logging like change log consumer or database provisioning
  Yes
• Does memberOf and hasMember always use the DN of the user or group?   generally yes
• Do we need to support updating group and user or will updating the group handle it?   this is an attribute provisioner
• Convert to some simple object structure from ldap?  or use the ldap objects for jexl?  do we need these with the sync groups? We have a low-memory object structure
  • Single valued attributes strings unless configured attributes are multivalued (use arrays since lightweight) Yes, single valued objects or array of strings
  • Last updated columns in ldap?  numbers: number of millis from epoch  Not used
  • User
    • Bean with single and multivalued attribute Similar to pspng, bean with dn and name/value pair attributes
• Can we support getting memberships or privileges (e.g. ADMIN/READ) Yes, interface to get memberships
• Option to constrain to subject source? Yes
• Do we know the types of ldap attributes?  Is there a metadata schema query? Shilen says generally strings
• Group DNs, can it be dynamic?  Can it be handled upstream in provisioning framework? We would like it to happen in the provisioning framework.  Note, complicates the all group filter
• Is memberOf or hasMember the only multi-valued attributes?  do types of attributes matter?  e.g. integer vs string?  objectclass is multivalued, and memberships, other attributes are arrays of strings

LDAP operations

If batches fail, do each individually.  This is the DAO implemented three ways: read/write to LDAP, mock for testing, and readonly dry run

Caching

Sync objects can cache information in LDAP.  Synced from full sync (if doesnt exist or if errors), incremental (if doesnt exist or if errors), and the nightly (scheduled) subject resolution daemon (full refresh)

Group of unique names example

ldapProvisioner.myOpenLdap.ldapPoolName = myOpenLdapServer
ldapProvisioner.myOpenLdap.ldapProvisioningType = groupMembershipsLdapLink
ldapProvisioner.myOpenLdap.subjectSourcesToProvision = mySubjectSource
ldapProvisioner.myOpenLdap.userSearchBaseDn = cn=users,dc=example,dc=edu
ldapProvisioner.myOpenLdap.userSearchAttributeName = uid
ldapProvisioner.myOpenLdap.userSearchAttributeValueFormat = ${syncMember.getSubjectId()}
ldapProvisioner.myOpenLdap.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu
ldapProvisioner.myOpenLdap.groupSearchAttributeName = gidNumber
ldapProvisioner.myOpenLdap.groupSearchAttributeValueFormat = ${syncGroup.getGroupIdIndex()}
ldapProvisioner.myOpenLdap.groupObjectClass = groupOfUniqueNames
ldapProvisioner.myOpenLdap.provisionedAttributeName = uniqueMember
ldapProvisioner.myOpenLdap.provisionedAttributeValueFormat = ${ldapUser.getDn()}
ldapProvisioner.myOpenLdap.groupDnType = flat

This will assume the following:

Active Directory groups example

ldapProvisioner.myActiveDirectory.ldapPoolName = myActiveDirectoryServer
ldapProvisioner.myActiveDirectory.isActiveDirectory = true
ldapProvisioner.myActiveDirectory.ldapProvisioningType = groupMembershipsLdapLink
ldapProvisioner.myActiveDirectory.subjectSourcesToProvision = mySubjectSource
ldapProvisioner.myActiveDirectory.userSearchBaseDn = cn=users,dc=example,dc=edu
ldapProvisioner.myActiveDirectory.userSearchAttributeName = samAccountName
ldapProvisioner.myActiveDirectory.userSearchAttributeValueFormat = ${syncMember.getSubjectIdentifier()}
ldapProvisioner.myActiveDirectory.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu
ldapProvisioner.myActiveDirectory.groupDnType = bushy
ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_attr_0 = displayname
ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_val_0 = ${group.getDisplayExtension()}
ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_attr_1 = description
ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_val_1 = ${group.getDescription()}
ldapProvisioner.myActiveDirectory.groupCreationLdifTemplate_maxLength_1 = 900

This will assume the following:

User attributes example

ldapProvisioner.openLdapUserAttributes.ldapPoolName = myOpenLdapServer
ldapProvisioner.openLdapUserAttributes.ldapProvisioningType = userAttributesSimple
ldapProvisioner.openLdapUserAttributes.subjectSourcesToProvision = mySubjectSource
ldapProvisioner.openLdapUserAttributes.userSearchBaseDn = cn=users,dc=example,dc=edu
ldapProvisioner.openLdapUserAttributes.userSearchAttributeName = uid
ldapProvisioner.openLdapUserAttributes.userSearchAttributeValueFormat = ${syncMember.getSubjectId()}
ldapProvisioner.openLdapUserAttributes.provisionedAttributeName = eduPersonEntitlement
ldapProvisioner.openLdapUserAttributes.provisionedAttributeValueFormat = g:${group.name}
ldapProvisioner.openLdapUserAttributes.allProvisionedValuesPrefix = g:

This will assume the following: