Grouper provisioners in Grouper 2.5 and above will be an "esb listener" and process EsbEvent objects.  However, unlike Grouper 2.4 listeners, the provisioner change log consumer will process multiple events at once, and not from a JSON representation.  This occurs in the provisioning framework.

Configure an incremental job

changeLog.consumer.<jobName>.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer
changeLog.consumer.<jobName>.quartzCron = 0 * * * * ?
changeLog.consumer.<jobName>.provisionerTarget = <provisionerConfigId>
changeLog.consumer.<jobName>.provisionerJobSyncType = incrementalProvisionChangeLog
changeLog.consumer.<jobName>.publisher.debug = false

Recalculate operations (stateful vs stateless)

When events are processed by the provisioning framework, they are one of two types:

  1. Without recalculate (stateful)
    1. Take an "add member 123 to group a:b:c" and send that operation to the target
    2. This will check the state in grouper (is the user a member), and the cached target state (does grouper think the user is a member in the target group?)
      1. If the Grouper state doesnt match the event, e.g. an add member and the user is not a member of the group), this will be converted into a recalc event
      2. If the cached state of the target doesnt match the event, e.g. if grouper thinks the user is already a member of the group in the target, and its a "add member" event, convert the event to a recalc event
    3. This is used by default for change log events
    4. Best performance, since there are fewer target operations
    5. Edge cases could produce temporary incorrect results (the target could be blissfully out of sync until the next full sync)
  2. Recalculate (stateless)
    1. There is no "add member" event, there is "check group a:b:c for subject 123". 
    2. Check the state on the target, compare with grouper, and issue the correct operation
    3. This is used for:
      1. Full sync, Group sync, User sync
      2. UI provision buttons pressed on full, group, user, or membership sync'ing
      3. Error correction.  If an operation produces an error, do a recalc on that object
      4. If state doesnt match in a stateful operation (described above)
    4. Performance not as good
    5. Target will be in the correct state
    6. Events can be replayed without problems.  Idempotent

If you want all operations to be stateless, you can change this provisioning setting:

provisioner.<provisionerConfigId>.recalculateAllOperations = true


Here is the workflow of a provisioner


ESB change log consumer generates events

the esb events from change log are handed to provisioner, and it kicks off the incremental provisioner.

all provisioning jobs make sure no other jobs are running, unless they are asynchronous then it is ok

1process incoming messages, add actions

check the messaging queue for this provisioner queue (every provisioner will have one by default).  process messages

(e.g. from UI: full sync, group syncs, user syncs, or membership syncs).  Or non blocking sync results.  Or when groups/folders are marked as provisionable.  Or other reasons.  Note, dont trust the provisioning sync state for these.  Recalc things from target

2propagate provisioning attributesmake sure that provisionable settings (attributes under the covers) are propagated to all descendent objects in folders
3convert esb change log events to provisioning actionsconvert esb events from change log into actions in the provisioner.  By default these will translate to actions without recalc.  i.e. if a membership is added in grouper, then try to add it to the target
4filter by full sync

if a full sync started after this event happened, filter it

5recalc events during full syncif events happen between when a (a)synchronous full sync started, and before it finished, do a full recalc
6look for sync errors and add eventsif there are sync errors that happened a (configurable randomness), find them and add esb events to list

filter non recalc actions that are captures by recalc

if there are non recalc group/entity/membership actions that are being recalced, then filter them

8retrieve sync group objectsretrieve all the group sync objects for all events
9query recalc group provisioning attributesretrieve provisioning attributes for recalc groups and adjust sync objects 
10filter if not provisionableif the sync group object says not provisionable for a non-recalc event, then filter it
11filter by group syncif a group has had a group sync after this event took place, and the event is related to the group, then filter it (dont need to do this action anymore)
12convert events to group syncif there are enough events over a configurable threshold, convert events to group sync
13convert events to full syncif there are enough events over a configurable threshold, and weighted by type of event, convert events to full sync
14recalc events during group syncif events happen between when a (a)synchronous group sync started, and before it finished, do a full recalc action
15retrieve sync membership objectsretrieve sync membership objects for all memberships in events
16retrieve sync member objectsretrieve sync member objects for all members in the queue
17retrieve grouper incremental dataretrieve grouper incremental data for groups, entities, and memberships, process the data into wrappers
18filter by unneeded actionsif the state of the action is not expected to change the target based on sync objects, filter the action
19convert inconsistent events to recalce.g. if a delete of membership is issued, but the membership exists, then just recalc
20copy incremental state to wrapperstake the incremental state and copy to the wrapper objects
21retrieve subject linkresolve subjects for subject link if recalc or for subjects missing data
22translate grouper groups/entities data to target formattake the grouper groups, entities and translate to the target format
23manipulate grouper groups/entities attributesbased on configs manipulate the defaults, types, etc for grouper target translated group/entity attributes and fields
24matching id of grouper groups/entitiescalculate the matching id of grouper translated group/entity data
25index matching id of grouper groups/entitiestake all the matching ids of grouper groups/entities and index those for quick lookups
26validate objects and filter invalidlook at objects and validate them and filter invalid objects
27retrieve incremental data from targetfor recalc, get data from target
28target object attribute manipulationprocess the target objects for attribute manipulation
29target id incremental objectsassign target ids to target objects
30create missing groups / entitiescreate missing groups / entities
31retrieve target group and entity linkbased on data retrieved, update the group and entity link
32translate grouper memberships to targettranslate grouper memberships to target after the link data is resolved
33manipulate grouper memberships attributesbased on configs manipulate the defaults, types, etc for grouper target translated membership attributes and fields
34matching id of grouper membershipscalculate the matching id of grouper translated memberships data
35index matching ID of grouper and target objectstake the matching IDs of the grouper side and the target side and index objects in the data index
36compare target objectslook at the grouper side and the target side and compare and generate the target actions
37send changes to targetsend inserts updates and deletes to target
38mark messages as readthe provisioner should take the messages and process them and acknowledge the message...

Messaging to control provisioning externally (e.g. from UI or WS)

You can send messages to provisioners to control provisioning, though there is no feedback about what happened.  Well, you could poll the messaging system to see if the message has been processed or deleted.  Generally this should take effect in a minute or two.  The default implementation will select all messages at once for all provisioners with the assumption that there arent that many at any given time.

Queue for provisioning: grouperProvisioningControl_provisionerName  (e.g. grouperProvisioningControl_myPspngProvisioner)

Java to send a message:

    ProvisioningMessage provisioningMessage = new ProvisioningMessage();
    String message = provisioningMessage.toJson();

        new GrouperMessageSendParam().assignGrouperMessageSystemName(GrouperBuiltinMessagingSystem.BUILTIN_NAME)

Message to do a full sync:


Message to sync some groups:


Message to sync some users:


Message to sync from memberships:

  • No labels