Page History

...

To provision in real-time triggered by the Grouper change log, enable the psp consumer in grouper-loader.properties and run the loader via

bin/gsh.sh -loader

Real-Time

...

Changelog Provisioning Details - Probably More Than You Ever Want to Know

Changes to grouper may be are consumed by change log consumers. Grouper change log consumers are managed by the grouper loader and are configured in grouper-loader.properties.

...

Grouper change log consumers process change log entries. A change log entry consists of single-valued attributes. A change log entry is a java representation of a row in the change log table grouper_change_log_entry.

For example, the psp logs the adding of when a member is added to a group as , the following is part of the psp log :

'ChangeLogEntry[timestamp=2012-05-31 11:59:56.321, sequence=344, category=membership, actionname=addMembership, fieldName=members, subjectId=test.subject.1, sourceId=ldap, membershipType=flattened, groupName=edu:groupA, ...]'

When a change occurs in grouper, a representation of that change is written to a temporary table grouper_change_log_entry_temp. The grouper loader periodically (every minute) reads this table, performs magic like potentially updating the PIT (point-in-time auditing) tables, and writes to the grouper_change_log_entry table.

Change log entries written to the grouper_change_log_entry table are processed in order of sequence number by every change log consumer. The grouper loader persists the last sequence number processed by every change log consumerconsumers.

Every minute, the grouper loader retrieves batches of 100 change log entries (grouper_change_log_entry rows), terminating at a maximum of 100k 100,000 change log entries. These batches of 100 change log entries are passed to each change log consumer.

...

The psp change log consumer determines the change log category (for example "membership") and change log action (for example "addMembership") and either processes or ignores the change log entry. Supported change log categories and actions are coded in the psp change log consumer, take a look at the EventType enum. To support a change log category and action that is not supported by the provided psp change log consumer, you will need to provide your own implementation, probably by extending the psp change log consumer class. For help, ask the grouper-dev@internet2.edu list.

Processing an "addMembership" change log entry results in a psp calc request, where the identifier (principal name) to be calculated is the change log sequence number :

...

The change log data connectors know how to retrieve a change log entry from grouper via the change log sequence number, and return attributes to the attribute resolver representing the change log entry. In general, the change log data connectors convert a grouper change log entry into (shibboleth attribute resolverauthority) attributes. For example, the attribute authority will return the following attributes for an "addMembership" change log event :

'groupObjectclasschangeLogMembershipGroupDn' : top
'groupObjectclass' : groupOfNames
'groupObjectclass' : eduMember
'changeLogMembershipGroupDn' : org.openspml.v2.msg.spml.PSOIdentifier@e6acf477
'changeLogMembershipMemberDn' : org.openspml.v2.msg.spml.PSOIdentifier@e6acf477
'memberObjectclass' : eduMember
'changeLogMembershipMemberDn' : org.openspml.v2.msg.spml.PSOIdentifier@97ebeb3b
'changeLogMembershipGroupName' : edu:groupA
'changeLogMembershipSubjectName' : test.subject.1
'changeLogMembershipLdapSubjectId' : test.subject.1

...

The next step in processing by the psp after calculating attributes is to resolve references. The attribute values for of the "changeLogMembershipLdapSubjectId" and "changeLogMembershipGroupName" attributes are converted to identifiers by executing psp calc requests.

...

After resolving reference identifiers, a calc response is returned by the psp for the calc request for the principal whose name is the change log sequence number. The calc response returned by the psp represents how an add or delete membership change log entry should be provisioned. For example :

<psp:calcRequest returnData='everything'>
  <psp:id ID='change_log_sequence_number:344'/>
</psp:calcRequest>

<psp:calcResponse >
  <psp:id ID='change_log_sequence_number:344'/>
  <psp:pso entityName='groupMembership'>
    <psoID ID='cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' targetID='ldap'/>
    <capabilityData ... >
      <spmlref:reference typeOfReference='member' ...>
        <spmlref:toPsoID ID='uid=test.subject.1,ou=people,dc=example,dc=edu' targetID='ldap'/>
      </spmlref:reference>
    </capabilityData>
  </psp:pso>
  <psp:pso entityName='memberMembership'>
    <psoID ID='uid=test.subject.1,ou=people,dc=example,dc=edu' targetID='ldap'/>
    <capabilityData ... >
      <spmlref:reference typeOfReference='memberOf' ... >
        <spmlref:toPsoID ID='cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' targetID='ldap'/>
      </spmlref:reference>
    </capabilityData>
  </psp:pso>
</psp:calcResponse>

The calc response returned by the psp represents how an add or delete membership next step in change log entry should be provisioned.The next step in processing by the psp change log consumer is to determine the difference between how each object should be provisioned (the calc response) and how the objects are currently provisioned.

For example, for each reference that should or should not be provisioned for an add or delete membership change log entry, the psp change log consumer performs an spml 'hasReference' search. Of course, the spml 'hasReference' search which must be supported by each provisioned target implementation.

In the following example, the object with identifier 'cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' is queried for whether or not it has a 'member' reference to 'uid=test.subject.1,ou=people,dc=example,dc=edu' :

...

The search response is successful but does not contain any provisioned service objects, which means that the reference does not exist.

Because the reference that should exist does not, a modify request is executed by the psp to add a 'member' reference with id 'uid=test.subject.1,ou=people,dc=example,dc=edu' to the object with id 'cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' :

<modifyRequest entityName='groupMembership' returnData='identifier' ... >
  <psoID ID='cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' targetID='ldap'/>
  <modification modificationMode='add'>
    <dsml:modification name='hasMember' operation='add' ... >
      <dsml:value>test.subject.1</dsml:value>
    </dsml:modification>
  </modification>
  <modification modificationMode='add'>
    <capabilityData ... >
      <spmlref:reference typeOfReference='member' ... >
        <spmlref:toPsoID ID='uid=test.subject.1,ou=people,dc=example,dc=edu' targetID='ldap'/>
      </spmlref:reference>
    </capabilityData>
  </modification>
</modifyRequest>

<modifyResponse status='success' ... >
  <pso>
    <psoID ID='cn=groupA,ou=edu,ou=groups,dc=example,dc=edu' targetID='ldap'/>
  </pso>
</modifyResponse>

Phew, done. In the above example, the psp has added a member to a group as a result of processing an add membership change log entry. It will also add the memberOf attribute to the member object, but I have omitted that xml.

For more detailed examples, take a look at the spml requests and responses in src/test/resources of the various psp-example-grouper-to-ldap projects.

Real-Time Provisioning Beta-Testing : Grouper Subject Sources

...