About Pipelines

Pipelines connect Organizational Identities, typically created from Organizational Identity Sources, to CO Person Records. Pipelines can be used to automatically enroll, update, and expire CO Person records linked to external sources. Pipelines are available in COmanage Registry v2.0.0 and later.

• The External Data Source holds person related records. This is typically a SQL or LDAP database, a flat file, an API, or another similar repository.
• The Organizational Identity Source is a configured Organizational Identity Source plugin, typically with a Sync Mode configured. It obtains information from the External Data Source and converts it to Organizational Identity Format.
  • As of Registry v4.1.0, Organizational Identity Sources support Data Filters.
• The Organizational Identity Source Group Mapping is a configuration, attached to the Organizational Identity Source configuration, that maps attributes from the External Data Source into candidate CO Group Memberships.
• The Organizational Identity Source Record is an artifact created when an Organizational Identity is instantiated from an Organizational Identity Source. It is a copy of the record of the External Data Source, linked to the Organizational Identity that was created from it.
• The Pipeline takes the Organizational Identity record (including any candidate CO Group Memberships), and syncs them to the operational CO Person, CO Person Role, and CO Group Membership records. As part of this process, the Pipeline may attempt to instantiate a match process to determine if a new Organizational Identity matches an existing CO Person record in some way.

The use of Organizational Identity Sources is not required in order to use a Pipeline, but other usage scenarios may not be fully implemented yet.

Configuring Pipelines

Match Strategies

Match Strategies are used to determine if an Organizational Identity should be connected to an existing CO Person. The following Match Strategies are supported:

• Email Address: The Pipeline looks for an existing CO Person record with an Email Address (of a specified type) that matches one attached to the Organizational Identity. The Email Address need not be verified, so be careful about matching on self-asserted email addresses.
• Identifier: The Pipeline looks for an existing CO Person record with an Identifier (of a specified type) that matches one attached to the Organizational Identity.
• External: Call out to an external matching service. For more information, see Integrating With ID Match.

Remember, while the source of attributes for searching is the Organizational Identity record provided to the Pipeline, the search target of Match Strategies are existing CO Person attributes, not Organizational Identity attributes.

If no existing CO Person is matched, then the Pipeline will create a new CO Person record.

If more than one candidate CO Person is found, an error is thrown.

Sync Strategies

Sync Strategies are used to determine when a CO Person record should be created or updated by a Pipeline, and whether an associated CO Person Role record should also be created/updated.

• Sync on Add/Update/Delete: These setting control when an Organizational Identity is processed using a Pipeline.

• New CO Person Status: If set, a new CO Person record created by the pipeline will be created with this status. The default is to create a new CO Person record with status set to Active.
  
  

  If the pipeline creates a new CO Person record it does so immediately after the Organizational Identity is processed. When the Organizational Identity is created by an Organizational Identity Source using the EnvSource plugin attached to an enrollment flow the creation of the new CO Person record by the pipeline happens after authentication but before the form is displayed to collect any petitioner attributes. Deployers should review what status for the newly created CO Person record is appropriate for the configured enrollment flow.

• Create CO Person Role Record: If checked, when the Pipeline executes it will create a CO Person Role record, not just a CO Person record. This is useful to (eg) automatically add someone to a COU based on their Organizational Identity Source.
• Sync to COU: If Create CO Person Role Record is set, this setting defines which COU the new Role Record will be placed into.
• CO Person Role Affiliation: If Create CO Person Role Record is set, this setting defines the affiliation given to the new Role Record. If not set, the affiliation of the source Org Identity will be used.
• Replace Record in COU: If the CO Person has an existing CO Person Role record in the specified COU, that Role Record will be expired. This is useful (eg) to expire a provisional record created before an authoritative record is received. Only executed on Add operations, not on Updates.
• Role Status on Delete: If Create CO Person Role Record is set and the Organizational Identity Source record is deleted (no longer valid), the corresponding CO Person Role will be set to the specified status.
• Sync Identifier Type: Identifier type used when syncing relationships, see Syncing Relationships below.

When a Sync Strategy executes, it copies all data provided by the Organizational Identity Source and any defined Group Mappings.

Sync Attributes

The following attributes are supported for sync from an Org Identity:

• Address
• EmailAddress (verified status is honored)
• Identifier
• Name (primary name flags are cleared)
• TelephoneNumber
• AdHocAttribute (as of Registry v3.3.0)

When a CO Person Role is created, the following attributes are supported for sync from the Org Identity:

• Affiliation
• O (organization)
• OU (department)
• Manager (as of Registry v4.1.0, see below)
• Sponsor (as of Registry v4.1.0, see below)
• Title
• Valid From
• Valid Through

Syncing Relationships

As of Registry v4.1.0, it is possible to map Manager and Sponsor relationships via Pipelines. To do so, Create CO Person Role Record must be enabled and the Organizational Identity Source must support populating manager_identifier and/or sponsor_identifier with an Identifier from the record of the target CO Person. The type of this Identifier is controlled by Sync Identifier Type. If the Pipeline can find an Identifier matching the inbound OIS field, it will insert the target CO Person ID into the manager_co_person_id or sponsor_co_person_role_id field of the new CO Person Role. If Sync Identifier Type is blank, then any Identifier type will match. If more than one Identifier matches, the first record returned by the database will be used. If no Identifier matches, the CO Person Role value is left blank.

There is currently no requirement that the Manager or Sponsor be in active status.

manager_identifier and sponsor_identifier are only intended for use with Organizational Identity Sources and Pipelines. While the fields are visible on Org Identity records, they cannot be edited there. Similarly, these fields cannot be set on Org Identities via Enrollment Flows (use the corresponding CO Person Role attributes there instead).

Connecting Pipelines

Pipelines are intended to be connected to Organizational Identity Sources as a means of automatically creating or linking CO Person records. Except when connected to an Enrollment Flow, when a Pipeline creates a new CO Person record, Identifier Assignment is triggered, and when a Pipeline creates or updates a CO Person or CO Person Role record, provisioning is triggered.

Pipelines are executed according to the current configuration, so it is possible for an Organizational Identity to be processed by a different Pipeline than the one it was originally attached to.

Enrollment Flows

Pipelines can be attached to Enrollment Flows in two different ways: An Enrollment Flow can trigger a Pipeline, and a Pipeline can trigger an Enrollment Flow. Care should be taken to avoid circular dependencies.

Triggering Pipelines From Enrollment Flows

Pipelines can be implicitly attached to Enrollment Flows via Enrollment Sources. If an Organizational Identity is created from an Enrollment Source attached to an Enrollment Flow, and if a Pipeline is attached to the Organizational Identity Source that the Enrollment Source is configured to use, then that Pipeline will be executed when the record is processed from the Source.

Triggering Enrollment Flows From Pipelines

As of Registry v3.3.0, Pipelines can be configured to trigger Enrollment Flows, with some constraints:

• Enrollment Flows are only triggered during Add Sync operations.
• In the Enrollment Flow configuration
  • Petitioner Enrollment Authorization should be set to CO Admin to reduce the likelihood of anyone starting it manually.
  • The Enrollment Flow Status must be Active.
  • Email Confirmation is required, as it is the mechanism by which control is passed to the Enrollee.
  • A Pipeline should not be attached via an Enrollment Source (as described above).
  • Identity Matching should be set to None.
  • Do not attach any Enrollment Attributes, since the attribute collection step will not run.
• In order to collect the authenticated user identifier, the use of EnvSource is recommended. (Use a configuration like the example provided for Self Signup.)
• Attach the Enrollment Flow to the Pipeline via Pipeline: Sync Strategy > Sync on Add.

The Pipeline must create a (or link to an existing) CO Person identity for the Enrollment Flow to successfully run.

Organizational Identity Source Sync

Pipelines can be executed during Organizational Identity Source sync processes, resulting in add, update or deletes being processed.

Default Enrollment

Pipeline execution during Default Registry Enrollment is not supported.

Manually Rerunning a Pipeline

An Organizational Identity can be reprocessed through a Pipeline by viewing the appropriate Org Identity (People >> Organizational Identities) and clicking Rerun Pipeline. The Pipeline run will be selected in the same order of preference as defined above.