feature in development, not available yet
Configuration
Threads
loader.incrementalThreads=true loader.incrementalThreadPoolSize=10
Jobs
otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob otherJob.incrementalLoader1.quartzCron = 0 * * * * ? otherJob.incrementalLoader1.databaseName=warehouse otherJob.incrementalLoader1.tableName=myincrementaltable otherJob.incrementalLoader2.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob otherJob.incrementalLoader2.quartzCron = 0 * * * * ? otherJob.incrementalLoader2.databaseName=warehouse2 otherJob.incrementalLoader2.tableName=myincrementaltable
Database setup
Add triggers on your existing tables that are used by the loader jobs such that new rows would be added to another table (e.g. myincrementaltable) when a user's membership changes (or potentially changes) for a loader job. The table has the following columns:
- id
- subject_id
- subject_identifier - mutually exclusive with subject_id
- source_id - optional
- loaderGroupName - Grouper group name of a group defined as SQL_SIMPLE or SQL_GROUP_LIST.
- timestamp - milliseconds
- completed timestamp - milliseconds
HSQLDB: CREATE TABLE myincrementaltable ( id INTEGER NOT NULL, subject_id VARCHAR(255), subject_identifier VARCHAR(255), source_id VARCHAR(255), loaderGroupName VARCHAR(1024) NOT NULL, timestamp BIGINT NOT NULL, completed_timestamp BIGINT, PRIMARY KEY (id) );
Daemon
- The daemon would retrieve the 10000 oldest rows, removing duplicates.
- Spawn off threads based on the thread configuration.
- If SQL_SIMPLE, query grouperLoaderQuery to see if user should be a member. Query and change Grouper if needed.
- If SQL_GROUP_LIST, query grouperLoaderQuery to see which groups the user should be a member of. Need to figure out which Grouper groups are managed by this loader job to see what changes to make. We will require the "like" attribute.
- Need to take into account grouperLoaderAndGroups. If configured, user must be in all all of the "and" groups to be in the loader group.
- Any special logic around include/excludes? No
- If group doesn't exist in Grouper, run full sync.
- If more than 100 changes for the same group, sleep for a minute, run full sync, mark changes in change log as completed.
- Fail safe? Covered by above.
- Update with time when complete and delete if more than a day.
- Log to grouper_loader_log.