Configuration

Threads

loader.incrementalThreads=true
loader.incrementalThreadPoolSize=10

Jobs

otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader1.quartzCron = 0 * * * * ?
otherJob.incrementalLoader1.databaseName=warehouse
otherJob.incrementalLoader1.tableName=myincrementaltable
 
otherJob.incrementalLoader2.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader2.quartzCron = 0 * * * * ?
otherJob.incrementalLoader2.databaseName=warehouse2
otherJob.incrementalLoader2.tableName=myincrementaltable

Database setup

Add triggers on your existing tables that are used by the loader jobs such that new rows would be added to another table (e.g. myincrementaltable) when a user's membership changes (or potentially changes) for a loader job.  The table has the following columns:

• id
• subject_id
• subject_identifier - mutually exclusive with subject_id
• source_id - optional
• loader_group_name - Grouper group name of a group defined as SQL_SIMPLE or SQL_GROUP_LIST.
• timestamp - milliseconds
• completed_timestamp - milliseconds

HSQLDB:
 
CREATE TABLE myincrementaltable
(
    id INTEGER NOT NULL,
    subject_id VARCHAR(255),
    subject_identifier VARCHAR(255),
    source_id VARCHAR(255),
    loader_group_name VARCHAR(1024) NOT NULL,
    timestamp BIGINT NOT NULL,
    completed_timestamp BIGINT,
    PRIMARY KEY (id)
);

Daemon

• The daemon would retrieve the 10000 oldest rows, removing duplicates.
• Spawn off threads based on the thread configuration.
• If SQL_SIMPLE, query grouperLoaderQuery to see if user should be a member.  Query and change Grouper if needed.
• If SQL_GROUP_LIST, query grouperLoaderQuery to see which groups the user should be a member of.  Need to figure out which Grouper groups are managed by this loader job to see what changes to make.  We will require the "like" attribute.  
• Need to take into account grouperLoaderAndGroups.  If configured, user must be in all all of the "and" groups to be in the loader group.
• Any special logic around include/excludes?  No
• If group doesn't exist in Grouper, run full sync.
• If more than 100 changes for the same group, sleep for a minute, run full sync, mark changes in change log as completed.
• Fail safe?  Covered by above.
• Update with time when complete and delete if more than a day.
• Log to grouper_loader_log.