feature in development, not available yet
Configuration
Threads
loader.incrementalThreads=true loader.incrementalThreadPoolSize=10
Jobs
otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob otherJob.incrementalLoader1.quartzCron = 0 * * * * ? otherJob.incrementalLoader1.databaseName=warehouse otherJob.incrementalLoader1.tableName=myincrementaltable otherJob.incrementalLoader2.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob otherJob.incrementalLoader2.quartzCron = 0 * * * * ? otherJob.incrementalLoader2.databaseName=warehouse2 otherJob.incrementalLoader2.tableName=myincrementaltable
Database setup
Add triggers on your existing tables that are used by the loader jobs such that new rows would be added to another table (e.g. myincrementaltable) when a user's membership changes (or potentially changes) for a loader job. The table has the following columns:
- id
- subject_id - mutually exclusive with subject_identifier/subject_id_or_identifier
- subject_identifier - mutually exclusive with subject_id/subject_id_or_identifier
- subject_id_or_identifier - mutually exclusive with subject_id/subject_identifier
- source_id - optional
- loader_group_name - Grouper group name of a group defined as SQL_SIMPLE or SQL_GROUP_LIST.
- timestamp - milliseconds
- completed_timestamp - milliseconds
HSQLDB: CREATE TABLE myincrementaltable ( id INTEGER NOT NULL, subject_id VARCHAR(255), subject_identifier VARCHAR(255), subject_id_or_identifier VARCHAR(255), source_id VARCHAR(255), loader_group_name VARCHAR(1024) NOT NULL, timestamp BIGINT NOT NULL, completed_timestamp BIGINT, PRIMARY KEY (id) );
Examples
Example of SQL_SIMPLE using Oracle (loader table has a group name field to allow multiple SQL_SIMPLE jobs): Say you have a loader table that looks like the following: CREATE TABLE myloadertable ( subject_id VARCHAR(255), group_name VARCHAR(255) ); With the following incremental table: CREATE TABLE myincrementaltable ( id NUMBER NOT NULL, subject_id VARCHAR(255), subject_identifier VARCHAR(255), subject_id_or_identifier VARCHAR(255), source_id VARCHAR(255), loader_group_name VARCHAR(1024) NOT NULL, timestamp NUMBER NOT NULL, completed_timestamp NUMBER, PRIMARY KEY (id) ); And a sequence for the primary key on the incremental table: CREATE SEQUENCE myincrementaltable_seq; And the following loader job: addRootStem("test", "test") addGroup("test", "loader1", "loader1") groupAddType("test:loader1", "grouperLoader") setGroupAttr("test:loader1", "grouperLoaderDbName", "grouper") setGroupAttr("test:loader1", "grouperLoaderType", "SQL_SIMPLE") setGroupAttr("test:loader1", "grouperLoaderScheduleType", "START_TO_START_INTERVAL") setGroupAttr("test:loader1", "grouperLoaderQuery", "select subject_id from myloadertable") setGroupAttr("test:loader1", "grouperLoaderIntervalSeconds", "86400") Assuming this is on the same database as Grouper, you could add the following configuration in grouper-loader.properties (run every 5 seconds): otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob otherJob.incrementalLoader1.quartzCron = 0/5 * * * * ? otherJob.incrementalLoader1.databaseName=grouper otherJob.incrementalLoader1.tableName=myincrementaltable And the following trigger: CREATE OR REPLACE TRIGGER mytrigger AFTER INSERT OR DELETE OR UPDATE ON myloadertable FOR EACH ROW DECLARE timemillis NUMBER; BEGIN select extract(day from(sys_extract_utc(systimestamp) - to_timestamp('1970-01-01', 'YYYY-MM-DD'))) * 86400000 + to_number(to_char(sys_extract_utc(systimestamp), 'SSSSSFF3')) into timemillis from dual; IF (:new.subject_id is not null) THEN INSERT INTO myincrementaltable (id, subject_id, loader_group_name, timestamp) values (myincrementaltable_seq.nextval, :new.subject_id, :new.group_name, timemillis); END IF; IF (:old.subject_id is not null) THEN INSERT INTO myincrementaltable (id, subject_id, loader_group_name, timestamp) values (myincrementaltable_seq.nextval, :old.subject_id, :old.group_name, timemillis); END IF; END;
Daemon
- The daemon would retrieve the 10000 oldest rows, removing duplicates.
- Spawn off threads based on the thread configuration.
- If SQL_SIMPLE, query grouperLoaderQuery to see if user should be a member. Query and change Grouper if needed.
- If SQL_GROUP_LIST, query grouperLoaderQuery to see which groups the user should be a member of. Need to figure out which Grouper groups are managed by this loader job to see what changes to make. We will require the "like" attribute.
- Need to take into account grouperLoaderAndGroups. If configured, user must be in all all of the "and" groups to be in the loader group.
- Any special logic around include/excludes? No
- If group doesn't exist in Grouper, run full sync.
- If more than 100 changes for the same group, sleep for a minute, run full sync, mark changes in change log as completed.
- Fail safe? Covered by above.
- Update with time when complete and delete if more than a day.
- Log to grouper_loader_log.