Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader1.quartzCron = 0 * * * * ?
otherJob.incrementalLoader1.databaseName=warehouse
otherJob.incrementalLoader1.tableName=myincrementaltable
otherJob.incrementalLoader1.skipIfFullSyncDisabled=true
 
otherJob.incrementalLoader2.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader2.quartzCron = 0 * * * * ?
otherJob.incrementalLoader2.databaseName=warehouse2
otherJob.incrementalLoader2.tableName=myincrementaltable
otherJob.incrementalLoader2.skipIfFullSyncDisabled=true

Note that skipIfFullSyncDisabled defaults to true.  This means that updates to groups will be skipped if the overall loader job associated with the group is disabled.  This status is available in the "All daemon jobs" page.

Database setup

Your incremental table should have the following columns:

...

Code Block
HSQLDB:
 
CREATE TABLE myincrementaltable
(
    id INTEGER NOT NULL,
    subject_id VARCHAR(255),
    subject_identifier VARCHAR(255),
    subject_id_or_identifier VARCHAR(255),
    subject_source_id VARCHAR(255),
    loader_group_name VARCHAR(1024) NOT NULL,
    timestamp BIGINT NOT NULL,
    completed_timestamp BIGINT,
    PRIMARY KEY (id)
);

Messaging


Code Block
Postgresql:  Note that the trigger (see examples below) does not need to fill in the timestamp as it
defaults in the table DDL.  The indexes are suggested if you are putting more than a 1000 or so rows
in here per day.  Most sites run the incremental_loader very often, so the query to find zero rows
should complete very quickly.

CREATE TABLE IF NOT EXISTS myincrementaltable(
    id SERIAL NOT NULL UNIQUE,
    subject_id VARCHAR( 256 ),
    subject_identifier VARCHAR( 256 ),
    subject_id_or_identifier VARCHAR( 256 ),
    subject_source_id VARCHAR( 256 ),
    loader_group_name VARCHAR( 1024 ) NOT NULL,
    timestamp BIGINT NOT NULL DEFAULT( EXTRACT(EPOCH FROM CLOCK_TIMESTAMP())*1000::BIGINT ),
    completed_timestamp BIGINT,
    PRIMARY KEY( id )
);

CREATE INDEX IF NOT EXISTS myincrementaltable_completed_timestamp_index ON
    myincrementaltable( completed_timestamp );
CREATE INDEX IF NOT EXISTS myincrementaltable_loader_group_name_index ON
    myincrementaltable( loader_group_name );

GRANT ALL ON myincrementaltable TO grouperROLE;
GRANT ALL ON myincrementaltable_id_seq TO grouperROLE;

--
-- This might be useful as well.
--

CREATE OR REPLACE VIEW view_myincrementaltable AS
    SELECT subject_id, TO_TIMESTAMP( timestamp/1000 ) AS queued_timestamp,
        TO_TIMESTAMP( completed_timestamp/1000 ) AS completed_timestamp,
        TO_TIMESTAMP( completed_timestamp/1000 ) -
            TO_TIMESTAMP( timestamp/1000 ) AS elapsed

        FROM myincrementaltable ORDER BY id DESC;

GRANT SELECT ON myincrementaltable TO grouperROLE;


Messaging

If you are If you are triggering updates via messaging instead of database triggers, the table above is still needed.  A builtin message listener will simply monitor your queue and add rows to the table.  Note that the "id" column has to be populated automatically on insert (i.e. use an auto increment type column or use a trigger to populate based on a sequence before the insert).

...

Code Block
messaging.listener.myCustomMessagingListener.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalMessagingListener
messaging.listener.myCustomMessagingListener.quartzCron = 0 * * * * ?
messaging.listener.myCustomMessagingListener.messagingSystemName = grouperBuiltinMessaging
messaging.listener.myCustomMessagingListener.queueName = abc
messaging.listener.myCustomMessagingListener.numberOfTriesPerIteration = 3
messaging.listener.myCustomMessagingListener.pollingTimeoutSeconds = 18
messaging.listener.myCustomMessagingListener.sleepSecondsInBetweenIterations = 0
messaging.listener.myCustomMessagingListener.maxMessagesToReceiveAtOnce = 20
# if there are 20 messages to receive at once, then do this 50 times per call max
messaging.listener.myCustomMessagingListener.maxOuterLoops = 50
messaging.listener.myCustomMessagingListener.incrementalLoaderJobName = incrementalLoader1
messaging.listener.myCustomMessagingListener.messageQueueType = queue
  • Update the messagingSystemName to point to your messaging system (Grouper supports a built in messaging system along with RabbitMQ, AWS, etc).
  • Update the queueName
  • Update incrementalLoaderJobName based on what was configured earlier in the Configuration section above.

...