Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added an example view of the incremental loader table.

...

Code Block
Postgresql:  Note that the trigger (see examples below) does not need to fill in the timestamp as it
defaults in the table DDL.  The indexes are suggested if you are putting more than a 1000 or so rows
in here per day.  Most sites run the incremental_loader very often, so the query to find zero rows
should complete very quickly.

CREATE TABLE IF NOT EXISTS myincrementaltable(
    id SERIAL NOT NULL UNIQUE,
    subject_id VARCHAR( 256 ),
    subject_identifier VARCHAR( 256 ),
    subject_id_or_identifier VARCHAR( 256 ),
    subject_source_id VARCHAR( 256 ),
    loader_group_name VARCHAR( 1024 ) NOT NULL,
    timestamp BIGINT NOT NULL DEFAULT( EXTRACT(EPOCH FROM CLOCK_TIMESTAMP())*1000::BIGINT ),
    completed_timestamp BIGINT,
    PRIMARY KEY( id )
);

CREATE INDEX IF NOT EXISTS myincrementaltable_completed_timestamp_index ON
    myincrementaltable( completed_timestamp );
CREATE INDEX IF NOT EXISTS myincrementaltable_loader_group_name_index ON
    myincrementaltable( loader_group_name );

GRANT ALL ON myincrementaltable TO grouperROLE;
GRANT ALL ON myincrementaltable_id_seq TO grouperROLE;

--
-- This might be useful as well.
--

CREATE OR REPLACE VIEW view_myincrementaltable AS
    SELECT subject_id, TO_TIMESTAMP( timestamp/1000 ) AS queued_timestamp,
        TO_TIMESTAMP( completed_timestamp/1000 ) AS completed_timestamp,
        TO_TIMESTAMP( completed_timestamp/1000 ) -
            TO_TIMESTAMP( timestamp/1000 ) AS elapsed

        FROM myincrementaltable ORDER BY id DESC;

GRANT SELECT ON myincrementaltable TO grouperROLE;


Messaging

If you are triggering updates via messaging instead of database triggers, the table above is still needed.  A builtin message listener will simply monitor your queue and add rows to the table.  Note that the "id" column has to be populated automatically on insert (i.e. use an auto increment type column or use a trigger to populate based on a sequence before the insert).

...