- Chris Hyzer, Penn, Chair
- Chad Redman, University of North Carolina Chapel Hill
- Shilen Patel, Duke
- Carey Black, the Ohio State University
- Vivek Sachdiva, independent
- Bill Thompson, Lafayette College
- Jeff Williams, University of North Carolina Greensboro
- Emily Eisbruch, Internet2
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
- Grouper Fish of a Fin at 2020 Global Summit in Indianapolis:
March 31, 2020 at 12:30pm https://meetings.internet2.edu/2020-global-summit/detail/10005807/
- Grouper Training April 29-30 at Greensboro: https://www.incommon.org/grouper-school-unc-greensboro/
- Bill T and Chris Hyzer will be trainers
- Promote this at your institutions
Grouper 2.5 release
Tasks needing to be done
- Gantt chart (chad)
- Misc ui page re-arrange
- Maven working well
- Library conflicts? Yes, fixed!
- Oracle jar not in public repositories, how to add?
- Download jar and put in Docker file , put in op container. Lib
- For Oracle users, will need special instructions
- Same issue for any JDBC connection ( via loader jobs too )
- Shilen, disable groups, there is a fix needed
- If you disable a group it disables every membership including privileges
- If not a Grouper admin and you disable group, you won’t be able to re-enable
- Probably Admin Privileges should not be removed
- If I don’t have access is it because I am not in a certain group? Has there been a backup? How to re-sync? Takes a lot of digging
- Pre sorting before provisioner has to process the event is wise
- Have wanted a log for each provisioner for history
- That’s hard to do with the current raw logs
Vivek wrap up installer work
- Shilen wrap up group disable work
- Then Chris will work with Vivek and Shilen to get PSPNG improvements into Grouper 2.5 release
- agrees about need for more readable logs for end users
- Users want option to sync Grouper stem to one of the provisioners
- Group out of sync and want full sync in background, also at stem level
- performance issues are important and urgent
- We must do something in that area, even if it pushes out Grouper 2.5 release
- Migration to new framework that Vivek wrote for PSPNG is more than a build # increment on a container so it should be included with Grouper 2.5
- Chris looking at Changelog consumer / ESB listener consumer
- Special changelog consumer to change messages so it’s a real java bean
- If messages go to JSON you can encrypt them
- Chris has been looking at workflow for ESB changelog consumer
- Push tasks that PSPNG does up the stack
- Look at messages , see if they are provisionable based on standard framework , see if full syncs or group syncs have happened we can pop some messages off queue
- Are there edge cases Chris is missing?
- Hard to troubleshoot
- Would like to see retooling of ESB consumer workflow
- Then look at PSP NG to see what we can tie into
- Instead of PSP NG doing its own checking to see if something is provisionable
- Instead of checking each event,
- Rely on upstream framework and do batching there
- For real time ESB events or for Full Sync
- Discussion last week w Shilen , Bert, Vivek,
- Perhaps use Shilen’s approach of full syncs
- For Grouper 2.5, use new ESB changelog approach
- Matt: agrees with this general approach
- Can’t get rid of change log temp
- But might be able to enhance it
- Only give provisioners events that they care about
- Look at things we can do with batching
- Good to know when there is a large queue
- Eventually would like see if a concept of user sync makes sense, for greater efficiency
- Right now there is a group sync
- Could have a general provisioner page that shows what is in the queue
- Automated threshold to get a notification
- Vivek: keeping systems in sync and Grouper maintaining info on what has been synced and what is configured to be synced
- What about problems when one system does not acknowledge receiving ?
- False positives
- Things can get out of sync
- If you are going to make a call to a target , you should get feedback
- Logic is used for failure situations
- JDBC syncing also does logic for failures
- At Duke, some cases where data must be in LDAP immediately
- Vivek : should there be a system between Grouper and LDAP?
- A queue , answer: that is what the change log is
- Jeff thoughts on PSP NG:
- Unit tests for PSP NG
- Jeff will let us know
- Run unit tests for PSP NG or create new ones?
- Bert has created script based GSH based tests for PSP NG
- Shilen needs to take a look at what is there now
- Perhaps convert to JUNIT
- Get info from Jeff?
- Shilen will get into the code first
- Chris, Plan is that from UI, click on user or provisioner and see in database what has happened recently
- Keep log of when last group syncs were
- There are several types of syncs
- Should have improved logs and reporting mechanisms, button you push
- Sync button could compare in the UI space
- See what is out of sync and send message to change log consumer
- Good to have button on a member or group or object to sync
- Not sure about the folder level, table this for now, not for 1st release
- Info stored in database so it can be seen in UI
- Jeff: Sometimes CIOs want a report or a log, don’t want to go to UI
- Logging in JSON may be easy lift
- Bill T - if Grouper product is going to have support for this provisioning, it should work fast
- In simple cases it needs to work
- Simple = deploy to LDAP and AD
- Though there can be complexity
- At Lafayette, looking at cloud to handle some of the issues
- Big lift for Grouper to reproduce what Rabbit MQ offers
- Chris: useful to have some logic in UI , to potentially keep user informed, so it’s not all done by messaging
- Chris : would like Grouper to be easy to use for provisioning
- Integrate sync with Grouper provisioner
- Lafayette will upgrade to Grouper 2.5 and then will provide feedback
- Chris: use case from Grouper Training : how to easily send this group to this LDAP DN and that group to that LDAP DN
- When you mark a group as provisionable, have an attribute for the target path
- Provisioner for main OU and another provisioner for ad hoc
- Can use threads
- Prioritize to avoid backlog
- Hope that more institutions will migrate to PSPNG after Grouper 2.5 release
Cron Job testing (Chad)
- Chad working on the cron job unit testing , using HSQL , loader and sync table, set database driver to postgress, then every test after does not work.
- There are Grouper properties to skip those.
- Chad will try that. Mostly runs. But not accurate representation ,
- Error in the tests , can’t find the root folder, can’t access the database, because of the driver issue.
- We do want to do HSQL. it is an easier life , running postgress involves Docker
- Or run postgress database and create new databases and delete them at end
- Grouper loader test , issue at line 147
- Overrides in config come out after every run?
- Should start from blank slate
- Should test every database
- Have a dedicated database scheme for testing each branch
- Schedule tests of branches to not happen at same time
- Chad created alternate port
- Create a quick docker empty postgres database and connect to it and then kill it when tests are done, to eliminate need to delete files
- Just start postgres container
- Still ports must be unique
- Sending summary every night or only if errors?
- Use email with attachment
- Or email with summary with a link to a file
- Will expose the directory
- To see the last week of runs
Current work tasks, and next tasks
Vivek – Building and packaging
- Build container is done.
- Allows creating a web app directory containing Grouper UI, web services, etc.
- Runs under Tomee .
- The intent is for Chris Hubing to build containers.
- Now working w Chris Hubing on building the container,
- Plan to be done by end of this week
- Chris Hubing is making container, then next step is Vivek working on how to install that container, needs to be easy as possible for the campuses
- Getting version # out of manifest of jars - not done yet
- Chad look at this issue?
- Grouper, Grouper client and web service needs to know its version
- Chad adjusted that in manifest and jar a year ago, possibly not the misc projects
- Might only need for client
- If we assume someone running the container
- May want to check other places...
Chris – SQL sync, bugs
- See question re expression language (below)
- Filter out what you don’t need like SOR groups
- Filter by policy group
- Worried someone is doing expression that won’t be covered by new approach
- Chris Hyzer (from Slack)
- We are going to change how PSPNG groups are identified for provisioning. In order to accomodate everyone's requirements, I'm interested in the various ways people configure this. For instance, at Penn, we just use the default, which means groupSelectionExpression is not configured, and it uses the provision_to and do_not_provision_to attributes. If you do not use the default, can you please reply to this thread with your config (groupSelectionExpression, attributesUsedInGroupSelectionExpression, attributesUsedInGroupSelectionExpressionAreComparedToProvisionerName) or your requirements (if no one else mentioned it). In addition to configuring provisioning attributes on folders and groups, I would assume we would allow an Expression Language to filter by name, and would also allow requiring types of groups (e.g. only policy). Does that capture it or do we need more?
Shilen – permissions issues, other 2.5 tasks
Chad – maven builds
Bill – ad hoc types
Jeff – pspng
Moving forward and handling JIRAS
- New Grouper 2.4 release in next couple of days
- Re JIRAs , will do bug roundup after Grouper 2.5 is released
Feb 5: Andrew M - am I supposed to be able to set searchResultPagingEnabled on the pspng configuration properties?
Feb 5 Andrew M - found a bug
Feb 6 Jeffrey C -Has there been any changes in the privileges evaluation?
Feb 6 : MG: - want to do LDAP to a table in the grouper DB -
Feb 6 Andrew M - group renames (name change), including moves, will always result in the deletion and re-creation of the group provisioned in the target.
Feb 6 Andrew M -The logic in processAnyChangesInGroupSelection() calculates deselectedGroups and newlySelectedGroups sets based on the hash of the group name.
Feb 7 Jeffrey C- split up our WS from our UI instance of grouper onto separate servers. I know this is best practice but you have the services combined in the non-production, why do you need extra servers to do the work that is combined today
Feb 7 Mathew B - Loader Job , groovy shell
Feb 8 Jeffrey C - a search from the Grouper UI takes a long time
Feb 11 you cant rename root
Feb 12 Justin R- container issues
Feb 14 Alex P - purging no-longer needed PIT and Audit data
Feb 15 Chris Hyzer -We are going to change how PSPNG groups are identified for provisioning
Feb 15 Jordan D - setting inherited privileges with the web services for some automation. It looks like I'd need to set 9 attributes for each
Feb 17 Jeffrey W: Ldap Loader : I have an LDAP_SIMPLE loader type that seems to work, but I'd like to specify the target group instead of it populating the loader group.
Feb 18 Michael G - Grouper integer on IDs issue
Feb 18 J Crawford - changelog temp to changelog performance
Feb 19 - Changelog consumers have next start time in the past.
Grouper Wiki updates
Grouper 2.4 and 2.5 release notes https://spaces.at.internet2.edu/display/Grouper/v2.5+Release+Notes
JIRA since Feb. 5, 2020
incremental SQL Loader job - grouperLoaderGroupsLike to support multiple patterns
grouper config screen error on elconfig properties
error creating sync tables in mysql
move remedy integration into Grouper API
add sync columns for percent complete and timestamp metadata was updated
index on wrong column in ddl
oracle timestamp error on sql sync
if config value is an asterisk it does not show on the edit screen, but it is in the database
insert update delete count on full sync in ui is wrong on sql sync
error getting database connection from client
sql sync is doing updates for all rows and not comparing numbers across different database vendors
number of updates in sql sync can be negative
need more batching on large SQL changes in the sql sync
null pointer on incrementalPrimaryKey
if you move a folder, it will keep its inherited type assignments
decimal and integers should be compared better in sql sync
create WS for running gsh
sql sync groupings using a non string (e.g. number) didnt work in some places
Next Grouper Call: Wed. March 4, 2020