- Chris Hyzer, Penn, Chair
- Chad Redman, University of North Carolina Chapel Hill
- Shilen Patel, Duke
- Vivek Sachdiva, independent
- Jeffrey Williams, University of North Carolina Greensboro
- Carey Black, The Ohio State University
- Emily Eisbruch, Internet2
New Action Items
- AI Chad , work with Chris to get Azure provisioner into new provisioning DAO, getting atomic operations in there, a lot of configuration
Intellectual Property reminder: http://www.internet2.edu/membership/ip.html
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
- Took place Oct. 13-16, 2020 https://www.incommon.org/academy/grouper/
- Chad and Chris did debrief a few days ago
- Good feedback from students
- Improved from the June 2020 training
- Improved look of the training slides and put materials into Canvas
- Created videos for students to do pre-work
- In June that work was most of 1st day
- More time for hands on during training
- Fewer Kahootz quizzes
- Next time: put more into prework, things that take a long time or are repetitive
- Redo the last day of training, to address practical helpdesk issues, setting up an app, provisioning framework example, add more quizzes,
- Put more on Grouper Deployment Guide into prework, make training modules that cover the GDG in a video
- Might have 3 Grouper trainings in 2021
- Students were great
InCommon CAMP / ACAMP, Virtual, November 16-20, 2020
- Chad plans to attend
SCIM - AWS SSO SCIM Implementation
- Today there is a software integration call , Keith Hazelton.
- chance to check off a few boxes,
- Chris: roughly follows our DAO
- The ones that are not supported are not as important
- Register a SCIM provider to provision to AWS
- Make a generic SCIM thing and point at AWS, point at Atlassian cloud
- Perhaps the new provisioning framework can handle this
- this is a fundamental shift away from SAML
- It’s not federated
- it can be random in terms of what is supported. It is REST based
- Might not be interoperable
- Does not seem like the right path
- Does not scale well enough
- AWS SSO is a product they have
- It’s not SAML
- AWS group is distinct of AWS roles
- You either use group (SAML) or roles
- Roles use SAML identities
- There are some things SAML does not do because AWS does not support it
- Related to roles
- You can pre provision
- The other layer of AWS behind this is what the role does and what the group does
- They could take one small step with SAML and make role self asserted
- SCIM does groups, SAML does roles
- Can’t remotely assert membership in an AWS group
- will do a proxy in the middle to handle this
- Dual control
- SAML does not allow
- Must have permission from the other user to log in
- No system now allows that out of the box
- Duke uses SAML with AWS
- This is just another target
- If AWS isn’t using it the right way, that is a separate issue
- We should get it working with SCIM
- Re each account managed separately in AWS, this is similar to some other systems we send data to
- Should one provisioner try to handle it
- Chris familiar w Zoom group
- Thought of using multiple provisioners
- Especially if there are just a handful
AWS SSO SCIM Interesting thing to add
- We should get going with it and we will learn more
- SAML is the recommended way to go (such as Penn groups with SAML)
Carey observed a few cases where a single job was running on multiple daemons at the exact same time. Is that a known issue? Any ideas how to troubleshot that? Might be related to local network issues that recently were brought to light by a core router failing.
Current work tasks, and next tasks
Vivek – Provisioning
- Vivek and Chris went thru the provisioning screens and decided what is best for an initial implementation
- Will make improvements later
- Vivek needs to work on API soon
- Two flags, something should be provisioned, something has been provisioned, no more digging thru point in time
- View Log
- Logs for provisioners at
- job level
- group level
- Person level if you do a re-sync
- Membership provisioning
- Can have a course grained eligibility group
- Carey: Entity has to be provisionable
- Good feature. But can also lead to harder to understand “sync” status. ( Grouper group has 100 members, and 10 are “synced”.. Is that “good and complete?” How would you know? ( Maybe a Membership filter option to have a way to check in the UI?
- Look at group in UI and see which are eligible to by synced
- Filter for eligible or filter for provisioned
- Chris: let’s add that in the future, Vivek make a note of that please
- Can see details at subject level
- Provisioning jobs
- Full sync or incremental sync
- Various types of jobs
- Provisioning Logs
- Description is a clob field so can put more than 4K
- Vivek will fix churn issue
- Chris: good starting point, let’s track new features and add them later
- Hope to release in 2-4 weeks, to get community testing and feedback
- Chad: a desired feature, provisioning subject types , looking at subject types to be sure what to sync, be sure it is multi valued, different subject sources, this is already implemented.
- Chad: comment: this provisioning work is great
Carey: Longer term idea/question:
- Include Provisioning logs/status in daily report output?
- Chris - that’s easy to do
- “Dry run” option to run at a higher log/detail level for only that run?
- Trouble shooting with read only
- Might want a point in time check,
- Chris: think this need has been satisfied in current version, let us know
Shilen: step in the right direction
- Question on UI screens, what is the privileging on that?
- Chris: need to be an admin of Grouper
- To view , you need read priv on the group
- DECISION TO CHANGE THIS TO GROUPER admins only
- If some stuff is sensitive this can be tweaked
- Want help desk people to see if data got to its target
- Logs should be only for Grouper admins
- Publish most recent data as meta attributes on the group itself
- desire to export data from Grouper to Splunk. This is a frequent request from staff. Events that occur and log messages. Would be easy if there was a provisioning event log
- Stuff that goes to database would go to a file.
- Or work off a query.
- Jeff: going to a file would be great
- Chris will make sure there is more event data going to a file
- Carey: has a change log consumer for data for SPLUNK to ingest
- Creating change log consumers for other teams.
- Been a journey to get audit from the change log consumer event flow
- Logged or not
- Getting messages and teaching change log consumer to handle them
- Get the system level events surfacable would be helpful
- Chris: we should revisit that
- Jeff: plus one on audit logs
Vivek: Next steps on provisioning: new activity, churn, implement the changes we just decided
Chris – Provisioning, training
- Commit for provisioning framework
- More from change log consumer to provisioning framework
Shilen – Provisioning
- Cleaned up the LDAP config
- Removed old properties , added new ones
- Can give you 1000 lines for attributes
- Tested setting up through the UI
- Updated the LDAP DAO to fix error handling
- Returning what fails
- Next: do load testing, set up a test for load testing to work when logic is ready
- See who the first adopters are and solve some of their use cases
- Test AD with LDAP DAO
- Good to keep kicking the tires
- In addition to Grouper training
- Handled GRP 3000
- Pulled date formats out of Java code into configuration
Issue Round Up
JIRAs from last 2 weeks
sort error on attributes
Chris could not reproduce , Carey will try
Move UI date formats into config properties FIXED
pspng error when deleting PIT data
Some UI date formats show 24:00 for 12AM FIXED
pspng: pit error on group FIXED
Daemon job page get job links working in new tab , issue w AJAX, opens new copy, give user control over refresh would be nice, filter criteria saved would be nice
ChangeLogConsumerBaseImpl membership_deleteMembership fails when subject is a deleted group FIXED impacts Azure provisioner
- AI CHAD , work with Chris to get Azure provisioner into new provisioning DAO, getting atomic operations in there, a lot of configuration
grouper should make sure shib user hasnt changed
ui-ws does not turn shib on
you should be able to overlay a config file in subimage or slashRoot in container
add debug info on loader query that fails
error viewing custom ui group when not a grouper admin
add "minus" to composite screen to add clarity
you should be able to mount a script hook into the container instead of having to bake in subimage
default groups.wheel.use to true in grouper.base.properties
make custom ui for logged in user more obvious
Visualization for Privileges
Grouper Users emails from last 2 weeks
Grouper wiki updates in past 2 weeks
Grouper Slack in past 2 weeks
Chris Hyzer Im going to see how long it takes to do some pspng provisioning (with some performance improvements) in my test env.
Peter -Working through our IS requirements, figured I'd ask if it is common when running the Grouper container to run the internals using a different user than 'tomcat'? (edited)
- Chad: SEEMS FIXABLE, make open shift group writable by root,
- Chris: when container starts have script that can run as pseudo and CH own and CH model
- This may contradict the purpose of pseudo
Beth - Is there a means by which to force a full sync from Grouper to Office 365?
Liam - going through the quickstart. I’ve run into a problem - our servers are RHEL7 and quickstart instructions don’t seem to work with the RHEL provided docker (1.13.1)
Beth Is there an obvious culprit associated with the Office 365 consumer log that would result in the creation and modification of the group in Office365, but not the deletion?
Liam -Can Grouper use mongodb as a data source? I see some meeting notes about UWash being interested in using it as a document source from several years ago..
Chris has spoken w Liam a few times, U Michigan moving along
Erik -having some difficulty with a SQL loader job against an Oracle DB.
Is it possible to add classes to the jexl evaluation in the pspng like the loader.ldap.el.classes configuration? Specifically, I want to be able to call a static function from a custom class.
Wil C- Is anyone using Grouper w/Sailpoint IIQ? I don’t have a specific problem, just want to connect with other schools to find out what, how, etc (we’re starting a Grouper project soon, already have IIQ)
Penn will be using similar product, interesting discussion
Chris Hyzer important bug to get fixed before one of your admins who has run the application template gets removed from the wheel group (or someone who was in the application admin "role" gets removed). fixed in upcoming 2.5.36
Jonathan -when doing a stem query is there a way to include its attributes in the response?
Mike M - new to Grouper and my team has a test install set up. We want to configure this to use Shibboleth. Found this article: https://spaces.at.internet2.edu/display/Grouper/Authentication+to+the+Grouper+UI
I'm understanding this to tell me that we should install mod_shib, thus having Shib SP protect grouper and talk to our Shib IdP. Is this the right track?
Liam - I’m executing my container with ui-ws, and I see it in the logs…
If I look at the environment variables set in tomcat’s environment, I see a bunch, but I’m missing stuff that I expected to be turned on for ui-ws (e.g. GROUPER_RUN_SHIB_SP is set to false, when https://spaces.at.internet2.edu/display/Grouper/Grouper+container+documentation+for+v2.5 says that ui-ws should turn shib on?)
Chris Hyzer - in 2.5.36+ you can overlay a config file in subimage or slashRoot without Grouper failing on patch or copying over top. Note, you can still do the other things too with script hooks or whatever else
Carey - I hope that I am just not seeing how to do this in the WS calls.... But maybe it is not there?
Are there any WS ways to get "deltas during a time slice"?
Scott K - Suppose in grouper-loader.properties I do indeed have a database defined with the name ‘warehouse’. Is the property db.warehouse.c3p0.max_size a hard upper limit on the number of connections that can be open to that database?
Liam can grouper subject sources be created using the “External Systems” UI?
Oren - Grouper and AWS SSO meeting has now been scheduled for Thursday, 10/22,
Liam H - What will subject.getSourceId() return? The attribute value used to provide subjectApi.source.example.param.SubjectID_AttributeType.value ?
Carey - Can Grouper types on objects values be extended locally?
I would like to add a type of "System of Record". ( SOR or "System" or something like that ...) And I could also see a few other "local values" being of use too.
Scott K - want to have a WS user that is not GrouperSystem and for whom we can restrict access to certain stems/groups. It should authenticate to the WS using simple basic auth. It is essentially a “system account”.
Is the right approach to create a “local entity” (ala https://spaces.at.internet2.edu/display/Grouper/Grouper+local+entities) and then set a password for it (ala https://spaces.at.internet2.edu/pages/viewpage.action?pageId=163120202)?
Chris H- Grouper v2.5.36 is released. Lots of PSPNG improvements especially regarding significant performance gains. And some other fixes.
If you upgrade to this soon please let us know your experience. Thanks!
Liam H - Do the loader diagnostics log anywhere? I don’t see them using docker logs or in grouper.log
Carey B - Looking at Grouper rules use case - Veto if not eligible by folder
Does that actually remove memberships when they become "not eligible"? ( via the rules daemon ? )
There are a lot of qualifications under the "(v2.4 patch) Rules daemon will look at memberships" part of that page that makes it a bit hard to parse.... "(e.g. allow group if should allow)"
Jeff -I’m trying to use a loader script to select folks who have changed their password in AD. AD requires paging for returns greater than 1000. Is there anything I need to set in the loader properties to indicate the loader should use paging with AD? We are on v2.4.
Justin - Grouper Hooks, We’ve built one that uses the MembershipHooks. It appears to only trigger on an immediate membership and not on children of group memberships. Is there a hook which can or a configuration which triggers it?
Carey - The "Home" page for Grouper shows "Recent activity" for the logged in user.
I happened to be doing somethings after midnight and before 1 AM last night.
Today the "Activity Date" shows as "2020/10/22 24:16 AM".
Is there any way to tweak a config to make that read "properly" as "2020/10/22 00:16 AM" ?
I used the Grouper WS call grouper-ws/servicesRest/json/v2_5_000/attributeAssignments to assign an attribute to a group. In the UI I would next click on Actions > Add metadata assignment . How can I do that (add a metadata assignment) with a WS call? I have looked at https://spaces.at.internet2.edu/display/Grouper/Assign+Attributes but I do not see an example or explanation for how to add a metadata assignment.
Liam H Suggestions for debugging a loader job that fails intermittently with?
java.lang.RuntimeException: Error in loader job: test:etc:loader:umichInstRoles, check logs: null
Lacey - have a lot of lengthy sql loader queries which cause long lines in GSH scripts, making readability difficult. It looks like groovy supports multi-line strings using triple quotes, which seems to work within the scripts, but does cause a small formatting issue in the UI. I can live with that if it’s the only side effect, but wanted to check if there is a better way to break up long strings, or if it’s not recommended for some reason.
Richard F -Talk "Migrating from AJP to HTTP: It's About Time" from ApacheCon is up: https://youtu.be/qUjUEvGFstI
Liam - I’ve set up a LDAP_GROUPS_FROM_ATTRIBUTES loader. The attribute it’s using (“umichInstRoles”) is multivalued. It’s creating all of the expected groups, and 22 of 23 of them have members.
Oren= Thanks to everyone who attended yesterday's discussion of AWS SSO and Grouper. I took away the assignment to ask around to find an expert on the SCIM interface with Grouper who can help inform Kevin Murakoshi from AWS about how it might lash up with SSO.
mikeporter - problem with using docker secrets for the database password and the ui service? I can start gsh in the ui container without a problem, so I know the password is basically Ok. However, the ui fails with a hibernate password error. The password is not “morphed”. In docker-compose.yml for the service, I have the environment variable GROUPER_DATABASE_PASSWORD_FILE=‘/run/secrets/pgpasswd’, and a secret of pgpasswd.
Scott K - I believe you do need to “morph” it before exposing it through the environment variable.
Erik - We have a developer running a small Python script against Grouper WS, and is complaining that though the script seems to work, it inexplicably dies after a few hundred successive queries.
Wil C- You might consider using triple-backquotes, instead of single, for pasting code
When it dies, does it raise an exception or does it just hang? If the latter, try enabling connection and response time-outs in the requests module.
Scott K- What is the simplest way to temporarily disable a custom change log consumer? My best naive idea is to change the cron configuration so that it is not set to run for far out into the future. Is there a better way?
Erik -What's the right syntax to enable a TRACE level log on the AzureAD Graph API calls?
Wil -User-Private Groups (POSIX) in LDAP, how you dealt with them w/Grouper (or if you needed to at all)? I guess I am also assuming that there was a need to mass import or sync between LDAP & Grouper, which might not actually be a valid assumption
When/why would group UUIDs change? I thought that was the point of UUIDs, that they could be unique, opaque & permanent (merging duplicates aside)?
Next Grouper Call: Wed. Nov. 11, 2020