Note: this was an off-week Grouper call since we did not meet on Jan. 20, 2021 due to inauguration.
Attending
- Chris Hyzer, Penn, Chair
- Shilen Patel, Duke
- Chad Redman, University of North Carolina Chapel Hill
- Carey Black, the Ohio State University
- Vivek Sachdiva, independent
- Jeff Williams, UNCG
- Emily Eisbruch, Internet2
Discussion
- https://internet2.edu/community/about-us/policies/internet2-intellectual-property-policy/
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
Grouper Training Online
- Registration is open for Grouper School Feb 9-12, 2021
- https://incommon.org/academy/grouper/
- New release will go in Grouper training container
- Getting ready to release CANVAS material to students
- Chad putting new provisioner info in Training slidesBlog on New Provisioning Framework, January 2021
New Grouper release coming along, first release in about 6 weeks.
- Will label it as slightly experimental
- Arizona requested syncing from another Grouper may lead to another Grouper release in early or mid Feb.
- See release notes https://spaces.at.internet2.edu/display/Grouper/v2.5+Release+Notes
Chad, question on visualization and new provisioner targets and what permissions / priv are needed to view. Is there security involved in letting non-root users know that their group is provisioned? Ideally, left to the deployer to decide, but that is not yet implemented. Right now, everything as wheel.
Current work tasks, and next tasks
Vivek – Subject source wizard
- GRP-3099 subject source wizard
- Changing how the configurations work
- Under Misc, now option for subject sources
- Coming from multiple sources, properties file and database
- Only things JDBC or LDAP are listed as editable
- To comply w the new format, some tweaks to config will be needed
- When adding subject source, only 2 targets supported, LDAP type and SQL
- Make max results size not required, have a sensible default
- May use convert button for non editable
- Hope conversion from old to new is not jarring
- Supporting 2 JDBC sub classes
- We are supporting too many old things
- Need to make plan
- Upgrade for 2.6
- Get rid of old
- If using old property keys, no changes
- If statements
- Converts properties
- Chris made JIRA , if using attributes,
- subject attribute matches source,
- or they don’t match or
- there is a translation
- There are diagnostics
- Shilen: for search attributes you can combine multiple into one,
- You would select new,
- You might need to make an internal ?
- Don’t want to have translations everywhere
- Matt: conversion question….
- Thinking about how the overlay will work
- Give people a clue that this subject source is defined in database and in the files
- Run the diagnostics, Safer to bounce
- Convert to the new
- End state is a better position
- Change the text to Subject API attributes
- Under actions button could have configure subject API
- Decide before we release Grouper 2.6
- UI is long, could use tabs but that has challenges too
- These are administrative screens not used that often
- This screen will likely be used once
- Setting up a subject source is painful now and this is an improvement
- Thank you Vivek
- Vivek: Need to add validation, working on it
- Attribute name and field name, get that code into this
- Chad , this is better than doing everything in config file
- Descriptions on the fields, more explanations will help
- Registries subjects, need to see what we want to do
- Chad created a subject source, did not need it, deleted it, but things got broken
- There should be testing on this issue of deleting a subject source
- Some subject things in Grouper properties
- Not used a lot
- Configs in files, for some properties if consistent among environments,
- Don’t have to replicate 100 attributes in multiple environments
- Source control is an advantage of using files
- Make changes through GSH
- That conflicts with having a UI to configure it
- Need to document best practices
- Jeff: subject defined in flat file
- Do you have to go thru intermediate step?
- Can have command line to convert a file
- Code in Grouper API and call from UI
- AI Vivek: look for subject source in other configs and develop plan, in subject properties and not source specific, get a handle on what’s out there
Chris – Provisioning, training
- Email notification
- Using properties wizard work, can get wizard for other jobs
- Can get a UI for things that are scheduled
- Include date things expire
- Two types of emails: send to end users or send a summary
- Only sent if there are items in there
- Can have an opt out if you don’t want to receive
- Templating
- List of maps, subject properties or columns
- Carey: suggestion to add file support
- Provisioning set up work
- Each attribute will have translation type
- Flat DN, group base DN
- Some straight maps from fields
- Search attribute is in object that the framework needs to search
- DAO does not map
- Search attribute does the mapping
- Group search filter
- Mark as search attribute
Shilen – Provisioning
- Will do some load testing
- Will look at LDAP tests
- Delete what does not work?
- Issue on Slack? w bad membership finder change log consumer
- Shilen has not been able to reproduce it
- On Daemon screen would be good to be able to change last sequence number for change log consumer,
- Need a new table?
- Think about how this will work
- Look in existing change log consumers
- Need an easy way to skip over a change
For LDAP and provisioning, we have more tasks
- DOA group entity
Issues around SCIM provisioner
- SCIM is catching on, hopefully the object models could translate to other things, with tweaks
- Hope to take this model and edit it and make it available for AWS
- On the Mock Service side it will be specific to the product being supported
- AWS does not fully support SCIM2
- On client side it’s SCIM2 with some options
- Mock service implements a web service of SCIM
- Take code and edit it so Grouper web service can serve SCIM against Grouper
- IN future, we Remove Penn State SCIM approach
- And support SCIM w Grouper
- Won’t need TOMEE , can return to Tomcat
- By Grouper 2.6
- Chris working with AWS resource
Chad – Provisioning, training
- Adding new provisioning targets for visualization
- Looks like PSPNG
- Working on Azure provisioner
- Make JUNIT test case to go against mock thing in Grouper UI
- Existing change log provisioner for Azure does not work
Issue Roundup
Jiras in past 3 weeks
add entityAttributes provisioning for Grouper training
add translation type and remote "isTranslation"
Ability to limit provisioning to specific targets by group - vis
UI sorting of LDAP subject search results from a "free form" search
unescape $newline$ when editing configs in ui
sudo should pass env in container
folder privs more actions button blank
Support unlimited count of favorites
Visualization display new provisioner targets
Advanced Membership UI: Ability to create a group based on the filtered result set.
enable grouper container to work with openshift
Grouper Local Auth ( and the GrouperClient )
Is there a way to send an email to the member who was just added to the group?
Grouper Emails in past 3 weeks
[grouper-users] Problems with 2.5 level 1 database upgrade, Andre Daniels, 01/12/2021
- Re: [grouper-users] Problems with 2.5 level 1 database upgrade, Black, Carey M., 01/12/2021
- [grouper-users] Structuring Scoped Roles in Grouper, Jonathan Keller, 01/14/2021
- Re: [grouper-users] Structuring Scoped Roles in Grouper, Hyzer, Chris, 01/14/2021
- Re: [grouper-users] Structuring Scoped Roles in Grouper, Jonathan Keller, 01/15/2021
- <Possible follow-up(s)>
- Re: [grouper-users] Structuring Scoped Roles in Grouper, Black, Carey M., 01/15/2021
- Re: [grouper-users] Structuring Scoped Roles in Grouper, Jonathan Keller, 01/15/2021
- [grouper-users] job opportunity at UWash, Nathan Dors, 01/21/2021
- [grouper-users] Grouper Provisioning Framework blog, Emily Eisbruch, 01/22/2021
Grouper wiki updates in past 3 weeks
- Grouper provisioner framework tasks
- v2.5 Release Notes
- DDL in Grouper v2.5+
- v2.5 Upgrade Instructions from v2.5
- Grouper subject source configuration wizard
- Grouper daily email notification - affiliation expiration report
- Grouper daily email notification use case - FERPA notifications
- Grouper daily email notification
- Development Items
- Countdown groups for password reset requirement
- Grouper Training Environment developer notes
- Nightly email reminder when training will run out
- Grouper demo Technical Administration
- Grouper generic provisioner framework
- Grouper custom template via GSH
- Grouper container documentation for v2.5
- How to Setup a Grouper Development Environment for Grouper v2.5
- Grouper container running on OpenShift
- Grouper provisioning SCIM
- Grouper provisioning failsafe
- GrouperShell (gsh)
- Grouper client script from Excel CSV
- Grouper Client
- Grouper membership SQL provisioner loader use case Arizona
Grouper Slack in past 3 weeks
Carey -I have a general GSH problem that I keep bumping into .. and avoiding… but I would really like a “way to do it”.
Example… trying to use the “removeIf()” method on a Set. ( REF: ………..
Justin -Asking a probably no brainer question that I have struggled to understand. For Grouper WS, how does the act as work? From an audit perspective, does it record the user that authenticated and the user that it was acting as?
Brett - Regarding the processing of group membership begin/end dates, how do folks typically handle timezone offset to ensure we're at midnight...? Is the locale a setting that controls this...? Is this in the schedule for one of the daemon jobs that we need to adjust manually when the time changes?
Carey -Rule question: RE: Email notification on flattened membership add to group
Is there a way to send an email to the member who was just added to the group?
AKA: The rule supports using JEXL to produce a CSV list of email addresses, but how do you get the new Member’s email address in the context of the JEXL?
Chris H - The winner of the poll is "manual". ….
Chris H - We have been discussing end users scripting. Here is an example of taking an Excel sheet and generating a grouper client script (note you can expand this to do other operations, this example only has create group and add member)
Tim D - Is there a way to configure grouper to not audit a set of groups or better yet, an entire folder? I'm trying to avoid a lot of the rows that loader is dumping into grouper_audit_entry (edited)
Carey - GRP-3086 Grouper Local Auth ( and the GrouperClient )
Carey -I think we may have finally “broken the UI” ( by scale and mySQL performance )….
I was told by an app owner that they can no longer “open a Grouper folder” in the UI. It just hangs/spins.
NOTE: I believe this is more of an annoyance report. It’s not a “production is broken” report.
The user can search for more specific folders and find them under that. ( I think. )
Scott K I have a Grouper deployment running successfully in AWS ECS using Fargate. I am using image i2incommon/grouper:2.5.29. I want to upgrade to using image i2incommon/grouper:2.5.39. ….
Justin R -I recall some discussion on the grouper.text.en.us.properties file. Has it been migrated to the database in the latest version of Grouper? We have 2.5.39 up now in our development environment and see it in the list, but it appears blank so we want to make sure that is expected or not.
Erin - Join us for Grouper training with Chris and Chad during the week of February 8th!
Carey - RE: Grouper rules use case - Veto if not eligible by folder
”Note, the ruleCheckArg0 is the subject source. If it is blank, then the rule applies to all subject sources. If it is filled in, then the rule only applies to that subject source.
” I think that is a bit misleading.
Carey - plug for upvotes/comments GRP-3089: Advanced Membership UI: Ability to create a group based on the filtered result set.
Liam Is it possible to assign privileges recursively through the UI?
And is it possible to hide portions of the folder structure from people?
Jeffrey - Has anyone used types to denote access groups for resources that are more sensitive than the garden-variety ones?
Liam - Why is the “actions” drop down empty in the group privs management screen?
Liam -Is there a gsh method that will just return the last component of a group name (the path id?) instead of the full path? Or do I need to do something with groovy to get that bit out?
Chris H - grouperUtil.extensionFromName(group.getName())
Carey -A question about Grouper daemon "other job" to run a script .
Is there a processing timeout that might interrupt such a script?
Liam - What do I need to run GSH outside of a container? libraries and conf files?
Liam Is it possible to see the status of jobs that have been sent to the background (e.g. deleting a folder structure from the UI)?
Carey -RE: Grouper+reporting Is there anyway to “Run now” for a given report? ( basically like the rest of the “Jobs” [Loader, etc…] )
Jeffrey - a behaviour that I've noticed with accessing the status, diagnosticType=<item> verb. On the WS there is no basic auth required so it's easy to use it as a load balancer check, healthcheck etc.
However the UI that location is still behind shib, should there be an exception for that url, or is that not possible? For the UI I've been running the healthcheck against tomee directly using port 8080, but I don't need to do that on the WS so it's a more complete test in my mind.
Chris B - at the risk of perpetrating violence against deceased equines, azure ad apparently uses "assigned" and "dynamic" to differentiate handcrafted vs. auto-populated groups
Jeffrey C I wanted to go back to converting data for Grouper from ??? to postgresql (Oracle in our case). Looking at https://spaces.at.internet2.edu/display/Grouper/Grouper+database+migration+utility I noticed the line “// note: make sure the source is readonly or your foreign keys will be hosed” I wanted to be clear that, that means that the “grouper” database has been set to read only via GRANT? I.e. there isn’t a need to make the entire database a read only database via a config or something to that effect?
Also I understand this is experimental but is there a plan to make this utility something more mainstream?
Tim D - For the parts of the Grouper UI that use a free form search, subjectApi.source.ldap.search.search.param.filter.value in my subject.properties, is there a way to tell Grouper to sort those search results?
Scott Koranda @mchyzer I don't think anything is copied from /slashRoot when the command is /opt/grouper/grouperWebapp/WEB-INF/bin/gsh.sh, is that correct?
Scott K Nevermind. The correct statement is "/usr/local/bin/gsh is exec'd before any files are copied from slashRoot so the bug with /usr/local/bin/gsh cannot be worked around by putting a fixed version in slashRoot".
Carey
RE: Grouper+SQL+database+provisioning+example+from+view
# fullSyncFull, fullSyncGroups, fullSyncChangeFlag, incrementalAllColumns, incrementalPrimaryKey
otherJob.test_sql_sync.syncType = fullSyncFull
What is the difference between those options?
Emily -Enjoy this blog by @mchyzer on the New Grouper Provisioning Framework https://incommon.org/news/flexible-and-powerful-to-address-community-needs-new-grouper-provisioning-framework/
Chris H Grouper email notifications:
https://spaces.at.internet2.edu/display/Grouper/Grouper+daily+email+notification
Wil C Has anyone setup a “backdoor” into Grouper when using Shib w/Apache, using an alternative URL and Apache’s native basic auth, for example?
Liam -If I want the subject API diagnostics to try the webservice, I’m guessing I’m going to need to set the “test” webservice settings in… grouper-ws.properties (e.g. ws.testing.host)? I’m running the container… but I want some visible proof that it’s working / configured correctly
Erik C I got test environment into a bit of a pickle... I added using the Grouper UI a new (3rd) LDAP external connector, and related subject sources. Turns out that connector won't work quite yet because the AWS subnet is not configured correctly to talk to that LDAP. Containers restarted, and inexplicably the logins for Grouper are attempting to go out to this new LDAP, and thus I can't get in. I never specified this new one as the "Default", I wanted to stick with the old subject source and LDAP, so I'm wondering can I (A) force logins to resolve using a single subject source, or (B) is there a single database entry I can mangle by hand to "break" this bad LDAP entry and/or subject source?
Carey Does the “Import config file” replace all existing config values for that file or does it append to the config in the DB?
If I upload a file (Say grouper.client.properties) with only one java property key=value line, will it remove all existing setting from the file or just add the new config to the existing values?
Liam - Is it possible to tie loader jobs to a specific instance of the daemon?
Justin R - Just upgraded to 2.5 in our stage env and had really good success. Kudos to all the devs working to enhance the product. All the features coming in are awesome! question, we had Duo functional prior, but it appears to be failing now. I don’t see anything in the logs - just that it couldn’t get all the way through the batch. It’s possible that it is failing due to issues getting outbound as i need to add opts to java for our proxy. Is there an arg for the daemon I should use for that?
Liam- Is it possible to add additional attributes to the csv group export? (specifically alternate subject indentifiers)?
Liam - Is it possible to set a custom attribute on a group with a loader job?
Chris H - Email notification use case:
Email summary use case:
Liam H - Anyone have suggestions as to what the map the grouper group attributes to when they provision into AD? Do you use the grouper ID # or the UUID?
Jeffrey C - messaging Queue question, Is RabbitMQ still the main queue technology supported by Grouper? I ask since I don’t believe RabbitMQ is one of the managed queue technologies in AWS. Also we would have to run our own RabbitMQ as our integrations team is not willing to run that service.
Steve Zoppi The queueing protocol supported by all of the Trusted Access Platform components is: AMQP ... RabbitMQ provides that as does almost any other viable messaging engine. The Trusted Access Platform uses the AMQP implemented by RabbitMQ as "standard" but any messaging engine that implements AMQP should work.
Liam - Can a group display name reference a different folder structure than the name?
We’ve got reference groups based on department ID numbers, but our stakeholders want us to display names instead. I guess we could use ref:hr:department for both, instead of ref:hr:deptid
Liam - Is it possible to cancel a running loader job? I’ve got a job that’s been running since 1/20. It normally takes 5 minutes to complete and, we’re going on “4 days 21 hours 23 minutes 23 seconds
Justin R - I noticed a new job after the upgrade - change log consumer for find bad memberships. Curious what this does? Sometimes it is very fast and other times may not finish.
Liam - Where are the additional attributes pulled onto subjects stored? It doesn’t seem to be in GROUPER_PIT_MEMBERS or GROUPER_MEMBERS. Are they looked up on demand? (edited)
Liam Is it possible to get Grouper to re-populate the DESCRIPTION column in GROUPER_MEMBERS? Our is based on a virtual attribute, and I updated the attribute defintion
Chris H - An idea we have had for a long time is a subject API configuration wizard. @Vivek Sachdeva has made this happen. Thanks! I need to document/polish it, but it does exist in 2.5.40 (will be released soon)
Carey Nice to see such an improvement from “managing files”. The project has made some real progress in dealing with config in the UI. ( Dynamic data/config driven.. very good stuff.)
Next Grouper Call: Wed February 3, 2021