Attending
- Chris Hyzer, Penn, Chair
- Chad Redman, University of North Carolina Chapel Hill
- Shilen Patel, Duke
- Carey Black, the Ohio State University
- Bert Bee Lindgren, GA Tech
- Vivek Sachdiva, independent
- Bill Thompson, Lafayette College
- Jeff Williams, University of North Carolina Greensboro
- Matt Wolfley, Unicon
- Action Items
DISCUSSION
- Administrivia
- http://www.internet2.edu/membership/ip.html
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
LOB discussion
- In Grouper 2.5.0 patches we will support BLOBS in the database
- for Grouper 2.5 Large Object Database Type
- https://spaces.at.internet2.edu/display/Grouper/Grouper+lob+database+type
- Framework takes it out of database and uses Temp directory in case it takes up a lot of space
- Could have a UI to help manage the table\
- For 2.5 put this in API, eventually in UI
- Vivek: issue with slowness in a CLOB he used
- Suggestion: Only leverage CLOB for over 4000
- Chris: could do this for attributes.
- Chris will think about this
- Q: extend this to group descriptions? So it would be possible to use any generic string and overflow from short to long storage,
- Consider String for attributes, or lob as type. Consider string / lobs
Bert
- Performance of pspng will be priority after Grouper 2.5 is released, in a month
- Grouper Provisioning: PSPNG
- In 2.5 patches, PSP performance will be a focus
- For many sites with medium PSP provisioners, it can take a long time on change log and full sync
- If Bert has time to help in about a month from now, that will be great.
- Bert has worked to get Grouper 2.4 instrumented
- Objects nest in tree structure
- Full full sync and group full sync
- Chain of instrumentation
- Output on special performance log
- Time threshold (10 seconds)
- If takes more, then it summarizes itself
- Grabbing those objects, full sync time can be saved as an attribute
- To capture performance info
- Applying this approach to other places in Grouper would be helpful
- Two phases: short term gains from this experience
- Remote workshop for 2-3 hours with Chris, Jeff, Shilen, Bert
- To see how to integrate this with direction for other provisioners
- To increase performance, to increase data gathering around performance
- May focus on this in 1-2 months
- Working with Gettes
- Will do some R&D in next few weeks and apply that to Grouper 2.5
- Should use a library (perf4j)?
- AI Bert Commit instrumented approach to branch from 2.4
- [AI] Bert create wiki draft of instrumented approach
- Could have per group stats? Perhaps
Containers Strategy for Grouper 2.5
- https://spaces.at.internet2.edu/display/Grouper/Grouper+packaging+and+versioning+strategy+for+2.5
- Have a config that says auto-upgrade to 2.5, and container would auto-upgrade
- Chris and Chad talking about how to compile JAVA
- Whoever is ready to make a version of Grouper will put a tag on a branch and that will have a certain format
- There will be a tag on a branch
- A process will publish Grouper to Maven Central (Chad working on this now)
- To get the container to Docker Hub, in Internet2 Grouper that does the packaging they’ll change Docker file to say what version they want. They have CI / CD, to check out from GIT and run the installer.
- New Installer Task for the packaging managers
- Downloads jars from Maven, uses GIT files that have been exported, runs script or Java commands, or something to Merge into one folder, a web app.
- So no compiling, no tarballs, no patches
- That container is listed as a release candidate, it can be tested, and deployed, then it will become the latest version. If a problem with it, we will move to the next number Grouper 2.5.8 can increment to Grouper 2.5.9
- Working to get UI WS Daemon and SCIM running in one directory
- There are still separate projects in GIT
- Ant will be removed
- Strategy to run a web app in GIT
- Bill Thompson has been documenting that
- May need one small Ant task for 3rd party dependency
- Maturity Level 0 is running a container on a server
- Maturity Level 1 , you might have a Docker File and can externalize config in other files, but still pwd and logs on server
- Maturity Level 2, you have everything externalized, have pwd manager
- Maturity Level 3, you are using orchestration
- Maturity Level 4 might be working in cloud
For Maturity Level 0 ,
- authentication can be a stumbling block
- Some use Tomcat authentication
- Problem because there is no way to dynamically tell Tomcat to use that, must change server XML,
- Must change Web XML
- Would like it to be a config option
So for Grouper 2.5, basic auth in the servlet filter. In addition, have filters based on source IP addresses and better auth for Web Services
Has been discussed on Slack: should Grouper have authentication or if it should always be externalized
Other local customizations might need to be in the model.
What is the methodology for injecting binary jars into containers?
Currently :
- Low Maturity Level: if you have the files on the server that’s running the container you can mount the files into the container.
When you fire up Docker, you mount a source folder from outside Docker to a file inside Docker. Exception is that Chris Hubing, Internet2, has some built ins. (conf, lib?)
2. Higher maturity: you have a Docker file; libs are in your source control, when it builds container, it copies from your source control into your container. You deploy your container, Not mounting anything to external file system
discussion:
- Managing Tomcat config is like injecting a Jar
- Matt: trying not to make JSP changes, concerned about the hooks and change log consumers
- Vivek adding authentication, removing tomcats,
- making a few changes to Data Model
- For Grouper 2.5 there will be a tables , including a table for recent logins
- There will be a GSH way to set a pwd
- Config that asks which of 3 approaches you want, basic auth, UI web service, or SCIM
- Will hash and encrypt the pwd into the database
- No more tomcat XML files
- Later on, do Jason Web Token (JWT) approach in server and client
- JWT is more secure
- Authentication approach would be selectable on a per account basis, flexible
- https://spaces.at.internet2.edu/display/Grouper/Grouper+authentication+in+2.5
Summary of work to date:
- Simplified GIT structure
- Chad working on Maven
- Got web app running in one dir
- Still need to do installer tasks for container installer
- Another installer task to cover maturity level 0, with a wizard
- Question: how does this impact major upgrades w DDL changes?
- Answer: Don’t have GSH outside a container anymore
- So spin up a container , could be maturity level 0, start GSH and do registry, as you do now. In future, might have a config option to do this automatically
- Ideally automatic with some human intervention
- Shilen: for just index adds or new tables, it’s fairly safe
- Don’t too often have restructuring data in ways you have to be careful, but we need to consider those situations
- When we get to that , we can look at why UI is not starting and try to absorb some of those errors
- Allow person to go to admin screen, authenticate a user and sort it out
- Or “Automatically upgrade to v 2.6”
- always able to run one version back
- If new container is running in older database, then it updates
- n-1 approach
- Chad:
- For web services, need to support many web clients, ORACLE etc.
- They have basic authentication built in
- Need to support basic authentication as well as allow JWT and other methods
- Can have self serve accounts for web services
- Local entity concept w privileges
- Generate pwd, either basic auth or JWT private key
- That account will be set for that authentication
- Question: Do you want to enable JWT at all?
- What is model for changing pwd?
- For web services, not going to allow people to set their own pwd. (similar to DUO)
- If Grouper is going to offer authentication, how much do we need to provide?
Chad
- Maven status
- Possibly we don’t need that grouper.version file
- See info on wiki
- https://spaces.at.internet2.edu/display/Grouper/Grouper+packaging+and+versioning+strategy+for+2.5
- Suggested file called grouper.version, to be accessed by the installer, but would this be a hassle to maintain?
- If Chris Hubing is editing his files manually
- Installer would have version built in
- ChrisHyzer: on downloads site, perhaps have a script working from a tag
- Jars are in maven as snapshots and for Grouper 2.4.0
- See tag, Start a release process, push to staging channel, administrator would release those manually, that’s described on the packaging wiki page as steps for developer.
- Suggestion to have a separate branch and call it release.
- Poms for maven now are -snapshot
- Check in code? Or just local?
- Can’t build from source code?
- Should have official version pom somewhere
- That’s a common workflow, Chad has read about
- One branch called release, for every release it merges master and changes the version
- DECISION: we don’t need a separate branch. Maven is the source of truth.
- What gets built is local and goes away
- Travis CI will parse and convert
- No snapshots in Maven
- Suggestion to ask Vivek to move his work to Master
- Removing ant
- Remove 3rd party libs from github
- Remove client from Grouper
- Publish instructions for how to develop that way
- May be a couple of speedbumps but will work them out
- How to run tests without Ant?
- Use Java or GSH
Shilen
- Permission change log changes (waiting for patch from Chris)
- Updates for views for enabled flag on groups
- Couple weeks ago finished propagation for attributes disabled when group disabled
- Web services stuff that Vivek had started
- Queries for M. Gettes made things worse for Harvard
- https://lists.internet2.edu/sympa/arc/grouper-users/2020-01/msg00002.html
- Undo for next patch? Yes this would be OK
- Try to batch up the queries?
- It’s a syntax thing
- Hibernate and branching capabilities, Branch at hibernate level?
- Shilen will do some testing and try to replicate the problem
- What else for Grouper 2.5?
- Show something disabled in the UI? This is for future.
Issue Roundup
Grouper WIKI pages updated
- Grouper provisioning / daemon tables
- Packaging and Versioning Strategy for 2.5
- Grouper SQL database incremental provisioning
- Grouper authentication in 2.5
- Install the Grouper 2.4 container with maturity level 0
- v2.1.0 Grouper Development Environment Using Maven
- Grouper dev and container strategy for 2.5
- Grouper patching (BrettB)
- Grouper development environment (BillT)
- v2.4 Release Notes
- Grouper provisioning / daemon tables
- Grouper rules use case - Disabled-date activation when removed from another group ( Carey - added v2.3 UI steps to add rule)
- Grouper Provisioning: PSPNG
=======
SLACK
Jan 8 - Sudheer ; using pspng to provision grouper groups to LDAP and if group description has new linefeeds in it, The provisioner is assuming the first line as actual group description and the remaining lines as LDAP attributes
(this got addressed)
Jan 8 - J Crawford : does permission changes show up in the audit log? GRP-2544
Jan 8 - Carey : possible to have a rule set to "act as" the user who triggered the rule….
Rules would let me "link one to many" and "decide later to add another" ( condition, source, target, etc....)
Two new features: Ignore direct memberships from "g:isa" ( To allow nested groups)
Inspect indirect Memberships and create direct memberships when they are missing. ( and do it as the user who added the user to the other group. )
Jan 9 - Jeffrey C and Greg H: UI problem (fixed in patch?)
Jan 10 -Jeffrey C - bootstrap a grouper structure for an application that is org based. don't want it to manage users, just the groups and stems. trick in the loader to allow it to manage groups and folders but not the actual memberships?
Jan 10 - Rachel -possible to create Attributes over GWS
Jan 10 - Lacey V - provisioning groups to Active Directory (via PSPNG). AD admins are insisting that the samaccountname attribute match CN and samaccountname cannot contain colons. They also prefer a flat group structure in AD, for performance reasons
Jan 10- Carey - Q #1) Is there a way to restrict "normal users' from assigning "EveryEntity" privileges?
Q #2) Is there a way to restrict "normal users' from assigning "EveryEntity" as a member of a group?
Jan 10 -Carey - a user added EveryEntity to a group in their user folder. And suddenly all users could see the other users "user folder". It looks really odd.. and took me a bit to find the reason. " Should everyone be able to do that? "
Jan 14 - Alex P - PSPNG limitation, We're trying to setup an attribute provisioner to write <subject username>@<subdomain> to an attribute, where subdomain is pulled from a group attribute. The latter part isn't a problem, but it looks like subject information isn't being passed into the jexl parser for provisionedAttributeValueFormat. (CHRIS LOOKED AT THIS, SEEMS SOME TWEAKS ARE NEEDED TO PASS USER TO WHERE EXPRESSION LANGUAGE IS EVALUATED. Passed per group or per subject? Bert will take a look at this)
Jan 14-ChrisH - The way audits are stored is not conducive to querying.
Jan 14 - Anthony Hill - Install issues, container is better
Jan 15 - J Babb - Oracle to Aurora Postgres
Jan 15 - Haverkamp - pathways for an unaudited membership?
Jan 15 - PaulR -Performance issues w mass full syncs between grouper and AD.
Jan 17- J Craford - JVM options on the different grouper types, ui, ws, daemon?
Jan 17 - Shilen -formatting the queries differently significantly helped performance on mysql. But it turns out that the new queries don't work well for them but the old ones do. See email https://lists.internet2.edu/sympa/arc/grouper-users/2020-01/msg00003.html
Jan 17 ChrisH -Shib impersonate with grouper groups for who can impersonate and who can be impersonated. See here
Jan 22: J Williams : Upcoming change – Microsoft to disable use of unsigned LDAP port 389
Jan 22 - Alex P -issue with a rule: a problem where a group creation failed in the UI
Jan 22 - Sudheer - Can we migrate folders,groups and group memberships from one instance to another
=======
Jiras
GRP-2560 Consider a fake user for grouper loading groups with no members
GRP-2559 add sql other job to easily run sql commands
GRP-2558 grouper custom composite should not return groups
GRP-2557 when filtering by users who have update priv, should also return admin priv
GRP-2556 invitation: option to not try and add user to external subjects source if they exis
GRP-2555 rules dont fire for add member in ui when go from disable to enable
GRP-2554 all user attributes in jexl for pspng group attribute resolver
GRP-2553 ui not compiling with patch
GRP-2552 Visualization: error when user graphs and something isn't accessible.
GRP-2551 Change log events for permission changes on subject
GRP-2550 Upgrade quartz schedule to 2.3.2 (resolved)
GRP-2549 Audit does not capture all direct Membership adds
GRP-2548 PSPNG function to strip carriage returns from strings.
GRP-2547 Remove full-sync at startup option
GRP-2546 background color on recent activity not correct for wide rows
GRP-2545 attribute to mark attributesDef with attributeName values to not put in PIT
GRP-2544 Updates to permissions do not seem to be tracked in the audit log
GRP-2543 add a way to change the morph pass
======
Grouper Users list Emails
- [grouper-users] findBadMemberships job failure after taking latest patches, Rahman, Mahbub, 01/05/2020
- Re: [grouper-users] findBadMemberships job failure after taking latest patches, Rahman, Mahbub, 01/07/2020
- Re: [grouper-users] findBadMemberships job failure after taking latest patches, Rahman, Mahbub, 01/17/2020
- Re: [grouper-users] findBadMemberships job failure after taking latest patches, Shilen Patel, 01/07/2020
- <Possible follow-up(s)>
- [grouper-users] findBadMemberships job failure after taking latest patches, Mahbub Rahman, 01/06/2020
- [grouper-users] PSP in v2.4, Morgan, Andrew Jason, 01/07/2020
- Re: [grouper-users] PSP in v2.4, Jeffrey Williams, 01/07/2020
- Re: [grouper-users] PSP in v2.4, Morgan, Andrew Jason, 01/07/2020
- Re: [grouper-users] PSP in v2.4, Jeffrey Williams, 01/09/2020
- Re: [grouper-users] PSP in v2.4, Morgan, Andrew Jason, 01/09/2020
- Re: [grouper-users] PSP in v2.4, Sudheer Singidi, 01/07/2020
- [grouper-users] Oracle Incompatibility, Oliver Trieu, 01/08/2020
- [grouper-users] Scoped eppn question, Bryan Wooten, 01/13/2020
- Re: [grouper-users] Scoped eppn question, Scott Koranda, 01/13/2020
- Re: [grouper-users] Scoped eppn question, Greg Haverkamp, 01/13/2020
- RE: [grouper-users] Scoped eppn question, Hyzer, Chris, 01/13/2020
- [grouper-users] How to remove inconsistent Grouper rules, Olivier Salaün, 01/14/2020
- Re: [grouper-users] How to remove inconsistent Grouper rules, Olivier Salaün, 01/15/2020
- RE: [grouper-users] How to remove inconsistent Grouper rules, Hyzer, Chris, 01/14/2020
- [grouper-users] PSPNG and (not)provisioning groups, Marwan Ali Shaher, 01/17/2020
- Re: [grouper-users] PSPNG and (not)provisioning groups, Marwan Ali Shaher, 01/17/2020
- RE: [grouper-users] PSPNG and (not)provisioning groups, Hyzer, Chris, 01/17/2020
Next Grouper Call: Wed Feb. 5, 2020