22-Jan-2020

Attending

Chris Hyzer, Penn, Chair
Chad Redman, University of North Carolina Chapel Hill
Shilen Patel, Duke
Carey Black, the Ohio State University
Bert Bee Lindgren, GA Tech
Vivek Sachdiva, independent
Bill Thompson, Lafayette College
Jeff Williams, University of North Carolina Greensboro
Matt Wolfley, Unicon
Action Items

Grouper Action Items are here

DISCUSSION

Administrivia

http://www.internet2.edu/membership/ip.html
Approve minutes
Review AIs Grouper Project Action Items (Google Doc)
Agenda bash

LOB discussion

In Grouper 2.5.0 patches we will support BLOBS in the database
for Grouper 2.5 Large Object Database Type
https://spaces.at.internet2.edu/display/Grouper/Grouper+lob+database+type
Framework takes it out of database and uses Temp directory in case it takes up a lot of space
Could have a UI to help manage the table\
For 2.5 put this in API, eventually in UI
Vivek: issue with slowness in a CLOB he used
Suggestion: Only leverage CLOB for over 4000
Chris: could do this for attributes.
- Chris will think about this
Q: extend this to group descriptions? So it would be possible to use any generic string and overflow from short to long storage,
Consider String for attributes, or lob as type. Consider string / lobs

Bert

Performance of pspng will be priority after Grouper 2.5 is released, in a month
Grouper Provisioning: PSPNG
In 2.5 patches, PSP performance will be a focus
For many sites with medium PSP provisioners, it can take a long time on change log and full sync
If Bert has time to help in about a month from now, that will be great.

Bert has worked to get Grouper 2.4 instrumented
- Objects nest in tree structure
- Full full sync and group full sync
- Chain of instrumentation
- Output on special performance log
- Time threshold (10 seconds)
- If takes more, then it summarizes itself
- Grabbing those objects, full sync time can be saved as an attribute
- To capture performance info
- Applying this approach to other places in Grouper would be helpful
- Two phases: short term gains from this experience
- Remote workshop for 2-3 hours with Chris, Jeff, Shilen, Bert
- To see how to integrate this with direction for other provisioners
- To increase performance, to increase data gathering around performance
- May focus on this in 1-2 months

Working with Gettes
Will do some R&D in next few weeks and apply that to Grouper 2.5
Should use a library (perf4j)?
AI Bert Commit instrumented approach to branch from 2.4
[AI] Bert create wiki draft of instrumented approach
Could have per group stats? Perhaps

Containers Strategy for Grouper 2.5

https://spaces.at.internet2.edu/display/Grouper/Grouper+packaging+and+versioning+strategy+for+2.5
Have a config that says auto-upgrade to 2.5, and container would auto-upgrade
Chris and Chad talking about how to compile JAVA
Whoever is ready to make a version of Grouper will put a tag on a branch and that will have a certain format
There will be a tag on a branch
A process will publish Grouper to Maven Central (Chad working on this now)
To get the container to Docker Hub, in Internet2 Grouper that does the packaging they’ll change Docker file to say what version they want. They have CI / CD, to check out from GIT and run the installer.
New Installer Task for the packaging managers
Downloads jars from Maven, uses GIT files that have been exported, runs script or Java commands, or something to Merge into one folder, a web app.
So no compiling, no tarballs, no patches
That container is listed as a release candidate, it can be tested, and deployed, then it will become the latest version. If a problem with it, we will move to the next number Grouper 2.5.8 can increment to Grouper 2.5.9
Working to get UI WS Daemon and SCIM running in one directory
There are still separate projects in GIT
Ant will be removed
Strategy to run a web app in GIT
Bill Thompson has been documenting that
May need one small Ant task for 3rd party dependency
Maturity Level 0 is running a container on a server
Maturity Level 1 , you might have a Docker File and can externalize config in other files, but still pwd and logs on server
Maturity Level 2, you have everything externalized, have pwd manager
Maturity Level 3, you are using orchestration
Maturity Level 4 might be working in cloud

For Maturity Level 0 ,

authentication can be a stumbling block
Some use Tomcat authentication
Problem because there is no way to dynamically tell Tomcat to use that, must change server XML,
Must change Web XML
Would like it to be a config option

So for Grouper 2.5, basic auth in the servlet filter. In addition, have filters based on source IP addresses and better auth for Web Services

Has been discussed on Slack: should Grouper have authentication or if it should always be externalized

Other local customizations might need to be in the model.

What is the methodology for injecting binary jars into containers?

Currently :

Low Maturity Level: if you have the files on the server that’s running the container you can mount the files into the container.

When you fire up Docker, you mount a source folder from outside Docker to a file inside Docker. Exception is that Chris Hubing, Internet2, has some built ins. (conf, lib?)

2. Higher maturity: you have a Docker file; libs are in your source control, when it builds container, it copies from your source control into your container. You deploy your container, Not mounting anything to external file system

discussion:

Managing Tomcat config is like injecting a Jar
Matt: trying not to make JSP changes, concerned about the hooks and change log consumers
Vivek adding authentication, removing tomcats,
making a few changes to Data Model
For Grouper 2.5 there will be a tables , including a table for recent logins
There will be a GSH way to set a pwd
Config that asks which of 3 approaches you want, basic auth, UI web service, or SCIM
Will hash and encrypt the pwd into the database
No more tomcat XML files
Later on, do Jason Web Token (JWT) approach in server and client
JWT is more secure
Authentication approach would be selectable on a per account basis, flexible
https://spaces.at.internet2.edu/display/Grouper/Grouper+authentication+in+2.5

Summary of work to date:

Simplified GIT structure
Chad working on Maven
Got web app running in one dir
Still need to do installer tasks for container installer
Another installer task to cover maturity level 0, with a wizard
Question: how does this impact major upgrades w DDL changes?
Answer: Don’t have GSH outside a container anymore
So spin up a container , could be maturity level 0, start GSH and do registry, as you do now. In future, might have a config option to do this automatically
Ideally automatic with some human intervention
Shilen: for just index adds or new tables, it’s fairly safe

Don’t too often have restructuring data in ways you have to be careful, but we need to consider those situations

When we get to that , we can look at why UI is not starting and try to absorb some of those errors
Allow person to go to admin screen, authenticate a user and sort it out
Or “Automatically upgrade to v 2.6”
always able to run one version back
If new container is running in older database, then it updates
n-1 approach
Chad:
For web services, need to support many web clients, ORACLE etc.
They have basic authentication built in
Need to support basic authentication as well as allow JWT and other methods
Can have self serve accounts for web services
Local entity concept w privileges
Generate pwd, either basic auth or JWT private key
That account will be set for that authentication
Question: Do you want to enable JWT at all?
What is model for changing pwd?
For web services, not going to allow people to set their own pwd. (similar to DUO)
If Grouper is going to offer authentication, how much do we need to provide?

Chad

Maven status
- Possibly we don’t need that grouper.version file

See info on wiki
https://spaces.at.internet2.edu/display/Grouper/Grouper+packaging+and+versioning+strategy+for+2.5
Suggested file called grouper.version, to be accessed by the installer, but would this be a hassle to maintain?
If Chris Hubing is editing his files manually
Installer would have version built in
ChrisHyzer: on downloads site, perhaps have a script working from a tag

Jars are in maven as snapshots and for Grouper 2.4.0
See tag, Start a release process, push to staging channel, administrator would release those manually, that’s described on the packaging wiki page as steps for developer.
Suggestion to have a separate branch and call it release.

Poms for maven now are -snapshot
Check in code? Or just local?
Can’t build from source code?
Should have official version pom somewhere
That’s a common workflow, Chad has read about
One branch called release, for every release it merges master and changes the version

DECISION: we don’t need a separate branch. Maven is the source of truth.
What gets built is local and goes away
Travis CI will parse and convert
No snapshots in Maven
Suggestion to ask Vivek to move his work to Master
Removing ant
Remove 3rd party libs from github
Remove client from Grouper
Publish instructions for how to develop that way
May be a couple of speedbumps but will work them out
How to run tests without Ant?
Use Java or GSH

Shilen

Permission change log changes (waiting for patch from Chris)
Updates for views for enabled flag on groups
Couple weeks ago finished propagation for attributes disabled when group disabled
Web services stuff that Vivek had started
Queries for M. Gettes made things worse for Harvard
https://lists.internet2.edu/sympa/arc/grouper-users/2020-01/msg00002.html

Undo for next patch? Yes this would be OK
Try to batch up the queries?
It’s a syntax thing
Hibernate and branching capabilities, Branch at hibernate level?
Shilen will do some testing and try to replicate the problem

What else for Grouper 2.5?
Show something disabled in the UI? This is for future.

Issue Roundup

Grouper WIKI pages updated

=======

SLACK

Jan 8 - Sudheer ; using pspng to provision grouper groups to LDAP and if group description has new linefeeds in it, The provisioner is assuming the first line as actual group description and the remaining lines as LDAP attributes
(this got addressed)

Jan 8 - J Crawford : does permission changes show up in the audit log? GRP-2544

Jan 8 - Carey : possible to have a rule set to "act as" the user who triggered the rule….
Rules would let me "link one to many" and "decide later to add another" ( condition, source, target, etc....)
Two new features: Ignore direct memberships from "g:isa" ( To allow nested groups)

Inspect indirect Memberships and create direct memberships when they are missing. ( and do it as the user who added the user to the other group. )

Jan 9 - Jeffrey C and Greg H: UI problem (fixed in patch?)

Jan 10 -Jeffrey C - bootstrap a grouper structure for an application that is org based. don't want it to manage users, just the groups and stems. trick in the loader to allow it to manage groups and folders but not the actual memberships?

Jan 10 - Rachel -possible to create Attributes over GWS

Jan 10 - Lacey V - provisioning groups to Active Directory (via PSPNG). AD admins are insisting that the samaccountname attribute match CN and samaccountname cannot contain colons. They also prefer a flat group structure in AD, for performance reasons

Jan 10- Carey - Q #1) Is there a way to restrict "normal users' from assigning "EveryEntity" privileges?

Q #2) Is there a way to restrict "normal users' from assigning "EveryEntity" as a member of a group?

Jan 10 -Carey - a user added EveryEntity to a group in their user folder. And suddenly all users could see the other users "user folder". It looks really odd.. and took me a bit to find the reason. " Should everyone be able to do that? "

Jan 14 - Alex P - PSPNG limitation, We're trying to setup an attribute provisioner to write <subject username>@<subdomain> to an attribute, where subdomain is pulled from a group attribute. The latter part isn't a problem, but it looks like subject information isn't being passed into the jexl parser for provisionedAttributeValueFormat. (CHRIS LOOKED AT THIS, SEEMS SOME TWEAKS ARE NEEDED TO PASS USER TO WHERE EXPRESSION LANGUAGE IS EVALUATED. Passed per group or per subject? Bert will take a look at this)

Jan 14-ChrisH - The way audits are stored is not conducive to querying.

Jan 14 - Anthony Hill - Install issues, container is better

Jan 15 - J Babb - Oracle to Aurora Postgres

Jan 15 - Haverkamp - pathways for an unaudited membership?

Jan 15 - PaulR -Performance issues w mass full syncs between grouper and AD.

Jan 17- J Craford - JVM options on the different grouper types, ui, ws, daemon?

Jan 17 - Shilen -formatting the queries differently significantly helped performance on mysql. But it turns out that the new queries don't work well for them but the old ones do. See email https://lists.internet2.edu/sympa/arc/grouper-users/2020-01/msg00003.html

Jan 17 ChrisH -Shib impersonate with grouper groups for who can impersonate and who can be impersonated. See here

Jan 22: J Williams : Upcoming change – Microsoft to disable use of unsigned LDAP port 389

Jan 22 - Alex P -issue with a rule: a problem where a group creation failed in the UI

Jan 22 - Sudheer - Can we migrate folders,groups and group memberships from one instance to another

=======

Jiras

GRP-2560 Consider a fake user for grouper loading groups with no members

GRP-2559 add sql other job to easily run sql commands

GRP-2558 grouper custom composite should not return groups

GRP-2557 when filtering by users who have update priv, should also return admin priv

GRP-2556 invitation: option to not try and add user to external subjects source if they exis

GRP-2555 rules dont fire for add member in ui when go from disable to enable