- Chris Hyzer, Penn, Chair
- Chad Redman, University of North Carolina Chapel Hill
- Shilen Patel, Duke
- Carey Black, the Ohio State University
- Jeff Williams, University of North Carolina Greensboro
- Vivek Sachdiva, independent
- Emily Eisbruch, Internet2
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
Grouper School June 2-3, 2020, ONLINE
- More than 30 people signed up
- Chris and Bill will be trainers
- Modernized the GTE, Grouper Training Environment
- container w MSQL , LDAP , SHIB IDP,
- Fire all that up, install the Grouper database, run GSH commands, make a snapshot and make that container
- EC2 instance with different containers
- Shell script commands that do docker things
- Have Grouper training content for containers now
- Content from wiki plus exercises
- Maturity levels, what you need to run Grouper
- Hands on work
- Maturity level -1 quickstart
Should we list the web browsers that Grouper supports on the specsheet? Or add the version number?
You have to be on this version or up? From Java script , standard with JQUERY,
There are issues with IE
Decided specsheet is OK as is
Current work tasks, and next tasks
- Grouper Data configuration
- Refactering common parts in config and put in abstract class for other use cases
- Adding more validation, such as read only
- Pre built classes for jobs , don’t want them to be changed
- So read only
- Properties are showing without text, so creating text
- For read-only w Expression language, remove checkbox
- Screen that edits a daemon, a job that has a class , you are never going to change class that runs,
- so can change schedule but not the class
- Hard coded, Not regex
- Will be done in a few days
- Then Vivek will start working on provisioning
- External systems and daemons are similar
- Vivek made a super class for those
- Base config class, thing specific, then sub classes
- Use base config as a super class of the provisioning config
- Think of what a provisioner is , configuration of a provisioner
- Pivoting from PSPNG , due to performance and other issues
- New Framework with Base class, some common framework things, using the sync tables in the database now, interface with changelog in form of ESB, ESB knows the provisioner is getting messages, will send in bulk,
- will say this is not a change, or this is not relevant.
- Important design moment, we all need to take time to focus and get requirements
- Then migrate current provisioners
- For Grouper 2.6, everyone migrates to the new provisioning framework
- At a high level, provisioner has a bunch of options, will get all memberships,
- Ways to write your own spin
- Maybe framework
- Share a general framework plan on the wiki with Grouper-users
- Pull in the customer base, ask which services most interested in for a provisioner
- Should we do a survey?
- CON ID Framework
- Adapter that goes to CON ID later?
- Chad - inherited a custom provisioning system, hope to move to new Grouper provisioning approach in next 6 months.
- Did work with Azure provisioner
- Implemented midpoint as part of CSP
- Currently 2 projects at UNC, management leading toward Midpoint
- Also deciding LDAP or AD
- Chris: for basic account work, Grouper can do it, and can do access part,
- Midpoint is still a good provisioning engine
- LDAP and SQL data moves, for fewer dependencies and higher performance, might use Grouper provisioning
- But to Box or Azure, might use Midpoint
- Even if you have provisioning system, good to have hooks for troubleshooting access, Grouper can offer read only operations
- Provisioner can read things,
- Bridge to Loader
- Reports on who has access
- To help deprovision
- Interactions to remove could go thru midpoint , but Grouper could at least know
- Agree with the goals
- In larger ITAP scope, each component can stand alone
- If designing as cleanline interface, lower complexity of understanding Grouper interface.
- Redoing Grouper for DUO was challenging
- MAVEN has helped
- Loosely coupled interface is good
- Message client interface could be good
- Suggest that the provisioning framework support attribute mappings from each Subject Source to a set of “provisioner attributes” for that provisioner.
- 2 camps:
- Change log consumers tightly coupled
- Change log consumers loosely coupled
- Change log consumers tightly coupled
- Option to run in Grouper loader demon , or run separately
- As we go forward, for troubleshooting and caching, it’s increasing difficult to support loosely coupled
- Those using containers say they can spin up a container for each provisioner
- But this results in too many containers, too many projects
- Better to use Grouper daemon and have fewer containers
- Shilen agrees, need to keep it simple
- For diagnostics , we know every daemon is working
- If using message queues instead of daemon, need to do more monitoring
- self contained library
- Can have a maven dependency on Grouper, this is done at UNC
- Single dependency for communication
- Goal: for simple provisioner, such as BOX, it should be very simple, Grouper will figure out the adds for you
- Data structures to be used for all provisioners
- Subject API for identity or for account
- Mappings between subject API and provisioners, good way to genericize
- Chris: with Grouper now, no easy way to have multiple accounts for a subject
- A subject is an entity
- So now subject API is an identity API
- Carey: advantages to subject API being an account
- Correlation level above subject API (An “Identity API” could be added that correlates Subject API’s )
- Chris: In training next week , every person is one subject , don’t have multiple subject sources,
- Chris will work on the planning documentation to move this forward, LDAP provisioning as initial push
- Vivek: the vision makes sense
- Fixed issue w Grouper recent memberships , related to display name
- Grouper V 2.5.29
- Socialize that more since it improves how loaders manage groups
- Improved how Enabling / disabling memberships work,
- UI issues w immediate membership attribute assignments
- Attributes on memberships at Duke: mostly for card system, keep track of some attributes, use comment field
- Fixed Issues w inheritance and resources and actions,
- Shilen will work on the USDU issue,
- Add column or add attributes?
- Go straight to members table
- Consider if subject is deleted don’t return it
- If limbo, do some logic
- Adding column is better to avoid performance issues
- Flag for unresolvable and flag for deleted
- Do we need to add timestamp, flag, or both?
- For PIT, don’t need to make a change
- Wrote a script to parse out config properties and see which not used
- Sent to the Grouper Dev list
- A few duplicate properties , should be cleaned up
- Huge number of unused properties
- About half of total
- Penn still has the LITE UI
- Some not in current source code
- But still in LITE UI
- Some are dynamically built
- Chad can add LITE UI to the mix and get smaller list
- Best path is to remove 1st instance of duplicates
- Put the others config properties that are rarely or not used in separate file
- Some are missed because there is a single JAVA class that is a bean window
- Great report, thanks Chad
- Looked at PSPNG and docker containers
- Jeff, testing harness for PSP NG was created by Bert , runs through a series of tests
- Docker Compose, depends on environment variables being set.
- One LDAP container
- Using it with different volumes
- Create a new docker container based on it that adds the eduperson schema
- If we can maintain own subcontainer or keep Docker container where you build it.
- Steps: you start sub container w a certain state, then you can run a JUNIT test
- Now everything is like a GSH script, but that is not efficient
- If Chad makes changes to PSPNG , what’s the best way to run the tests?
- Use the README
- Was run locally on mac
- Chad has LINUX
- Chad will try to run the test
- Provisioning tests and LDAP loader tests
- Best to have a Docker container
- Grouper Training Environment
- Grouper Training Environment developer notes
- Grouper Training Environment - text to copy and paste
- Connecting to the AWS Training EnvironmenT
- GTE commands
- Grouper container documentation for v2.5
- Install the Grouper v2.5 container with maturity level 1 manually
- Install the Grouper v2.5 container with maturity level 0 manually
- Install the Grouper v2.5 container maturity level -1 quick start
- Install docker postgres database
Grouper database migration utility
- Organizing services in Grouper
- Grouper subject API diagnostics in UI
- Grouper configuration files and overlays
- Upgrade from Grouper v2.4 to v2.5 on the demo server
- Grouper Container v2.5 running as non-root
- v2.5 Release Notes
- Grouper web services - authentication - built-in Grouper
- Grouper configuration in the database and UI
- Grouper Security Issue Report
- Grouper Packaging and Versioning for v2.5+
- Using SQL to do things in Grouper
- Grouper v2.5 container unit test
- Authentication to UI and Web Services in Grouper v2.5+
- GrouperShell (gsh)
- Grouper Container v2.5 running as non-root
- Grouper v2.5 customize container config files
- Grouper Wiki Issues
- Grouper demo Technical Administration
- Attribute Definition Save
- Grouper Web Services
- Get Audit Entries
- Starting with Grouper
- Organization hierarchies via the grouper loader
- Syncing groups between group management systems
- Grouper Web Services Versioning
- Add Member
- Group Save , etc . Chad updated many Web Services many pages
- Sample change log consumer
Lacey - need to set the ServerName directive in grouper-www.conf to allow shib to generate endpoints over https when behind a load balancer with SSL offloading.
Andy - using good ol' patch in our Dockerfile to fix up a few things. It is safer than overwriting upstream files because we see when we have drifted far enough away from upstream to need review or rebase of patches.
Chris Hyzer -this immutable container thing is just an issue. i would like to be able to make a 2.5.28 and not annouce it and do tests, and if i need to retag with 2.5.28 iw ould like to be able to do. at that point, after announcing, immutable...
Chris Hyzer -Grouper v2.5.28 sha256:537747 is released.
Shilen upgraded to 2.5.28 and love how easy upgrades are now once you get to 2.5
Andy -We have 2 custom changelog consumers that we have been using for a long time. When trying to validate that they work correctly in v2.5, I can't get DEBUG logs out of them
Michael Gettes 5:13 PM
could we get a “startup” and “shutdown” entry written to the loader log for loader/ws/ui/scim? happy to submit a jira.
Carey -dev server v2.5.28 container ( I am not clear on how to use the Basic authn built into Grouper. )
Chris Hyzer - could set a default folder for UI, I think we have one for WS, but fully qualified should do the trick, or a subject source
Michael G AD and LDAP - If you see Error 12 it could be you are hitting a Result Set config issue with AD.
Chris Hyzer Security advisory GRP-2705 for Grouper
Ryan R Quick question, is there a process in Grouper that builds the resulting group if the group is a composite?
Michael i run find bad membership dmn every 3 hours.
Robin Is there a different approach to change the "Institute of Higher Education" string on the mainpage and the footer than customizing the templates?
Erik C after I figured out the nagging problem of hung changelog consumers and fast-forwarding the counter, I keep getting these stuck full sync jobs as seen in my daemon jobs dashboard
Andy With 2.5.23, when I run the container as "gsh", I see the slf4j warning
Andy There are access logs being written inside the container at /opt/tomee/logs/. These are caused by the access log valve in /opt/tomee/conf/server.xml. That should be turned off or redirected somehow so that we don't write logs inside the container. … Double logging issue
Paul C I'm working on fixing this same thing for the shib IdP. In that case, they'll be sent to stdout using the mkfifo thing that we do in other places...
Chris Hyzer This is fixed: https://todos.internet2.edu/browse/GRP-2802
uses unix "patch".
Paul C having trouble wiring the mkfifo pipe for tomcat access logs in the shib IdP. Tomcat seems to ignore the existing pipe when it starts up and writes its own file with the same name as the pipe... THIS WAS FIGURED OUT
Carey would like to use a loader job to load a tree of groups. ( on the order of 30K groups in the tree. )
Paul R A number of us have been scratching our heads with a particular SQL loader job.
Chris Hyzer Grouper v2.5.29 is released. dozen fixes. no upgrade notes.
Chris Hyzer I updated wiki to explain our versioning strategy: https://spaces.at.internet2.edu/pages/viewpage.action?pageId=163119272
Patrick H running version 2.5.22 and have users reporting errors while trying to assign values to attributes using the UI.
Some users will run into it one day and the next be fine. I’ll put the error in this thread.
Lacey Trying to use the custom shell hooks in 2.5.28,
Andy sent this to grouper-users mailing list in February. I'm posting it here, too:
Chris Hyzer copied the grouper demo server 2.4 database to one for 2.5 and set that up with the 2.5.29 container
Josh Question about v2.5.29 using ENV variables in our Dockerfile...
If we have in our Dockerfile the ENV variable
Andy interested in having our git repo be environment agnostic.
Chris Hyzer you have the dockerfile, docker run command (with env vars), system env vars, system files, some sort of secrets manager. all those are option at your disposal
Andy How can we manage our configuration if it is stored in the database? I mean, one nice thing about baking the config into the image is that I know exactly what "version" of the config I'm running.
Is it possible to export the config from the database (non-default values)?
Andy when upgrading from v2.4 to v2.5 of Grouper, I see the following SQL logged during the daemon container startup
Paul R Anyone have experience with the usdu utility and the -delete option?
Carey With the new ldaptive subject API config ... how does one set a "init-param" value for subjectIdToFindOnCheckConfig ? (edited)
Nicholas Roy Is the ability to configure a loader job on a folder that is different from the target of the loader output a feature , bug, or “meh”?
Alex secret management question: I see from the wiki that consuming secrets via env vars is discouraged. I'm curious about the security implications here.
Josh getting CSRF errors when I set grouper-ui(2.5.29) properties for overriding org image and/or custom css file.
Carey where to find the stuff to customize that "Help" link ( In the upper right hand corner ) ?
Richard F would be nice to be able to change where that help link goes.
text config in database in 2.5.30, its already implemented and ready to go
Alex aurora cluster spun up in our eng apphosting env , trying the experimental database migration tool mentioned here. Getting a vague trace:
Brett -request for update wiki with screenshots from the new UI? I'm pretty new to attribute definitions and am having trouble translating to what the new UI has. https://spaces.at.internet2.edu/display/Grouper/Organizing+services+in+Grouper
Lacey -previously working subject db connection in the subject.properties file started failing
Andy - a way to control access to the UI, like the etc:webServiceClientUsers group for WS?
Carey I think the "container host name" is not helpful in the log information.
Carey What causes this configuration message on startup?
Carey -Questions about Grouper daemon "other job" to run a script :
Specifically about the GSH script part.
Josh - issue in 2.5.29 using a GROUPER_DATABASE_URL_FILE variable.
Josh I see container environment variables for running Apache/Shib SP inside the container, but just noticed that those are "up to v2.5.27"... wanting to "kick the tires" on that concept, but we're running 2.5.29.
Carey Problem with WS authN in 2.5.29 based image.
grouper demo email smtp is broken and stopping registrations from happening
env vars from files arent being trimmed
auto stop or restart daemon when not doing work after X days
members tab from group screen doesnt work when editing reports
make gsh addGroup idempotent (and other operations?) using GroupSave
improve logging of data migration
grouper installer press any key should say press enter
grouper installer is broken, will not install container
in installer for installing container ignore sources.xml line
grouper database migration should drop foreign keys and indexes first
allow customizable help url
allow consistent formats of image and css overrides
messaging to ws bridge not work for multiple configs
subject api diagnostics dont work in gsh
address tomee errors on startup
add ability to set log level in container, adjust defaults
config custom tag should have nowrap on the required indicator
container should be able to be stopped and started
tomcat can wait until other services are up before starting
pspng log bugs
allow overlays of script hooks
add container option to change users when running tomcat as user e.g. in demo server
add externalized text to ui configuration editor
Issues with attribute def name and action inheritance
Audit error in UI for effective membership attribute assignments
Error deleting immediate membership attribute assignments
gsh.sh should always be able to find java in the container
adjust tomcat access logs
provide ability to turn off full-sync of groups during pspng incremental
make sure grouper built in passwords do not have colons, and allow local entities to be able to log in
pspng throws errors when folders are deleted
change container to slf4j .25
Enabling / disabling memberships and attributes causes child objects to be deleted
provide an option such that when i select a folder/group from the graph ui that it opens it in a new window?
grouperRecentMemberships don't allow non-default display names and descriptions
grouper_loader_log start/stop/other operations
add GROUPER_APACHE_SERVER_NAME option in container
GRP-2792 add ability to decrypt file in container e.g. for ssl keys
GRP-2791 starting from gsh does not initialize database connections correctly
GRP-2790 add variables for tomcat ports in container
Grouper Users list emails
- Re: [grouper-users] Upgrade process from Grouper 2.2.2 to 2.5, Robert Bradley, 05/13/2020
- [grouper-users] Letting wheel group members to access the Miscellaneous page, Olivier Salaün, 05/14/2020
- [grouper-users] Security advisory GRP-2705 for Grouper, Hyzer, Chris, 05/14/2020
- [grouper-users] Problem with inherited Rights, Tibor Rudas, 05/18/2020
- RE: [grouper-users] Problem with inherited Rights, Hyzer, Chris, 05/18/2020
- [grouper-users] creating Grouper USER Subjects, T-Heetderks, 05/22/2020
- Re: [grouper-users] creating Grouper USER Subjects, Bill Thompson, 05/22/2020
RE: [grouper-users] creating Grouper USER Subjects, Black, Carey M., 05/22/2020
Re: [grouper-users] creating Grouper USER Subjects, Kevin Rooney, 05/22/2020