Attending
- Chris Hyzer, Penn, Chair
- Chad Redman, University of North Carolina Chapel Hill
- Shilen Patel, Duke
- Carey Black, the Ohio State University
- Vivek Sachdiva, independent
- Bill Thompson, Lafayette College
- Matt Wolfley, Unicon
- Andy Morgan, Oregon State University
- John Gasper, Unicon
- CU Boulder
- Emily Eisbruch, Internet2
Action Items
New Action Items from this call
- AI Chad will work on the cron job unit testing
- AI Chris and Shilen will discuss automating the Grouper upgrade process
- AI Chad set up CI tests on Internet2 machine (maybe i2midev); try HSQL as db first; email group whether success or failure
DISCUSSION
- Administrivia
- http://www.internet2.edu/membership/ip.html
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
Upcoming Grouper Training
- 4/29 - 4/30 at UNC Greensboro.
- @mchyzer and @Bill Thompson will be leading the class.
- https://incommon.org/grouper-school-unc-greensboro/
- Please promote at your institutions
Timing for Grouper 2.5 Release
- In next 2 weeks, wrap up work on Grouper 2.5 release.
- Then start testing and release procedures.
- Docker containers is part of Grouper 2.5 Grouper dev and container strategy for v2.5
- Some backporting may be needed
- CI with testing
- In Travis, hard limit of 50 minutes but it takes 3 hours to run the tests
- https://spaces.at.internet2.edu/x/UoLVCQ
- Should have a Cron job on demo server to run the tests every day
- Or a github hook
- HSQL database is fastest way to run tests
- mySQL is installed
- AI Chad will work on the cron job unit testing
- If failure it will send email
- Daily email to core list about the unit testing would be OK
Development Environment
- How to Setup a Grouper Development Environment for Grouper 2.5
- How To Setup a Grouper Development Environment for Grouper 2.4
- Need to understand the steps to do troubleshooting
- No more Ant
- TOMEE (which is tomcat 8.5 with more) instead of Tomcat
- Need Java 8, can be Eclipse
- Removed some source directories
- All compiles to one directory
- For the API, can set up a source folder in eclipse
- Copy the client source, copy configs
- Don’t need to do a maven install
- To get started w snapshot dependencies, must do an install from parent project, can do that from eclipse
- We are not running the client from a jar, we are running it from the source
- Supersedes the snapshots
- Will read the compiled classes
- You don’t have to build a jar
- You are editing java and saving, don’t have to do any builds
- If doing an eclipse build, and doing a maven build, it downloads client jar and looks at maven class,
- So if changing something in client need to do a maven install from the parent project?
- Chris: don’t need to run from the Maven jars, just work from source
- Stay on eclipse
- Put everything in one commit
- Include target class? This is a gap now?
- Container puts everything in web app
- In Dev environment , running GSH, easiest to do target classes and fully qualified class name for GSH
- Latest version of eclipse does not take input and output in console?
- Adding another layer, There’s Grouper, client and this new thing
- You save time in long run
- For Web App for web service, UI or SCIM
- It’s similar to Grouper 2.4
- But no ANT dev task to copy jars
- Make a source path for jars
- Have them copy over, maven copy dependencies
- If class not found, run the maven copy dependencies
- Fire up the web app for the 3rd party jars
- Set up TomEE and eclipse and point to that directory
- Some classes not found during preliminary testing
- External folder helps
- Should be an efficient approach
- Troubleshooting involves looking in the folder
- see if the arrows are configured right
- If anyone wants to edit the wiki for IntelliJ please do
- With TomEE running everything, maybe the SCIM and web service should be merged into one , if someone runs Web Service they have SCIM available. Then maybe more people would use SCIM.
Containers
- Vivek has worked on container
- Chris Hubing, Internet2 would download installer
- Not from Internet2 download site
- Rather download from Maven
- Run it w Java
- Use a config file to answer questions
- Downloads Maven and TomEE
- Does a Git Export based on a tag convention
- Goes to each project to do dependencies
- Makes a webapp dir
- To copy 3rd party dependencies
- Put TomEE next to it
- Copy UI web service and SCIM
- WebXML , made dynamic, don’t have to map
- No tomcat authentication
- Basic authentication is inside Grouper, can turn on w config switch
- Don’t need to redeploy your container
- Can set username and pwd to do authentication module
- Q: does that break existing?
- A: that will be in upgrade steps doc
- Some may have to handle tomcat issues, may have to edit webxml
- What about cas, shib?
- remote user will still work
- Can turn this on for a quick start
- For maturity level zero config, can you copy in from a config file?
- Answer: just mount what we provide
- One webapp directory in the container
- Should make image size smaller and easier to use
- Note to avoid overlap, make order for copying
- Now we pass in web service or daemon
- Does container need to know if it’s a UI?
- Build out any directory structure?
- Chris Hyzer will make a release notes page for each version
- Will state if it’s ready for use or if it is experimental
- Could have a column for known issues
- Question: pass in arguments, for Docker containerized environment, you put secrets in the orchestration layer. Requires 3 or 4 sets of hibernate files, Would prefer as implementer and user, would want those separate.
- One hibernate file and and environment variable
- Needed to spin up apache and Shib
- Answer: secrets are another issue. Will work as it does now. Chris Hyzer will add to the diagram. Could have same config files and overlays
- All properties files should be same except grouper hibernate properties
- Can use environment variables, scriptable from entry point
- Chris Hubing: fun same container image, task definition tells it what to consume
- Jexl can say , what did you call?
- What about logs?
- Another argument you pass in?
- Chris Hyzer: these rotate
- Container best practice, best to write to a volume
- Standard docker storage layer is not designed for read /write
- Can do grouper/logs
- But want it to be isolated
- Use EFS , cloudwatch
- Tell maturity log 0 how to overlay the log4J file
- Parameters in the logs, Internet2 or TIER logging methodology
- All logs have a set of tokens
- User specified token
- Containers running multiple services, can be hard to parse the stream
- Produce best practices Docker container
- Chris Hyzer: at Penn, doing the containers there was an issue with having files I want to put there, have a directory structure I want to put there, it can replace the web app directory, so tar that and commit it to GIT, then untar it so it will overlay but nor replace
- Would be nice to simplify that
- Use an rsync command?
- There are docker and kubernetes methods for copying config
- Anything that changes between environments
- One place to put things to overlay would make life easier
- Stack everything into one directory structure
- Oregon State: best practices
- Image should be same in each environment you run in
- Then move it into production
- Just pass in the secrets or config
- Building a local image
- Oregon State will evolve their approach over time
- Maturity level 0, container has no secrets, there is no docker file
- AWS secrets manager is interesting, has cool capability
- Suggested guidance, never commit a docker file with a secret
- Next step: Grouper team will get 1st pass to Chris Hubing for testing
- Will discuss how to pass the 8 switches
- Release steps are here : Release steps
Vivek – Building and packaging
Chris – SQL sync, bugs
Shilen – permissions issues, other 2.5 tasks
- Working on web service changes for enable and disable
- Need to do testing
- Changed queries required by web services
- Suggestion: for initial 2.5 release, stay basic and do more in a 2.5 patch
- There will be a list of To Dos
Chad – maven builds
- Travis is working https://travis-ci.org/Internet2/grouper
- https://spaces.at.internet2.edu/x/UoLVCQ
- Commit in master branch
- Will do build
- Because it’s a tagged version will do maven steps
- Should autoclose
- Got out to the maven repository
- Extract of PGP signature, encrypted, need to update documentation on that
- Run GSH from target, without building whole container
- Two workarounds, can do it from eclipse, or do a java command w target dependency
- Just say what the java command is
- Grouper installer container method that looks for jar conflicts
- Need to make sure no Grouper sub projects have same library as parent project
- Could have transitive dependencies
- If that happens we can fix in maven , by doing an exclude
- Right now we are deleting the older version
- Detected by searching for same file
- Good to get those into maven so we test with the right version
- Should container build fail if there’s a conflict?
DECISION: Container should succeed in building with warning message if there is a conflict. So we clean up the issue. - We don’t change libs too often
- Chad working on visual job project. Issue with horizontal spacing
Bill
- Will work on ad hoc types
- Put together the 2.5 Dev environment
Jeff – not on today’s call, working on pspng
Comments on issue round up
- For install issues, we should be telling the community to use the container
- For Grouper 2.5 would like to automate the upgrade process
- AI Chris and Shilen will discuss automating the upgrade process for Grouper 2.5 and above,
- Built in power user built in group would be helpful, or we could give advice on how to make this yourself.
- People want to be able to do things over GSH.
- For screens in the UI, we should develop functionality so that it’s easy to do it over GSH also.
- Chris has been adding more diagrams to the Grouper wiki
- Use Gliffy Images in Grouper wiki doc when possible. Don’t use JPG diagrams! Gliffy is easy to update
- Meeting on Thursday 4pm ET to discuss provisioning
Issue Roundup
Slack
- Jan 23: Jeffrey C - gsh needs a lot of memory just to add a new folder or group? Rules, https://todos.internet2.edu/browse/GRP-2566
- Jan 23: Tommy D - Grouper managing CCURE
- Jan 24:Gettes - database technology
- Jan 24: UI bug: i submit an import of a large group or file or list of member IDs - and i get no spinner and no indication of doing work. GRP-2567
- Jan 24 : Chris H -Selenium as a training tool
- Jan 25: Carey - put ID's in html elements" should be a requirement of all web apps
- Jan 27: Jeffrey C - stop a specific daemon job that's running GRP-2569
- Jan 27: M Gettes - print graphs in Grouper is painful
- Jan 27: Paul E With a pspng attribute provisioner, is there any way to get the ldap dn of the group?
- Jan 27: Carey - Container , bashing , Docker
- Jan 28: Rachel L - use a script to import grouper-loader.properties, grouper-ws.properties, etc. settings into the DB GRP-2571
- Jan 28: SteveG - migrating from 2.3 in a VM to 2.4 in a container
- Jan 29: Alex P - nassign an attribute+attr value to a particular membership in the new UI
- Jan 29 : Jeffrey C - using loader jobs to create the or hierarchy in folders, inheritance seems to break
- Jan 30: Josh Gross GRP-2573 readonly sysadmin should be able to see privs
- Jan 31: Rachel - Can folder types be set over GSH?
- Jan 31: BrettB-Grouper daemon Dockerfile HEALTHCHECK commands
- Jan 31: Andy M -Grouper container issues https://todos.internet2.edu/browse/GRP-2578
- Feb 4: Erin M : registration for the upcoming Grouper class is now live https://incommon.org/grouper-school-unc-greensboro/
WIKI updates
- Grouper provisioning / daemon tables
- Grouper provisioning strategy
- Release steps
- Grouper SQL database incremental provisioning
- Grouper packaging and versioning strategy for 2.5
- Grouper container 2.5
- Grouper installer task to build the container v2.5
- Grouper provisioning in UI shadow tables
- Grouper pspng changes in 2.5 initial release
- How to Setup a Grouper Development Environment for Grouper 2.5
- How To Setup a Grouper Development Environment for Grouper 2.4
- Export / import Grouper MySQL database
- GrouperShell (gsh)
- Grouper configuration in UI read-mode
- Install the Grouper 2.4 container with maturity level 0
- Grouper developers coding standards
- Grouper authentication in 2.5
- v2.4 Upgrade Instructions from v2.3
- Grouper dev and container strategy for 2.5
JIRAs
GRP-2580 Better detection of real vs non-real issues in logging of composite related membership adds
GRP-2579 permission limits not working in ui
GRP-2578 logging and container for 2.5
GRP-2577 container duplicate logs
GRP-2576 grouper.properties should not be in container for hsql, confusing
GRP-2575 UI (maybe WS too?) ability to bulk add list of Groups to a folder from a "CSV"/"copy and paste" inputs
GRP-2574 Please add a parentStem function to PspJexlUtils.java
GRP-2573 readonly sysadmin should be able to see privs
GRP-2572 fix cas in ui in 2.5
GRP-2571 allow import configuration to database from gsh
GRP-2570 Text Visualization for composites lacks "Total Member" count the way the graphical Visualization shows.
GRP-2569 Provide ability to kill a specific job/task/thread within Loader.
GRP-2568 grouper loader should give full stack in grouper_loader_log table if there is an error
GRP-2567 No busy spinner on large subject add in UI
GRP-2566 Performance issues for serveral operations when a lot of Rules are assigned
GRP-2565 Performance issues due to query changes in GRP-2117
GRP-2564 Internationalization broken by grouper_v2_4_0_api_patch_55 and grouper_v2_4_0_ui_patch_34
GRP-2563 add content security policy option to grouper ui
GRP-2562 point in time memberships can show 24 o clock
GRP-2561 replace newlines in descriptions with html newlines brs
Emails
- [grouper-users] Grouper patching I don't fully understand, Olivier Salaün, 01/22/2020
- Re: [grouper-users] Grouper patching I don't fully understand, Olivier Salaün, 01/22/2020
- RE: [grouper-users] Grouper patching I don't fully understand, Hyzer, Chris, 01/22/2020
- RE: [grouper-users] Grouper patching I don't fully understand, Black, Carey M., 01/22/2020
- [grouper-users] Novice users performing a fresh install of Grouper v2.4, LDAP connection issues, Prasad Karkhanis, 01/24/2020
- Re: [grouper-users] Novice users performing a fresh install of Grouper v2.4, LDAP connection issues, (continued)
- Re: [grouper-users] Novice users performing a fresh install of Grouper v2.4, LDAP connection issues, Shilen Patel, 01/24/2020
- [grouper-users] PSPNG with bushy LDAP, Morgan, Andrew Jason, 01/30/2020
- Re: [grouper-users] PSPNG with bushy LDAP, Greg Haverkamp, 01/31/2020
- [grouper-users] maven and grouper 2.5, Hyzer, Chris, 02/02/2020
[grouper-users] Oracle performance with membership queries, and histogram statistics, Redman, Chad, 02/03/2020
Next Grouper Call: Wed Feb. 19, 2020