Grouper
Child pages
  • 5-Feb-2020
Skip to end of metadata
Go to start of metadata

  

 

Attending 

  • Chris Hyzer, Penn, Chair
  • Chad Redman, University of North Carolina Chapel Hill
  • Shilen Patel, Duke
  • Carey Black, the Ohio State University
  • Vivek Sachdiva, independent
  •  Bill Thompson, Lafayette College
  • Matt Wolfley, Unicon
  • Andy Morgan, Oregon State University
  • John Gasper, Unicon
  • CU Boulder 
  • Emily Eisbruch, Internet2


  Action Items

 Grouper Action Items are here


New Action Items from this call

  • AI Chad will work on the cron job unit testing
  • AI Chris and Shilen will discuss automating the Grouper upgrade process
  •   AI Chad set up CI tests on Internet2 machine (maybe i2midev); try HSQL as db first; email group whether success or failure

 

DISCUSSION


Upcoming Grouper Training

Timing for Grouper 2.5 Release

  • In next 2 weeks, wrap up work on Grouper 2.5 release.
  • Then start testing and release procedures.
  • Docker containers is part of Grouper 2.5   Grouper dev and container strategy for v2.5
  • Some backporting may be needed
  • CI with testing
  • In Travis, hard limit of 50 minutes but it takes 3 hours to run the tests  
  • https://spaces.at.internet2.edu/x/UoLVCQ
  • Should have a Cron job on demo server to run the tests every day
  • Or a github hook
  • HSQL database  is fastest way to run tests
  • mySQL is installed 
  • AI Chad will work on the cron job unit testing
    • If failure it will send email
  • Daily email to core list about the unit testing would be OK

Development Environment 

  • How to Setup a Grouper Development Environment for Grouper 2.5
  • How To Setup a Grouper Development Environment for Grouper 2.4
  • Need to understand the steps to do troubleshooting
  • No more Ant
  • TOMEE (which is tomcat 8.5 with  more) instead of Tomcat
  • Need Java 8, can be Eclipse
  • Removed some source directories
  • All compiles to one directory
  • For the API, can set up a source folder in eclipse
  • Copy the client source, copy configs
  • Don’t need to do a maven install
  • To get started w snapshot dependencies, must do an install from parent project, can do that from eclipse
  • We are not running the client from a jar, we are running it from the source
  • Supersedes the snapshots
  • Will read the compiled classes
  • You don’t have to build a jar
  • You are editing java and saving, don’t have to do any builds
  • If doing an eclipse build, and doing a maven build, it downloads client jar and looks at maven class,
  • So if changing something in client need to do a maven install from the parent project?
  • Chris: don’t need to run from the Maven jars, just  work from source
  • Stay on eclipse
  • Put everything  in one commit
  • Include target class? This is a gap now?
  • Container puts everything in web app
  • In Dev environment , running GSH, easiest to do target classes and fully qualified class name for GSH
  • Latest version of eclipse does not take input and output in console?
  • Adding another layer, There’s Grouper, client and this new thing
  • You save time in long run
  • For Web App for web service, UI or SCIM
  • It’s similar to Grouper 2.4
  • But no ANT dev task to copy jars
  • Make a source path for jars
  • Have them copy over, maven copy dependencies
  • If class not found, run the maven copy dependencies
  • Fire up the web app for the 3rd party jars
  • Set up TomEE and eclipse and point to that directory
  • Some classes not found during preliminary testing
  • External folder helps
  • Should be an efficient approach
  • Troubleshooting involves looking in the folder  
  •  see if the arrows are configured right
  • If anyone wants to edit the wiki for IntelliJ please do

  • With TomEE running everything, maybe the SCIM and web service should be merged into one , if someone runs Web Service they have SCIM available. Then maybe more people would use SCIM. 

Containers 

Grouper dev and container strategy for 2.5

Grouper installer task to build the container v2.5

Grouper container 2.5


  • Vivek has worked on container
  • Chris Hubing, Internet2 would download installer 
  • Not from Internet2 download site
  • Rather download from Maven
  • Run it w Java
  • Use a  config file to answer questions
  • Downloads Maven and TomEE
  • Does a Git Export based on a tag convention
  • Goes to each project to do dependencies
  • Makes  a webapp dir
  • To copy 3rd party dependencies
  • Put TomEE next to it
  • Copy UI web service and SCIM
  • WebXML , made dynamic, don’t have to map
  • No tomcat authentication 
  • Basic authentication  is inside Grouper, can turn on w config switch
  • Don’t need to redeploy your container 
  • Can set username and pwd to do authentication module
  • Q:  does that break existing?
  • A: that will be in upgrade steps doc
  • Some may have to handle tomcat issues, may have to edit webxml
  • What about cas, shib?
  •   remote user will still work
  • Can  turn this on for a quick  start
  • For maturity level zero config, can you copy in from a config file?
  • Answer: just mount what we provide 
  • One webapp directory in the container
  • Should make image size smaller and easier to use
  • Note to avoid overlap, make order for copying  
  •  Now we pass in web service or daemon  
  • Does container need to know if it’s a UI?
  • Build out any directory structure?
  •  
  • Chris Hyzer will make a release notes page for each version
  • Will state if it’s ready for use or if it is experimental
  • Could have a column for known issues

  • Question: pass in arguments, for Docker containerized environment, you put secrets in the orchestration layer. Requires 3 or 4 sets of hibernate files, Would prefer as implementer and user, would want those separate.
  • One hibernate file and and environment variable
  • Needed to spin up apache and Shib
  • Answer: secrets are  another issue. Will work as it does now. Chris Hyzer will add to the diagram. Could have  same config files and overlays
  • All properties files should be same except grouper hibernate properties
  • Can use environment variables, scriptable from entry point
  • Chris Hubing: fun same container image, task definition tells it what to consume
  • Jexl can say , what did you call?
  •  
  •  What about logs?
  • Another argument you pass in?
  • Chris Hyzer:  these rotate
  • Container best practice, best to write to a volume
  • Standard docker storage layer is not designed for read /write
  • Can do grouper/logs
  • But want it to be isolated
  • Use EFS , cloudwatch
  • Tell maturity log 0 how to overlay the log4J file
  • Parameters in the logs, Internet2 or TIER logging methodology
  • All logs have a set of tokens
  • User specified token
  • Containers running multiple services, can be hard to parse the stream


  • Produce best practices Docker container
  • Chris Hyzer: at Penn, doing the containers there was an issue  with having files I want to put there, have a directory structure I want to put there, it can replace the web app directory, so tar that and commit it to GIT, then untar it so it will overlay but nor replace
  • Would be nice to simplify that
  • Use an rsync command?

  • There are docker and kubernetes methods for copying config
  • Anything that changes between environments
  • One place to put things to overlay would make life easier
  • Stack everything into one directory structure
  • Oregon State: best practices
    • Image should be same in each environment you run in
    • Then move it into production
    • Just pass in the secrets or config
    • Building a local image
    • Oregon State will evolve their approach over time
  • Maturity level 0, container has no secrets, there is no docker file
  • AWS secrets  manager is interesting, has cool capability
  • Suggested guidance, never commit a docker file with  a secret

  • Next step: Grouper team will get 1st pass to Chris Hubing for testing
  • Will discuss how to pass the 8 switches
  • Release steps are here  :  Release steps

Vivek – Building and packaging

Chris –  SQL sync, bugs

Shilen – permissions issues, other 2.5 tasks

  • Working on web service changes for enable and disable
  • Need to do testing
  • Changed queries required by web services
  • Suggestion: for initial 2.5 release, stay basic and do more in a 2.5 patch
  • There will be a list of To Dos


Chad – maven builds

  • Travis is working https://travis-ci.org/Internet2/grouper
  • https://spaces.at.internet2.edu/x/UoLVCQ
  • Commit in master branch
  • Will do build
  • Because it’s a tagged version will do maven steps
  • Should autoclose
  • Got out to the maven repository
  • Extract  of PGP signature, encrypted, need to update documentation on that
  • Run GSH from target, without building whole container
    • Two workarounds, can do it from eclipse, or do a java command w target dependency
    • Just say what the java command is
  • Grouper installer container method that looks for jar conflicts 
    • Need to make sure no Grouper sub projects have same library as parent project
    • Could  have transitive dependencies
    • If that happens we can fix in maven , by doing an exclude
    • Right now we are deleting the older version
    • Detected  by searching for same file 
    • Good to get those into maven so we test with the right version

  • Should container build fail if there’s a conflict?
    DECISION: Container  should  succeed in building with warning message if there is a conflict. So we clean  up the issue.  
    • We don’t change libs too often

  • Chad working on visual job project. Issue with horizontal spacing


 Bill 

  • Will work on ad hoc types
  • Put together the 2.5 Dev environment


Jeff –   not on today’s call, working on pspng


Comments on issue round up

  • For install issues,  we should be telling the community to use the container 
  • For Grouper 2.5 would like to automate the upgrade process 
  • AI Chris and Shilen will discuss automating the upgrade process  for Grouper 2.5 and above, 
  •  
  • Built in power user built in group would be helpful, or we could give advice on how to make this yourself. 
  • People want to be able to do things over GSH. 
  •  For screens  in the UI, we should develop functionality so that it’s easy to do it over GSH also. 
  • Chris has been  adding more diagrams to the Grouper wiki
  • Use Gliffy Images in Grouper wiki doc when possible. Don’t use JPG diagrams! Gliffy is easy to update
  • Meeting on Thursday 4pm ET to discuss provisioning 


Issue Roundup


Slack

  • Jan 23: Jeffrey C -  gsh  needs a lot of memory just to add a new folder or group? Rules, https://todos.internet2.edu/browse/GRP-2566
  • Jan 23: Tommy D - Grouper managing CCURE
  • Jan 24:Gettes - database technology
  • Jan 24: UI bug:  i submit an import of a large group or file or list of member IDs - and i get no spinner and no indication of doing work. GRP-2567
  • Jan 24 : Chris H -Selenium as a training tool
  • Jan  25: Carey - put ID's in html elements"    should be a requirement of all web apps
  • Jan 27: Jeffrey C - stop a specific daemon job that's running GRP-2569
  • Jan 27: M Gettes - print   graphs in Grouper is painful
  • Jan 27: Paul E With a pspng attribute provisioner, is there any way to get the ldap dn of the group?
  • Jan 27: Carey - Container , bashing , Docker
  • Jan 28: Rachel L - use a script to import grouper-loader.properties, grouper-ws.properties, etc. settings into the DB GRP-2571
  • Jan 28: SteveG -    migrating from 2.3 in a VM to 2.4 in a container
  • Jan 29: Alex P - nassign an attribute+attr value to a particular membership in the new UI
  • Jan 29 : Jeffrey C - using loader jobs to create the or hierarchy in folders,   inheritance seems to break
  • Jan 30: Josh  Gross GRP-2573 readonly sysadmin should be able to see privs
  • Jan 31: Rachel - Can folder types be set over GSH? 
  • Jan 31: BrettB-Grouper daemon Dockerfile HEALTHCHECK commands
  • Jan 31: Andy M -Grouper container issues https://todos.internet2.edu/browse/GRP-2578
  • Feb  4: Erin M : registration for the upcoming Grouper class is now live https://incommon.org/grouper-school-unc-greensboro/     


WIKI updates

JIRAs 

GRP-2580 Better detection of real vs non-real issues in logging of composite related membership adds
GRP-2579 permission limits not working in ui

GRP-2578 logging and container for 2.5

GRP-2577 container duplicate logs

GRP-2576 grouper.properties should not be in container for hsql, confusing

GRP-2575 UI (maybe WS too?) ability to bulk add list of Groups to a folder from a "CSV"/"copy and paste" inputs

GRP-2574 Please add a parentStem function to PspJexlUtils.java

GRP-2573 readonly sysadmin should be able to see privs

GRP-2572 fix cas in ui in 2.5

GRP-2571 allow import configuration to database from gsh

GRP-2570 Text Visualization for composites lacks "Total Member" count the way the graphical Visualization shows.

GRP-2569 Provide ability to kill a specific job/task/thread within Loader.

GRP-2568 grouper loader should give full stack in grouper_loader_log table if there is an error

GRP-2567 No busy spinner on large subject add in UI

GRP-2566 Performance issues for serveral operations when a lot of Rules are assigned

GRP-2565 Performance issues due to query changes in GRP-2117

GRP-2564 Internationalization broken by grouper_v2_4_0_api_patch_55 and grouper_v2_4_0_ui_patch_34

GRP-2563 add content security policy option to grouper ui

GRP-2562 point in time memberships can show 24 o clock

GRP-2561 replace newlines in descriptions with html newlines brs

Emails

[grouper-users] Oracle performance with membership queries, and histogram statistics, Redman, Chad, 02/03/2020

 Next Grouper Call: Wed Feb. 19, 2020

  • No labels