Note: this was an off-week Grouper call since we did not meet on Jan. 20, 2021 due to inauguration.

Attending 

  • Chris Hyzer, Penn, Chair
  • Shilen Patel, Duke
  • Chad Redman, University of North Carolina Chapel Hill
  • Carey Black, the Ohio State University
  • Vivek Sachdiva, independent
  • Jeff Williams, UNCG
  • Emily Eisbruch, Internet2


Discussion

Grouper Training Online


New Grouper release coming along, first release in about 6 weeks. 


Chad, question on visualization and new provisioner targets and what permissions / priv are needed to view.  Is there security involved in letting non-root users know that their group is provisioned?  Ideally, left to the deployer to decide, but that is not yet implemented. Right now, everything as wheel.

Current work tasks, and next tasks


Vivek – Subject source wizard

  • GRP-3099 subject source wizard
  • Changing how the configurations work
  • Under  Misc, now option for subject sources
  • Coming from multiple sources, properties file and database
  • Only things JDBC or LDAP are listed as editable
  • To comply w the new format, some tweaks to config will be needed
  • When adding subject source, only 2 targets supported, LDAP type and SQL
  •  
  • Make max results size not required, have a sensible default
  • May use convert button for non editable
  • Hope conversion from old to new is not jarring
  • Supporting 2 JDBC sub classes
  • We are supporting too many old things
  • Need to make plan
  • Upgrade for 2.6
  • Get rid of old
  • If using old property keys, no changes
  • If statements
  • Converts properties
  • Chris made JIRA , if using attributes,
    •  subject attribute matches source, 
    • or they don’t match or
    •  there is a translation
  • There are diagnostics
  • Shilen: for search attributes you can combine multiple into one,
  • You would select new, 
  • You might need to make an internal ?
  • Don’t want to have translations everywhere
  • Matt: conversion question….
  • Thinking about how the overlay will work
  • Give people a clue that this subject source is defined in database and in the files
  • Run the diagnostics, Safer to bounce
  • Convert to the new
  • End state is a better position
  • Change the text to Subject API attributes
  • Under actions button could have configure subject API
  • Decide before we release Grouper 2.6
  • UI is long, could use tabs but that has challenges too
  • These are administrative screens not used that often
  • This screen will likely be used once
  • Setting up a subject source is painful now and this is an improvement
  • Thank you Vivek
  • Vivek: Need to add validation, working on it
  • Attribute name and field name, get that code into this
  • Chad , this is better than doing everything in config file
  • Descriptions on the fields, more explanations will help
  • Registries subjects, need to see what we want to do 
  • Chad created a subject source,  did not need it, deleted it, but things got broken
  • There should be testing on this issue of deleting a subject source
  • Some subject things in Grouper properties
  • Not used a lot
  • Configs in files, for some properties if consistent among environments,
  • Don’t have to replicate 100 attributes in multiple environments
  • Source control is an advantage of using files
  • Make changes through GSH
  • That conflicts with having a UI to configure it
  • Need to document best practices
  • Jeff: subject defined in flat file
  • Do you have to go thru intermediate step?
  • Can have command line to convert a file
  • Code in Grouper API and call from UI
  • AI Vivek: look for subject source in other configs and develop plan, in subject properties and not source specific, get a handle on what’s out there


Chris – Provisioning, training

  • Email notification
  • Using properties wizard work, can get wizard for other jobs
  • Can get a UI for things that are scheduled
  • Include date things expire
  • Two types of emails: send to end users or send a summary
  • Only sent if there are items in there
  • Can have an opt out if you don’t want to receive 
  • Templating
  • List of maps, subject properties or columns
  • Carey: suggestion to add  file support 


  • Provisioning set up work
  • Each attribute will have translation type
  • Flat DN, group base DN
  • Some straight maps from fields
  • Search attribute is in object that the framework needs to search
  • DAO does not map
  • Search attribute does the mapping
  • Group search filter
  • Mark as search attribute
  •  


  Shilen – Provisioning

  • Will do some load testing
  • Will look at LDAP tests
  • Delete what does not work?
  • Issue on Slack? w bad membership finder change log consumer
    • Shilen has not been able to reproduce it
  • On Daemon screen  would be good to be able to change last sequence number for change log consumer, 
  • Need a new table?
  • Think about how this will work 
  • Look in existing change log consumers
  • Need an easy way to skip over a change


For LDAP and provisioning, we have more tasks 

  • DOA group entity


Issues around SCIM provisioner

  • SCIM is catching on, hopefully the object models could translate to other things, with tweaks
  • Hope to take this model and edit it and make it available for AWS
  • On the Mock Service side it will be specific to the product being supported
  • AWS does not fully support SCIM2
  • On client side it’s SCIM2 with some options
  • Mock service implements a web service of SCIM
  • Take code and edit it so Grouper web  service can serve SCIM against Grouper
  • IN future, we Remove Penn State SCIM approach
  • And support SCIM w Grouper
  • Won’t need TOMEE , can return to Tomcat
  • By Grouper 2.6
  •  
  • Chris working with AWS resource


 Chad – Provisioning, training

  • Adding new provisioning targets for visualization
  • Looks like PSPNG
  • Working on Azure provisioner
  • Make JUNIT test case to go against mock thing in Grouper UI
  • Existing change log provisioner for Azure does not work



Issue Roundup 


Jiras in past 3 weeks


GRP-3101

add entityAttributes provisioning for Grouper training

GRP-3100

add translation type and remote "isTranslation"

GRP-3099

subject source wizard

GRP-3098

Ability to limit provisioning to specific targets by group - vis


GRP-3097

Add LDAP to SQL sync tool   


GRP-3096

notifications in Grouper


GRP-3095

UI sorting of LDAP subject search results from a "free form" search


GRP-3094

unescape $newline$ when editing configs in ui


GRP-3093

sudo should pass env in container


GRP-3092

folder privs more actions button blank


GRP-3091

Support unlimited count of favorites


GRP-3090

Visualization display new provisioner targets


GRP-3089

Advanced Membership UI: Ability to create a group based on the filtered result set.


GRP-3088

enable grouper container to work with openshift


GRP-3087

aws scim provisioner


GRP-3086

Grouper Local Auth ( and the GrouperClient )


GRP-3085

Is there a way to send an email to the member who was just added to the group?



Grouper Emails in past 3 weeks


[grouper-users] Problems with 2.5 level 1 database upgrade, Andre Daniels, 01/12/2021



Grouper wiki updates in past 3 weeks




Grouper Slack in past 3 weeks

 

Carey -I have a general GSH problem that I keep bumping into .. and avoiding… but I would really like a “way to do it”.

Example… trying to use the “removeIf()” method on a Set. ( REF:  ………..

Justin  -Asking a probably no brainer question that I have struggled to understand. For Grouper WS, how does the act as work? From an audit perspective, does it record the user that authenticated and the user that it was acting as?

Brett  - Regarding the processing of group membership begin/end dates, how do folks typically handle timezone offset to ensure we're at midnight...? Is the locale a setting that controls this...? Is this in the schedule for one of the daemon jobs that we need to adjust manually when the time changes? 


Carey -Rule question: RE: Email notification on flattened membership add to group

   Is there a way to send an email to the member who was just added to the group?

  AKA: The rule supports using JEXL to produce a CSV list of email addresses, but how do you get the new Member’s email address in the context of the JEXL?

 

Chris H - The winner of the poll is "manual".  ….


Chris H  - We have been discussing end users scripting.  Here is an example of taking an Excel sheet and generating a grouper client script (note you can expand this to do other operations, this example only has create group and add member)

 

Tim D  - Is there a way to configure grouper to not audit a set of groups or better yet, an entire folder? I'm trying to avoid a lot of the rows that loader is dumping into grouper_audit_entry (edited) 


Carey   - GRP-3086 Grouper Local Auth ( and the GrouperClient )

Carey -I think we may have finally “broken the UI” ( by scale and mySQL performance )….

I was told by an app owner that they can no longer “open a Grouper folder” in the UI. It just hangs/spins.

  NOTE: I believe this is  more of an annoyance report. It’s not a “production is broken” report.

  The user can search for more specific folders and find them under that. ( I think. )

 

Scott K  I have a Grouper deployment running successfully in AWS ECS using Fargate. I am using image i2incommon/grouper:2.5.29. I want to upgrade to using image i2incommon/grouper:2.5.39. ….

Justin R -I recall some discussion on the grouper.text.en.us.properties file. Has it been migrated to the database in the latest version of Grouper? We have 2.5.39 up now in our development environment and see it in the list, but it appears blank so we want to make sure that is expected or not.

 

Erin  - Join us for Grouper training with Chris and Chad during the week of February 8th!  


Carey  - RE: Grouper rules use case - Veto if not eligible by folder

”Note, the ruleCheckArg0 is the subject source. If it is blank, then the rule applies to all subject sources. If it is filled in, then the rule only applies to that subject source.

”    I think that is a bit misleading.

Carey   - plug for upvotes/comments GRP-3089: Advanced Membership UI: Ability to create a group based on the filtered result set.

Liam  Is it possible to assign privileges recursively through the UI?  

 And is it possible to hide portions of the folder structure from people?


Jeffrey  - Has anyone used types to denote access groups for resources that are more sensitive than the garden-variety ones? 

Liam  - Why is the “actions” drop down empty in the group privs management screen?

Liam  -Is there a gsh method that will just return the last component of a group name (the path id?) instead of the full path?  Or do I need to do something with groovy to get that bit out?


Chris H - grouperUtil.extensionFromName(group.getName())

Carey  -A question about Grouper daemon "other job" to run a script .

   Is there a processing timeout that might interrupt such a script?

  

Liam  - What do I need to run GSH outside of a container?  libraries and conf files?

Liam  Is it possible to see the status of jobs that have been sent to the background (e.g. deleting a folder structure from the UI)?

Carey  -RE: Grouper+reporting Is there anyway to “Run now” for a given report? ( basically like the rest of the “Jobs” [Loader, etc…] )

Jeffrey  -     a behaviour that I've noticed with accessing the status, diagnosticType=<item> verb. On the WS there is no basic auth required so it's easy to use it as a load balancer check, healthcheck etc.

However the UI that location is still behind shib, should there be an exception for that url, or is that not possible? For  the UI I've been running the healthcheck against tomee directly using port 8080, but I don't need to do that on the WS so it's a more complete test in my mind.

 

Chris B   - at the risk of perpetrating violence against deceased equines, azure ad apparently uses "assigned" and "dynamic" to differentiate handcrafted vs. auto-populated groups

 

Jeffrey C  I wanted to go back to converting data for Grouper from ??? to postgresql (Oracle in our case).  Looking at https://spaces.at.internet2.edu/display/Grouper/Grouper+database+migration+utility I noticed the line “// note: make sure the source is readonly or your foreign keys will be hosed” I  wanted to be clear that,  that means that the  “grouper” database has been set to  read only via GRANT? I.e. there isn’t a need to make  the  entire  database a read only database via a config  or something  to that effect?

Also I  understand this is experimental but is there a plan  to  make this utility something more mainstream?

 

Tim D - For the parts of the Grouper UI that use a free form search, subjectApi.source.ldap.search.search.param.filter.value in my subject.properties, is there a way to tell Grouper to sort those search results?

 

Scott Koranda  @mchyzer I don't think anything is copied from /slashRoot when the command is /opt/grouper/grouperWebapp/WEB-INF/bin/gsh.sh, is that correct?

 

Scott K Nevermind. The correct statement is "/usr/local/bin/gsh is exec'd before any files are copied from slashRoot so the bug with /usr/local/bin/gsh cannot be worked around by putting a fixed version in slashRoot".

 

Carey  

RE: Grouper+SQL+database+provisioning+example+from+view

# fullSyncFull, fullSyncGroups, fullSyncChangeFlag, incrementalAllColumns, incrementalPrimaryKey

otherJob.test_sql_sync.syncType = fullSyncFull

What is the difference between those options?

 

Emily  -Enjoy this blog by @mchyzer on the New Grouper Provisioning Framework https://incommon.org/news/flexible-and-powerful-to-address-community-needs-new-grouper-provisioning-framework/  

 

Chris H  Grouper email notifications:

https://spaces.at.internet2.edu/display/Grouper/Grouper+daily+email+notification


Wil C  Has anyone setup a “backdoor” into Grouper when using Shib w/Apache, using an alternative URL and Apache’s native basic auth, for example? 

Liam -If I want the subject API diagnostics to try the webservice, I’m guessing I’m going to need to set the “test” webservice settings in… grouper-ws.properties (e.g. ws.testing.host)?  I’m running the container… but I want some visible proof that it’s working / configured correctly  

 

Erik C  I got   test environment into a bit of a pickle... I added using the Grouper UI a new (3rd) LDAP external connector, and related subject sources. Turns out that connector won't work quite yet because the AWS subnet is not configured correctly to talk to that LDAP.  Containers restarted, and inexplicably the logins for Grouper are attempting to go out to this new LDAP, and thus I can't get in. I never specified this new one as the "Default", I wanted to stick with the old subject source and LDAP, so I'm wondering can I (A) force logins to resolve using a single subject source, or (B) is there a single database entry I can mangle by hand to "break" this bad LDAP entry and/or subject source?

Carey  Does the “Import config file” replace all existing config values for that file or does it append to the config in the DB?

If I upload a file (Say grouper.client.properties)  with only one java property key=value line, will it remove all existing setting from the file or just add the new config to the existing values?

Liam - Is it possible to tie loader jobs to a specific instance of the daemon?


Justin R -  Just upgraded to 2.5 in our stage env and had really good success. Kudos to all the devs working to enhance the product. All the features coming in are awesome!   question, we had Duo functional prior, but it appears to be failing now. I don’t see anything in the logs - just that it couldn’t get all the way through the batch. It’s possible that it is failing due to issues getting outbound as i need to add opts to java for our proxy. Is there an arg for the daemon I should use for that?

 

Liam-  Is it possible to add additional attributes to the csv group export?  (specifically alternate subject indentifiers)?

Liam  -  Is it possible to set a custom attribute on a group with a loader job?

 

Chris H - Email notification use case:

https://spaces.at.internet2.edu/display/Grouper/Grouper+daily+email+notification+use+case+-+FERPA+notifications

Email summary use case:

https://spaces.at.internet2.edu/display/Grouper/Grouper+daily+email+notification+-+affiliation+expiration+report


Liam H - Anyone have suggestions as to what the map the grouper group attributes to when they provision into AD?  Do you use the grouper ID # or the UUID?


Jeffrey C -  messaging Queue question, Is RabbitMQ still the main queue technology supported by Grouper? I ask since I don’t believe RabbitMQ is one of the managed queue technologies in AWS. Also we would  have to run our own RabbitMQ as our integrations team is not willing to run  that service.


Steve Zoppi   The queueing protocol supported by all of the Trusted Access Platform components is: AMQP ... RabbitMQ provides that as does almost any other viable messaging engine.  The Trusted Access Platform uses the AMQP implemented by RabbitMQ as "standard" but any messaging engine that implements AMQP should work.


Liam  - Can a group display name reference a different folder structure than the name?

We’ve got reference groups based on department ID numbers, but our stakeholders want us to display names instead.  I guess we could use ref:hr:department for both, instead of ref:hr:deptid

 

Liam - Is it possible to cancel a running loader job?  I’ve got a job that’s been running since 1/20.  It normally takes 5 minutes to complete and, we’re going on “4 days 21 hours 23 minutes 23 seconds


Justin R -  I noticed a new job after the upgrade - change log consumer for find bad memberships. Curious what this does? Sometimes it is very fast and other times may not finish.


Liam   - Where are the additional attributes pulled onto subjects stored?   It doesn’t seem to be in GROUPER_PIT_MEMBERS or GROUPER_MEMBERS.  Are they looked up on demand? (edited) 


Liam  Is it possible to get Grouper to re-populate the DESCRIPTION column in GROUPER_MEMBERS?  Our is based on a virtual attribute, and I updated the attribute defintion

 

Chris H - An idea we have had for a long time is a subject API configuration wizard.  @Vivek Sachdeva has made this happen.  Thanks!  I need to document/polish it, but it does exist in 2.5.40 (will be released soon)


Carey Nice to see such an improvement from “managing files”. The project has made some real progress in dealing with config in the UI. ( Dynamic data/config driven.. very good stuff.)  


 

Next Grouper Call: Wed February 3, 2021




 

  • No labels