Attending 

  • Chris Hyzer, Penn, Chair
  • Shilen Patel, Duke
  • Chad Redman, University of North Carolina Chapel Hill
  • Carey Black, the Ohio State University
  • Vivek Sachdiva, independent
  • Jeffrey Crawford, UCLA
  • Keith Hazelton, Internet2
  • Chris Hubing, Internet2
  • Emily Eisbruch, Internet2

Discussion

 

 Grouper Training Online Feb 9-12, 2021 - went well

Current work


Vivek

  • GSH Templates
  • Allows Grouper users to run arbitrary GSH scripts
  • Hope to externalize with web services
  • Must configure template same way
  • Text will go back to web service
  • Have daemons that run GSH
  • Running each GSH line as its own autonomous statement
  • Will change how Grouper runs a script so it works same way
  • Figuring out what variables are available for the script.
  • There’s an output object to print lines of output back to caller
  • Can have from field output
  • Script might do some validation and end early
  • Can't just return from GSH so added a method
  • There will a wiki explaining how to handle  scripts
  • How to specify output
  • Discussion on Slack on support for web service types going forward
  • Will support JSON REST for new operations


  • This was Chris Hyzer's note on Slack:
  • “Seems like we might be ready to move in the direction of REST/JSON for web services.  i.e. keep supporting all the backwards compatible stuff (SOAP, REST/XML, REST/XHTML) we already have, but new services or new options to services will be REST/JSON only.  and by "REST", I mean our current HTTP/POJ formats. 
    Also we can more fully support SCIM as a service endpoint which is more restful and generic but not grouper specific.  We will also migrate grouper client to REST/JSON (instead of REST/XML) for examples.   ”


    • Keep other approaches in place for what already exists
    • Chris does not like REST response codes
    • An http code could help
    • GSH has built in commands that are not very useful
    • Starting point for a handful of APIs
    • Don’t want things to stop compiling
    • Save classes, stem save,
    • Method chaining
    • You won’t add a parameter
    • Finder classes, Grouper Finder , Stem finder
    • We will add more find classes and save classes for these scripts
    • Examples: dealing w memberships, privileges, attestation

    • Chris Hubing: the use case for Internet2 
    • Want Internet2 staff or a community members to be able to spin up working groups
    • And do operations thru the COmanage user interface
    • Spin up a new working group and associated services for that
    • Creating a group, adding a member to the group, attestation and privileges
    • Arbitrary attributes…. 
    • Example of attribute : Group in sympa needs to be moderated
    • Chris Hubing is working on a taxonomy around that based on the downstream applications
    •  
    • Perhaps attestation every 365 days… for the working group
    • Down the road, integrate with MidPoint, MidPoint integrates w Slack


  • Next tasks for Vivek
    • Work on web service wrapper
    • View button may not be critical path
    • Chris Hyzer and Vivek will talk about what’s needed for the Internet2 use case

Keith Hazelton:

  • Interested in API approach
  • Hope to  tell  people if you don’t have Grouper, but do have some other group system, here is what you do for the Trusted Access Platform 
  • Keith may create a write-up on this


More discussion

  • Vivek and Chris Hyzer are hoping to finish the script work by end of weekend and create release.
  • Will there be conditionals in the template?
  • Chris Hubing: yes
  • Chris Hubing may consult w Chris Hyzer on writing script
  • Running a GSH from JAVA now runs it wrong. Will change it to how Grouper Shell runs
  • Why do we need externalized labels and descriptions?
  • Same person setting up template is setting up the config.
  • ChrisHyzer: externalized text file has all the text for the UI,
  • Consistency
  • Could have trick where form shows text and puts it in externalized text file
  • Internationalization issues
  • Text and config files
  • Fall back to externalized text in config 
  • Shilen: have the form elements on page and put on externalized file in database
  • Hard to configure?
  • In the JSON config say “This is an externalized text link”
  • Chad: need to export two places when you export the config
  • Must remember when you change a field name in form, also change it in config
  • Decision: Think about putting  externalized Text file in database

  • Drop-down field type.. Is it hard coded?  
  • Look for groups in a certain tree for example
  • Chris Hyzer : that is in the the plans
  •  
  • Set a provisioner with one line, could be used in GSH templates

  • Security : Only Admins can get in to write GSH script
  • Create a super group of admins for GSH Scripts 

Shilen: 

  • Duke has some Grouper admins   who should not have access to this GSH template script feature
  • Chris Hyzer and Vivek will work on this
  • Create a group of template managers or super admins
  • Could send a notification if someone gets added to the template managers group
  • Is there an option to decide if this runs with a certain transaction? 
  • If there is a failure partway through
  • Makes it easier to retry
  • Do this in a transaction, will default to true
  • Yes, will implement this
  • Don’t want a partial template to run

Jeffrey C., UCLA

  • Challenge onsite keeping computer clocks in sync
  • Not Grouper's fault
  • Creating groups and assigning memberships 
  • Change log got add a member to group before it got create group
  • Change log got locked
  • Instead of using the timestamp of the node? , use timestamp based on database time
  • It was hard to find out what happened.
  • Found a page Chris Hyzer had written that provided clues
  • Had 28 groups impacted
  • VMware should sync to time servers?
  • Changelog goes by time
  • Getting changelog events correctly ordered, using SQL clock
  • Can do in changelog temp
  • Or use a database specific sequence
  • Worry about long running transaction
  • Assigning the time to that is better than doing a sequence
  • Database clock , get a function 
  • Issue w views and how they are concatenated
  • Have  grouper database function library
  • So all views would not be same? 
  • Inserts w hibernate
  • Just call a function in database
  • Opportunity to add more than just functions and views
  • Outside of hibernate, writing an application you want to default to current timestamp
  • It fills in a SQL level
  • Have timestamp column and don’t set it 
  • Need to capture this and see what’s possible
  • Not sure the priority  if Jeffrey C is all set
  • For Oracle
  • Put the solution into Slack
  • Query goes thru and finds create group, finds some other operation earlier
  • Creates an update statement, 
  • Concat
  • Update statement
  • Will give a series of commands,
  • Can fix in SSL or Shib
  • Harder to fix in Grouper
  • If JVM detects time is off in database, just exit?
  • Get current time from database and compare.
  • Get  hibernate to set time on a column..


Chris Hyzer

    • U. Arizona wants to sync folders from one Grouper to another Grouper using SQL
    • Wish the loader was more modular
    • New package Sync to Grouper
    • Behavior class
    • Sync groups, stems, composites
    • Go get from SQL, beans, Grouper
    • Or from another Grouper
    • Syncing composites from a database
    • Simple setting “I want to sync from another Grouper”
    • Carey: UUIDs, could have a membership save
    • Attributes on memberships
    • Move between MYSQL and other databases
    • Will this approach work?
    • Yes that’s what AZ is doing.
    • Better to do a cutover
    • This does not set the create date, Does do UUID
    • Issue of downtime involved
    • Stopping the database
    • This seems more in flight
    • This work can evolve
    • It’s intended for “I set up in test, want to move to prod”


    • Consider a Grouper version where we redo the database
    • Databases not good at managing UUID
    • Make your own UUIDs 


  • Here is  Chris Hyzer’s Slack note about this
  • We already have an integer id on several of the major tables . And we have a way to do this in a database independent way!  id_index. 
    So what if the id_index (int?) were used for primary keys and foreign keys and we do something better for uuids (store as binary? 
    try to have a more deterministic prefix?)  it would take a lot of effort, but it is worth considering...   
    We could mock up a bunch of groups and memberships both ways and see if there is a noticeable gain before we go down that path... 
    The attribute_assign table has 10 uuid cols out of 20. 
    The membership table has 10 uuid cols out of 16. 
    The group_set table has 18 uuid cols out of 22. 
    this design decision pre-dates everyone on the team including me, but I guess its never too late to revisit stuff 


Chad

  • Working on properties
  • Config values are missing
  • Tracking down
  • Updated JIRA
  • Script work: UNC can be testers
  • Chris: flesh out APIs , method chaining pattern
  • Attestations
  • Chad work on this?
  • For Azure, Chad can look at it 
  • Can Azure provision teams?
  • Another API call after a unified group in Azure
  • Need to be able to mark as unified or not
  • Other things needing support from Azure
  • Welcome message, may want to turn off in some cases
  • Carey: Good use case for a GSH script template? 
  • Change log consumer supports Jexel for name, description, Email , 
  • If we can’t support it, figure out what to do for people depending on it
  • Chris and Chad will chat about the gaps around Azure


Shilen

  • Membership finder changes
  • Able to sort and page using membership data
  • Sort based on membership time 
  • Make max page size configurable
  • Local entities weren’t being sorted properly
  • Fixed
  • Provisioning work: 
  • If you had one config for provisioning that was invalid, the daemon would delete all provisioning configs, it was calling attribute assign finder
  • It’s about marking things as configurable
  • Switched provisioning to store config ID instead of true false
  • Will change how propagation happens
  • Main problem, the provisionable stuff , when it propagates,
  • The daemon was inefficient and incorrect
  • Looking at the types and other things that propagate


Issue Roundup 


Jiras in past two weeks


GRP-3153

grouper custom templates via gsh


GRP-3152

AttributeDef trace privileges links back to subject page, not attributeDef page

 

GRP-3151

sync grouper from another grouper via sql

 

GRP-3150

when looking at provisioner, have edit button (other buttons too?)

 

GRP-3149

entity group link error should not appear

 

GRP-3148

loader queries should be able to start with "with"

 

GRP-3147

allow loader queries to be longer than 4k

 

GRP-3146

Local entities should sort by display extension instead of name by default

 

GRP-3145

MembershipFinder paging/sorting by membership fields

 

GRP-3144

subject wizard ldap search subject scope has one option

 

GRP-3143

provisioning metadata should not be assigned if no metadata there

 

GRP-3142

deleting an incremental daemon doesnt delete that daemon

 

GRP-3141

if you edit a provisioning daemon config it doesnt show the provisioner config id

 

GRP-3140

grouper instrumentation needs to clear out old server names

 

GRP-3139

ability to “run loader diagnostics” processes in a CI/CD pipeline

 

GRP-3138

add filter to usdu ui

 

GRP-3137

attestation email content (body) should be able to include more details about the group and/or memberships


GRP-3136

Allow MembershpFinder max pageSize to be configurable

 

GRP-3135

Better error message for users not allowed to login to Grouper UI

 

GRP-3134

client should show usage if no usage file on classpath


GRP-3133

Issue with daemon that deletes invalid provisioning assignments

 

GRP-3132

attributes where subject has attribute_read should not see it in the results of the combobox while finding attributes to assign


GRP-3131

folder privs should not show inherited attribute read/update if subject has create


GRP-3130

provisioningDoProvision should store the provisionerId if provisionable instead of true/false


GRP-3129

view provisioning on subject throws grouper session error


GRP-3128

add ldap loader filter (with cache)


GRP-3127

add ldap loader lookup by dn (with cache)


GRP-3126

add ldap loader lookup by dn


GRP-3125

config refresh doesnt work, casts to int when should be long


GRP-3124

Ability to limit provisioning to specific targets by group for WS


GRP-3123

check certain source for registry subjects after adding (default "jdbc")


GRP-3122

provisioning incrementals finds multiple subjects with same matching id, but they are the same


GRP-3121

Duo integration does not support more than one Duo environment ( sub account nor multiple accounts ) in configuration structure


GRP-3120

convertDnToSpecificValue does not unescape commas


GRP-3119

daemon configuration for reports


GRP-3118

grouperExternal subject source gets removed


GRP-3117

grouper sqs should take in region


GRP-3116

UI function to union/complement an id list


Grouper Emails in past two weeks



Grouper wiki updates in past two weeks



Grouper Slack in past two weeks

Jeffrey C  is there a per loader job failsafe limit available?  I found a global setting.

 

Jeffrey C  Most of our campus uses MSSQL servers, so we are forced to use them as a source for loader jobs.   today one of the DB’s was taken offline late because a processes that feeds it data took unusually long.   the loader didn’t seem to think it was an error that our user was kicked off and just happily removed a bunch of users since they were no longer in the loaded data.

I’m not sure how MSSQL reacts when a DB is taken down but I think it would have generated an error that I don’t think the grouper loader processes  detected.

 

Paul R anyone have experience with the aws sqs messaging in grouper 2.4.   


Ryan R  when new attributes are added to the subject.properties file what needs to happen in Grouper for the new attribute to be populated?

 d


Liam  Is it possible to run a membership audit report on a group of groups (composite or otherwise), and see the changes in the member groups?  Or if you want the changes in those member groups, do you have to go all the way down to the original children?


 

Erik  topic of membership math,   which Grouper database memberships views do what? Is there any documentation on how these are configured, and what are the differences (_v vs _all_v) ?  

 


Liam  Do I need to worry about this?  It comes up every time I go into the Daemon jobs..

Error: can’t find daemon config for jobName grouper_report_9d681135d4084d0ab06b992e47615c4b_71c5e29a75764ddc8b4ca2f6544738ff

 


Erik I've got several working LDAP loader jobs, I want to create an LDAP_GROUP_SIMPLE to slurp an AD group membership into a Grouper group. 


 

Carey  GRP-1747 Group 2.3 Function (UI button) --> View Audit Log SQL timeout

 

 

Carey  I have an app that has multiple folder structures ( dev, test, prod ) and they would like to “review the environments”  for deltas in Grouper privileges, missing groups, extra groups, etc…

  

Chris H  Seems like we might be ready to move in the direction of REST/JSON for web services.  


Justin  After the 2.5 upgrade, we’ve been trying to work through the Duo connector. 


Liam  anyone playing with the new provisioner yet?  Is it possible to only provision the policy groups from a policy structure that’s been applied to the application structure,  

 

Liam  If the provisioner throws this error…

 Why do multiple entities have the same matching id???..

 

Liam  Is there a grouper provisioner for Dropbox?

 

Erik  When I export a group in the UI (2.5.33), even if I say export "All Member Data", I am not able to pull out their subjectIdentifier or any other attributes that I made available via the subject.properties.  


Carey  I keep seeing this in my daily Grouper report… but I am not sure how to track it down. Any clues?

 

Liam   trying to run an incremental provisioner using the new provisioning framework, and I’m getting an INAPPROPRIATE_MATCHING error…


 Liam  Does the SCIM 2 provisioner work yet?


jasonrap    Is anyone running Grouper in Azure AKS?

Sudheer Are these properties used to grant access to non-admin users to configure loader jobs?


Sudheer   I’m getting this error when I try to configure a loader job by logging in into grouper as non-grouper admin user:

 

Erik  The way I understand the design, loader jobs are best left for the Grouper Admins, because they run as "root" and people could create loaders that impact folders they may not have access to. Basically, it's all-or-nothing with the privileges.

  

Mike P  Was looking at the Grouper database and was checking out grouper_change_log_entry.  Each entry has a change_log_type_id which points to one of grouper_change_log_type.  Why is a 40 char key used for this?  Wouldn’t an int suffice and save the rather massive indexes?

 


Jeffrey C  Our test grouper environment seems to have gotten the CHANGE_LOG_changeLogTempToChangeLog stuck with an error. I’ll put the error in a thread. This may  be related to an obliterateStem operation

 


Carey 

GRP-3137 attestation email content (body) should be able to  


Chris H

Some loader LDAP documentation / enhancements:

 

Andy M   I've got some SQL for a loader job that begins with a "WITH" clause.  The loader throws an error message, "Invalid query, must start with select". 

 

Liam  Any one know of a JEXL validator / tester?  (other than trying to do stuff directly in java?)


Liam In the new provisioner, if I set the group name to be required, could I use the validation expression to prevent the allow and deny groups from being provisioned? 



Erik our security team wants our 2FA-challenged group to be permanent-- you can get in, but you can't get out. So we want to create a "one-way" policy group,  


Drew 

In our Dev tier, we keep getting this error. I've tried to research it, and I find some references to others having had this, but I can't find causes or solutions.

 

Liam 

I’m trying to set up a provisioner to AD, and I’m not sure what attributes are required.  


Michael G 

anyone interested in sharing experiences running Grouper with GKE in GCP?  I'm looking to do it and hoping for advice/wisdom/perspective from those experienced doing so.


Bill K 

Someone from Indiana University of Pennsylvania is interested in see a Grouper demo.  Do we have any volunteers?

 

jasonrap 

  I need some help/ suggestions related to this article: https://spaces.at.internet2.edu/display/Grouper/Install+the+Grouper+v2.5+container+with+maturity+level+0+manually  I am on step 29, none of the log files are outputting to my system. 


Justin R 

  I am working with some email messages from Grouper and trying to insert a new line into the message from a rule. I’ve tried adding \n to the message, but it seems to be coming through as just a \n.  

 

Liam 

Anyone have suggestions as to a active directory attribute to use for the grouper group identifier other  than gidNumber?  We’re already using that for something else..

 

Liam  

Googling for the grouper javadocs sends me here - https://software.internet2.edu/grouper/doc/2.5.x/grouper-parent/index.html  - and I’m not seeing any documentation, just liinks to mostly blank pages.  Am I in the right place?


Michael L

my first research task is to extract a list of "folders" from our grouper environment for Splunk purposes...i'm hoping this is a place that can help me with that


Next Grouper Call: Wed. March 3, 2021

  • No labels