Child pages
  • 11-Nov-2020
Skip to end of metadata
Go to start of metadata

 

Attending 

  • Chris Hyzer, Penn, Chair
  • Chad Redman, University of North Carolina Chapel Hill
  • Shilen Patel, Duke
  • Vivek Sachdiva, independent
  • Jeffrey Williams, University of North Carolina Greensboro
  • Bill, Georgia Tech
  • Emily Eisbruch, Internet2

 

New Action Item from this call


Discussion

Reminder, Grouper Session at InCommon CAMP, Tuesday Nov. 17, 2020

Current Work

   Vivek – Provisioning

  • Has configured one test provisioner
  • Shows what activity has taken place 
  • Can show up to 400
  • GC Grouper Sync table
  • Horizontal scroll , how to handle where there is a lot of data
  • Chris: See Daemon screen, there are extended details available
  • This screen is to see things happen, there are other screens for specific details
  • This screen is fine for now, could add a MORE button later
  • Entries sorted based on time, most recent
  • Change title to “Recent Provisioning Activity” 
  • Quick way to see if provisioner is doing something



 Chris – Provisioning

  • Grouper Release last week, then security release this week v2.5 Release Notes
  • Make two dot releases of the container
    • 2.5.37.1
    • 2.5.36.1
  • For open JDK, when Grouper 2.5.38 is released perhaps we should deliver 180265
  • Maybe it’s fixed.


Null member attribute issue

  • Added attribute metadata for default value
  • Must be sure it does not say around if it’s not blank
  • This works for now 
  • Dummy field for non blank
  • Hard to maintain this list of attributes in the config
  • Something to generate this? Or a way to express in the metadata that dynamic screens could look at? 
  • Would be nice, this is too big
  • Something in the configuration that will iterate, then we need a way in the metadata to say repeat this 20 times or something
  • To reduce the copy and paste
  • Chris and Vivek will look at the attribute issue


Three kinds of provisioning 

    • Memberships that are a group attribute 
    • Take attribute where memberships will go
    • Get memberships into the attribute
    • When memberships are evaluated there are many variables?
    • Provisioner has grouper data, sync data and target data,
    • When you do full sync it would be nice to take what’s in target we know about and update the sync tables. 
    • Change the flag that’s in the target
    • Hard to get the value and get back to the member
    • Shilen: If you have the DN of an LDAP entry , getting back to the object in Grouper…
    • Take the expression, make sure it is subject ID
    • Use scripting
    • Instead of translate expression from membership, you need to put something in the link field in the entity, 
    • In the sync tables we have data, subject ID, subject Identifier 0
    • Should that be in from or to ID fields?
    • Config go to one of the from or to ID fields?
    • Expression from membership picks one of those fields
    • It goes thru each membership and does an expression language and format the data to put into the attribute value
    • Instead when we go thru the members we should run the script for every member
    • Then at the membership part, just copy it
    • Shilen: makes sense, agree with the proposed approach
  • Chris will add new attribute


  • For full syncs what are the other gaps?
  • Shilen will do load testing
  • Chris doing incremental stuff, many tasks, will do a commit
  • Two types of data can be used 
  • Can take things to do without a full recalc from the target and things to do WITH a recalc
  • By default when things get pumped thru changelog we will NOT recalc
  • If group you are adding to is provisionable, will do  an ADD
  • If sync is messed up (already in target) we will recalc
  • Each event has timestamp
  • If there are many incrementals and then there is a full sync
  • Then when it goes back it can toss the ones that happened before the full sync
  • Events that happen during a full sync might need a full recalc
  • If you have async full sync, it will be OK
  • It’s the synchronous sync that causes more issues
  • Or maybe both async and sync need to be considered 
  • Moving from ESB listener to provisioner
  • Soon it will be time for a release where users can kick the tires
  • Vivek: can you send same event multiple times? For example, you add a member to group in Grouper and then add same member to group in LDAP side
  • Chris: change log is flattened. So it’s a no op
  • DAO will likely throw an exception, depends on the DAO implementation, when we give advice,  we can include suggestion for checking to be sure if an event is done
  • If sync state is wrong, it should be updated and say it’s in the target
  • Vivek: Re running full provisioner and incremental, do we have to handle those cases or can we rely on logic used for doing same operation multiple times?
  • Chris: trying to improve on that, to avoid blips
  • Issue of last-sync timestamp start and end
  • Chris will do some DDL work to implement 
  • There is grouper sync membership view to help troubleshoot
  • Will join with sync tables


Shilen – Provisioning

  • LDAP , renames now work properly
  • Did first pass, will need more work later
  • For some LDAPs if you change location of an object there can be failures
  • Might need to delete and recreate an object 
  • Started on Box provisioner
  • Will have another pass at that later this week
  • Calling the Box jar directly
  • In the commands class , no way to look up an individual
  • But the Box jar can look up an individual
  • Stop using the commands class, ( except possibly to make the connection)
  • There is a Box external system
  • Code to do the connection might be in the Box external system
  • All the connection-related config can use from the external system config?
  • Chris: 
  • Did not want to break existing things for external system
  • So made other things
  • Can copy and paste from external system
  •  
  • Shilen will do load testing
  • Chris will get incremental unit tests to work
  • Shilen  will work on Ldap specific tests, before the load testing

   Goal: two weekends from now

Chad – Provisioning

  • PSP NG testing added, worked last week 
  • Not working now, Failed last night
  • When kicks off Docker job, creates name
  • Name gets stored in Docker processes
  • This will be one off container that gets deleted
  • Solution: Use no name
  • Add a step to delete it
  • Another option, instead of -D , use --RM
  • So it automatically goes away
  • But can you do that and have it run in background?
  • On unit tests for the CI about 68 errors, used to be hundreds
  • Nine things in code but not in externalized text, need to see if they are used 
  • AI Chris handle JIRA 3016
    GRP-3016
    Extended audit log shows $$not found: groupAuditLogFilterColumnServerUsername$$
  • Chris will look at CI email

  • UNC has templates for policies, needs a template for a filter group
  • So people leaving the dept can get immediately removed
  • Is it straightforward to create a subclass for the templates?
  • To create a filter, 
  • Would still be in ad hoc
  • And would be eventually removed from ad hoc
  • Could create a rule, but that only works for one group
  • Filter group, source group, 
  • Chris and Chad will discuss this today at 3pm
  • Chris: Interested in more dynamic filters with GSH

Issue Roundup

Grouper JIRAs in past two weeks



Grouper Emails in past two weeks

Grouper Wiki Updates in past two weeks


Grouper Slack in past two weeks

Chris Hyzer   I got an email back from Kevin at Amazon.  He explained this as a SCIM push based provisioning, which we should put in the new provisioning system.  Do you know who needs this to work with Grouper and when?  

Liam  -I think that Dropbox also supports SCIM.

if you’re going to implement SCIM, I’d try to do it in a way that wasn’t limited to AWS

 

Keith Hazelton would you be able to join the Software Integration WG call this afternoon for 15-20 minutes to talk about the SCIM situation? The call is from 3 to 4 pm Eastern, you could choose any time in that hour. If not, we can discuss here on Slack.

 

There are a fair number of SCIM clients and SDKs now, I’m doing a quick survey now


Liam- Are there recommended dimensions for logos in the grouper ui setting?


Carey- I have observed a few cases where a single job was running on multiple daemons at the exact same time. Is that a known issue? Any ideas how to troubleshot that?

 

Scott K - I see that there are knobs for a loader job to prevent too many deletions. Is there a knob to rate limit the creation of groups and/or memberships? Something like “Do not create more than 1000 groups and 5000 memberships per run of this loader job”?

 

Erik C - I saw this in my logs today: uofi_urbana: Cache of grouper subjects is very full (100%). Provisioning performance is much better if grouperSubjectCacheSize is big enough to hold all grouper subjects.  

Peter -In the case where a number of Grouper groups were renamed (eg changed round parentheses to square brackets) and the api logs are full of references to the old group names (with the round parens) during provisioning. I can't find any instances of groups with the old names in grouper_groups, grouper_change_log_entry or when I query the grouper membership views. Is there another place that the old name of the group is being persisted? (in 2.2.1) (edited) 

Zachary - Pulling my hair out with privileges to apply templates.  

Liam- can grouper ingest “group” objects?  (our course enrollment information looks like ldap groups)…

Andy M   We create groups of groups using SQL loader.  We have a member view with subjects that are other Grouper groups.  We do this to roll-up some HR data (position profiles roll-up into job families).

 

Liam - wondering if grouper could ingest objects like that in an meaningful fashion

Chris Hyzer   Use the LDAP loader 

 

Will - you’re putting enrollment data like that in an LDAP/AD directory? Or the data that you’re getting just looks like LDIF?

Liam  We have enrollment data like that in eDirectory that’s used to provisioned course groups into Google.

 

Will  putting that in LDAP surprises me because it’s a lot of churn for what is usually expected to be a read-mostly data store

Christopher -we've had enrollment data in eduCourseMember in LDAP directory for quite a while (and in a local attribute prior).  we're planning to move away from that to having grouper populate regular enrollment groups and stitching them together into the policy/ref/basis model, and making the handful of apps that consume eCM switch to consuming groups instead

churn is not as bad since we include future, current, and a couple past terms, so we get a burst as the "bucketed" enrollment chunks of the student population enroll, then onesy-twosy changes the rest of the year

Jeffrey C    has anyone done an IGA gap analysis of Grouper? 

 

Chris Hyzer   Grouper 2.5.37 is released.   

Liam We’re using oracle as our database backend and the daemon has some hibnernate index errors when it starts.

Liam I’m trying to set up a group subject source.

Chris Hyzer  

Grouper has two ways to provision to AD/LDAP:

Flat: the CN is essentially the full ID path (system name) of the group (or some substring)

Bushy: the folder extensions in Grouper correspond to OU's in LDAP, and the CN is the group extension

For AD it seems like the CN cant be more than 80 chars or some constraint like that, so Penn does bushy AD provisioning.  Now we are integrating with a firewall where it wants to filter groups, and you cant filter by DN to get only groups in a certain OU since that is a dynamic attribute.  Flat would have worked fine, but bushy not so much.  So... what about flat and bushy at the same time?  The OU structure is bushy, but we also provision the full ID path of the group in an attribute that can handle more than 64 chars.  Then we can filter on full group name.  Best of both worlds?  Does anyone do this?  Is it recommended?  

BTW, does anyone know if they use a palo alto firewall with groups from AD (which in our case will be from Grouper)?  We are trying to get it to work now but if someone has experience that would be useful.. 


Andy  -We put the full ID path into the samAccountName so that there are no conflicts.   

Justin -Are there recommendations on the VM sizing for a Grouper Postgres database?

Richard F -   where to change some settings regarding the UI and what it does.  How does one configure the "Member name" part on a group filter.   Second is how does one configure the attributes exported, when one chooses Export Members -> All member data. 

Marwan  - link to the SCIM source repo referenced in the https://software.internet2.edu/grouper/doc/2.4.0-SNAPSHOT/grouper-misc/grouper-scim/scm.html returns a 404 page. Is there another working link to it? 

 

Liam  - Does convertDnToSpecificValueOrGroup still exist in grouper 2.5?  and if so… am I correct in interpreting the javadocs in thinking if it can’t find a match in the subject source, it will create a group in grouper?

Liam  - Is it possible to have two loader jobs populate the same group?  

Chris Hyzer Has anyone seen this intermittent apache error for tomcat?  

Chris Hyzer -We hope to see many of you at the Virtual InCommon CAMP and ACAMP coming up November 16-20.  

mikeporter What exactly does the SCIM component of Grouper do?

Carey - Question about Grouper rules use case - Veto if not eligible by folder

If you have two of those rules on the same folder but they are for different Subject Sources.

Would both rules be used depending on the subject source of the Subject being added?

 OR

Does that violate the “This is a special rule in that only one can fire, and it needs to be hierarchical.” restriction? 

Carey -Question about Grouper rules use case - Veto if not eligible by folder

   If I use that on a large number of folders… will it be a performance issue?

Variation on that…

  If I use it “near the root folder” is it better for performance than having it on hundreds of folders?

James B -Thinking about using Grouper Rules to send notifications on a few groups when a membership is about to expire (based on this example: https://spaces.at.internet2.edu/display/Grouper/Grouper+rules+use+case+-+Email+notifications+on+disabled+dates ).  We are on Grouper 2.4 (api patch 96).

Question:  Does the email that a group membership is about to expire in that above example get sent every time MAINTENANCE__rules runs as long as the expiration is within that timeframe?

 

Liam  - @black.123 you’d mentioned something in at thread (https://internet2.slack.com/archives/C7V0UQDJ4/p1604442062007300?thread_ts=1604435560.005100&cid=C7V0UQDJ4) about pre-digesting LDAP -> SQL

  if you want to do a “pre step” and write a LDAP to RDBMS table process. ( could be a GSH script )

Carey Black- And I do see the gap between having multiple attributes feed the Subject’s from a given LDAP source object. 

mikeporter   -The home page of Grouper says “Institute of Higher Education”  - How do I get that to say University of Delaware?  Or Mike’s Playland?

 

mikeporter  -A docker inspect of the ui service shows I have GROU.The database config shows it as http........s://teacup.nss.udel.edu/grouper.  Changing it in the database “sticks”, but the web page still has the old link.  


Chris Hyzer  Simple example of a GSH scheduled task to do some database cleanup

https://spaces.at.internet2.edu/pages/viewpage.action?pageId=175964952

Chris Hyzer -Grouper has a security vulnerability, affecting the following versions of Grouper:

  

Michael S I work for Unicon and we have a client that is having issues with the GcTableSyncTableMetadata.retrieveTableMetadataFromDatabase(String, String) method.  


Erik I'm attempting to upgrade to image 2.5.37, but things have turned sideways. Has there been some sort of change relating to certificate handling in Java?  The containers are stuck in an infinite loop of exceptions.  

 

mikeporter  I’ve gotten grouper-ui to proper load balance multiple containers (using HAProxy in a swarm).  Question: is there a simple way to configure grouper-ui to work over http?  Right now, on the load balancer, my frontend is decrypting the SSL, inserting/checking a cookie, dispatching to a backend (one of the grouper-ui) and re-encrypting.  While this works, it is obviously slower than not re-encrypting.  A magic environment variable I’ve missed?


Scott K  There is an environment variable, but I cannot recall it at the moment (sorry, on a call). I know it is documented. I use Traefik and terminate TLS there before routing to the Grouper UI container.


Next Grouper Call: Wed Nov. 25, 2020

  • No labels