Grouper Training Environment - text to copy and paste - 301.4

Reference Links

• Grouper wiki command reference for GSH
•  https://spaces.internet2.edu/pages/viewpage.action?pageId=14517859 
• Groovy syntax: https://groovy-lang.org/documentation.html
• For a quick language intro, see the Style guide page
• Also useful is The Groovy Development Kit page
• Grouper API Javadoc: https://software.internet2.edu/grouper/doc/ 
• 2.5.x (active release branch) -> Grouper API -> Project Reports -> Javadoc
• 3rd Party site Baeldung - Groovy blog posts: https://www.baeldung.com/tag/groovy/
• 3rd Party site MrHaki - Groovy blog posts: https://blog.mrhaki.com/

Exercise 301.4.1 - Getting started

Start up your Grouper docker stack. You will want the UI running (log in as banderson) so you can see visually how the shell commands impact objects in Grouper. 

With a container started and running, GSH can initiated by running:  

    ./gte-gsh

On start, you’ll see some helpful information about your grouper environment that will look something like this: 

Detected Grouper directory structure 'webapp' (valid is api, apiMvn, webapp)
Using GROUPER_HOME:           /opt/grouper/grouperWebapp/WEB-INF
Using GROUPER_CONF:           /opt/grouper/grouperWebapp/WEB-INF/classes
Using JAVA:                   /usr/lib/jvm/java-1.8.0-amazon-corretto/bin/java
Using CLASSPATH:              /opt/grouper/grouperWebapp/WEB-INF/classes:/opt/grouper/grouperWebapp/WEB-INF/lib/*
using MEMORY:                 64m-750m
Grouper starting up: version: 2.5.35, build date: 2020/09/16 05:32:38 +0000, env: <no label configured>
grouper.properties read from: /opt/grouper/grouperWebapp/WEB-INF/classes/grouper.properties
Grouper current directory is: /opt/grouper/grouperWebapp/WEB-INF
log4j.properties read from:   /opt/grouper/grouperWebapp/WEB-INF/classes/log4j.properties
Grouper is logging to file:   /tmp/logpipe, at min level WARN for package: edu.internet2.middleware.grouper, based on log4j.properties
grouper.hibernate.properties: /opt/grouper/grouperWebapp/WEB-INF/classes/grouper.hibernate.properties
grouper.hibernate.properties: root@jdbc:mysql://localhost:3306/grouper?CharSet=utf8&useUnicode=true&characterEncoding=utf8

subject.properties read from: /opt/grouper/grouperWebapp/WEB-INF/classes/subject.properties
sources configured in:        subject.properties
subject.properties internalsource id:g:isa
subject.properties ldap source id:   ldap: demo
subject.properties groupersource id: g:gsa
subject.properties groupersource id: grouperEntities
Type help() for instructions
Groovy Shell (2.5.0-beta-2, JVM: 1.8.0_265)
Type ':help' or ':h' for help.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> :load '/opt/grouper/grouperWebapp/WEB-INF/classes/groovysh.profile'
groovy:000> 

Note that the Grouper version is indicated, information on what the connection string to the Grouper database is, where various configuration files are read from, and available subject sources. If your subject source is based on a database instead of ldap, you will see information on the connection string to that database too.

A friendly GroovyShell prompt greets us at the end (groovy:000>) and we are ready to start interacting with GSH. We’ll start by looking up information about the subject banderson.

subj = findSubject("banderson")

Result:

===> Subject id: banderson, sourceId: ldap, name: Bob Anderson (banderson, )

Note that you do not need to declare your variables in GSH ahead of time and that GSH will output information about the last run command. This is helpful in validating that the banderson we found is in fact Bob Anderson, but let’s make sure that this is really Bob Anderson, the Grouper administrator, and not some other imposter Bob Anderson. 

getMembers("etc:sysadmin")

This time, we did not specify a variable at the end because we did not care to use the result for anything else. When we run that command though, we see an error:

ERROR edu.internet2.middleware.grouper.exception.GroupNotFoundException:
Cannot find group with name: 'etc:sysadmin'
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3GroupDAO.findByName (Hib3GroupDAO.java:1182)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3GroupDAO.findByName (Hib3GroupDAO.java:1144)
        at edu.internet2.middleware.grouper.GroupFinder.findByNameNoCache (GroupFinder.java:586)
        at edu.internet2.middleware.grouper.GroupFinder.findByName (GroupFinder.java:556)
        at edu.internet2.middleware.grouper.GroupFinder.findByName (GroupFinder.java:519)
        at edu.internet2.middleware.grouper.app.gsh.getMembers.invoke (getMembers.java:79)
        at edu.internet2.middleware.grouper.app.gsh.getMembers$invoke.call (Unknown Source)
        at groovysh_evaluate.getMembers (groovysh_evaluate:4)

GSH displays a friendly version of the problem (highlighted in red on most consoles) and we can quickly see that we just put in the wrong group name. The additional stack information is helpful if you ever need to report a problem out to the grouper-users email list or to the Grouper developers.

Let’s fix our mistake. Use the up arrow to go to your previous command to fix it up without having to retype the entire thing so it matches the command below:

getMembers("etc:sysadmingroup")

Result:

===> ['banderson'/'person'/'ldap']

There we go. The getMembers(groupName) method returns a list of Member objects representing the group's members. The result shows there is one member in the sys admins group. The output uses the object's toString() method to show a readable form of the members. 

In this next exercise, we will use that subject we found to create a few folders, a group, and add some members.

Exercise 301.4.2 - Creating a folder/group structure

Before we get started manipulating anything in Grouper, we will need to start a session. There are two ways to accomplish this, the first is starting a root session:

groovy:000> session = GrouperSession.startRootSession()
===> 7c847cea3822464dad3ccd12a22e3d2d,'GrouperSystem','application'

Or you can start a session as a specific subject and act as that subject:

groovy:000> session = GrouperSession.start(subj)
===> d0d33eeac9de4c54b96167d96dd7c3be,'banderson','person'

Note that we used subj from the above exercise where we looked up and found banderson. Let’s create a folder called tmp:

addRootStem("tmp", "Temporary Folder at Root")
===> Stem[displayName=Temporary Folder at Root,name=tmp,uuid=1bbb3572abe34e30bf1efcdf1842de5d,creator=3f8487547b2e4b118a61b66476f849c4]

The arguments for that command are the folder id and then the friendly name of the folder. Now let’s create a folder within tmp:

groovy:000> addStem("tmp", "subfolder", "Temp Subfolder")
===> Stem[displayName=Temporary Folder at Root:Temp Subfolder,name=tmp:subfolder,uuid=45d119187a4240169933fadc063febbc,creator=3f8487547b2e4b118a61b66476f849c4]

We specified the folder to make this folder in, the id of the folder, and the friendly name of the folder. If you make a mistake, whether it is a missing argument, or that it cannot find something you referenced in an argument, GSH will help you out:

groovy:000> addStem("tmp", "subfolder2")
ERROR groovy.lang.MissingMethodException:
No signature of method: groovysh_evaluate.addStem() is applicable for argument types: (String, String) values: [tmp, subfolder2]
Possible solutions: addStem(java.lang.String, java.lang.String, java.lang.String)


groovy:000> addStem("tmp2", "subfolder2", "fail subfolder")
ERROR edu.internet2.middleware.grouper.exception.StemNotFoundException:
Can't find stem by name: 'tmp2'
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.findByName (Hib3StemDAO.java:2139)
        at edu.internet2.middleware.grouper.StemFinder.findByName (StemFinder.java:229)
        at edu.internet2.middleware.grouper.StemFinder.findByName (StemFinder.java:199)
        at edu.internet2.middleware.grouper.app.gsh.StemHelper.addStem (StemHelper.java:73)
        at edu.internet2.middleware.grouper.app.gsh.addStem.invoke (addStem.java:71)
        at edu.internet2.middleware.grouper.app.gsh.addStem$invoke.call (Unknown Source)
        at groovysh_evaluate.addStem (groovysh_evaluate:4)

IMPORTANT: You are often acting as a super user in GSH, so do be careful with any commands to ensure you do not accidentally destroy any data in your Grouper environment. GSH will not prompt you if it is OK to delete something once you hit enter on the command.

Take a look in the UI (you will want to login as banderson for these exercises). We will see our new folder and subfolder created:

Note that the creator of the folder is Bob Anderson, because we started the Grouper session as banderson. If you started a root session, the creator would instead be GrouperSysAdmin.

Click on More Actions -> View audit log. 

Note that the audit log shows that this folder was created through a GSH command. This can be very valuable in determining whether it was Bob Anderson logged in through the UI creating folders or someone acting as Bob Anderson via GSH that created the folder instead. We’ll look at an example audit log with both when we create a group next.

Back in your GSH window, let’s now create a group:

groovy:000> addGroup("tmp:subfolder", "test_group", "Testing Group")
===> Group[name=tmp:subfolder:test_group,uuid=9a81949a4c09467192e933d946fe28c4]

...and add a member to it:

groovy:000> addMember("tmp:subfolder:test_group", "jsmith")
===> true

GSH prints out true because it successfully added the member ‘jsmith’ to ‘tmp:subfolder:test_group’. Here is a case where it fails to add the membership:

groovy:000> addMember("tmp:subfolder:test_group", "jsmith2")
ERROR edu.internet2.middleware.subject.SubjectNotFoundException:
subject not found: jsmith2
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.thereCanOnlyBeOne (SourcesXmlResolver.java:489)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier (SourcesXmlResolver.java:531)
        at edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier (CachingResolver.java:406)
        at edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier (ValidatingResolver.java:203)
        at edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier (SubjectFinder.java:365)
        at edu.internet2.middleware.grouper.app.gsh.addMember.invoke (addMember.java:144)
        at edu.internet2.middleware.grouper.app.gsh.addMember$invoke.call (Unknown Source)
        at groovysh_evaluate.addMember (groovysh_evaluate:4)
        at groovysh_evaluate.addMember (groovysh_evaluate:3)

In this example, we tried to add a subject that does not exist. GSH points us in the direction of where we went wrong.

(As a reminder, the arguments for all of these commands are available on the GSH wiki page) 

Let’s take a look in the UI at our new group we just created:

While you are in the UI, go ahead and also add ‘banderson’ to the group too as a member. Once you have done that, take a look at the audit log for the group:

Note how even though both group member adds were done as Bob Anderson, we can see that the second add was down through the UI while the first was done as a GSH command.

Exercise 301.4.3 - Working with the API

In addition to help methods like addGroup() and addMember(), All of the Java public methods in the Grouper jar -- plus methods from all the other jars in the WEB-INF/lib directory – can be called from within GSH. Up to now, there have been simple helper methods to work with groups and subjects. However, there isn't a helper command to retrieve the group memberships for a subject. But we can access other API methods to get them.

In this case, the GSH wiki section membership scripts has some code that looks useful as a starting point. It uses methods in the MembershipFinder class to set up a finder, add a subject, and do a search. The search results will be a set of MembershipSubjectContainer objects, which wrap the membership information including the subject and group. Looping through the resulting collection, we will extract the group from each container object and print the name.

First, let's make sure our subj variable is still banderson.

groovy:000> subj
===> Subject id: banderson, sourceId: ldap, name: Bob Anderson (banderson, )

Now we do the search to get the collection, and then loop through and print the groups found.

def msc = new MembershipFinder().addSubject(subj).findMembershipResult().getMembershipSubjectContainers()

for (def membershipSubjectContainer : msc) {
    println(membershipSubjectContainer.getGroupOwner().getName());
}

/* output:
etc:grouperUi:grouperUiUserData
tmp:subfolder:test_group
etc:sysadmingroup
===> null
*/

Note the addition of some Groovy syntax here. Groovy has type checking, which means that you can specify the type of a variable, to ensure you are getting back the type of object you expect. If you don't care about the type or don't feel like typing it in or looking it up in the API, "def" just indicates a generic object.

The example above shows Java-like syntax for looping through the collection. But Groovy has an alternate syntax that is less typing:

msc.each { membershipSubjectContainer -> println membershipSubjectContainer.groupOwner.name }

This is a common way to execute loops in Groovy. The "each" function take a block of code, setting the variable to each value in turn. Also note that the getXXX() methods from Java can be shortened to XXX, to save even more typing.

Exercise 301.4.4 - Automation

We have received a request from an application owner to:

• Create a folder app:foo:userGroups
• Given a list of user ids, create a group named as the id, but make the display name the user's first and last name (i.e., the LDAP "cn" attribute)
• Grant each user read and update permissions for their group

Depending on how many users are in the list, this is potentially tedious to do through the UI. All these steps can be automated with GSH. For the incoming ids, a data object can be created within the script and initialized. Depending on their needs, modifying this step to read data from a file can be a later enhancement.

The users' names don't need to be part of the incoming data. Because of how the LDAP subject source is configured, Grouper can access certain attributes from a subject.

groovy:000> subj.getAttributes()
===> [uid:[banderson], givenname:[Bob], sn:[Anderson], cn:[Bob Anderson], mail:[], employeenumber:[], title:[], dn:[uid=banderson,ou=people,dc=internet2,dc=edu], displayname:[Bob Anderson (banderson, )]]

groovy:000> subj.getAttributeValue("cn")
===> Bob Anderson

First, set up the ids, and create the folder.

def ids = [
    'abutler743',
    'awhite522',
    'cvales15',
    'elopez987',
    'jdavis393',
    'jwalters559',
    'lpeterson153',
    'mmartinez190',
    'omartinez57',
    'sgasper878',
]

addStem("app", "foo", "foo")
addStem("app:foo", "userGroups", "userGroups")

Do a quick test to make sure the subjects all resolve, and have a name field we can use.

ids.each { id ->
    Subject subj = findSubject(id)
    def cn = subj.getAttributeValue("cn")
    println cn
}

/*
Ava Butler
Andrew White
Colin Vales
Emma Lopez
John Davis
John Walters
Lexi Peterson
Mary Martinez
Olivia Martinez
Sophia Gasper
*/

Everything looks good, so do the loop again, but now create the groups and assign the privileges. The grantPriv helper command takes Privilege Java objects, in this case Privilege.READ and Privilege.UPDATE.

ids.each { id ->
    Subject subj = findSubject(id)
    def cn = subj.getAttributeValue("cn")
    Group g = addGroup("app:foo:userGroups", id, cn)
    grantPriv(g.getName(), id, Privilege.READ)
    grantPriv(g.getName(), id, Privilege.UPDATE)
    println "Done with ${g.getName()}"
}

/*
Done with app:foo:userGroups:abutler743
Done with app:foo:userGroups:awhite522
Done with app:foo:userGroups:cvales15
Done with app:foo:userGroups:elopez987
Done with app:foo:userGroups:jdavis393
Done with app:foo:userGroups:jwalters559
Done with app:foo:userGroups:lpeterson153
Done with app:foo:userGroups:mmartinez190
Done with app:foo:userGroups:omartinez57
Done with app:foo:userGroups:sgasper878
===> [abutler743, awhite522, cvales15, elopez987, jdavis393, jwalters559, lpeterson153, mmartinez190, omartinez57, sgasper878]
*/

In the UI, verify the groups were created. In the Privileges tab, spot check that read/update privileges have been created for the user matching the group name.

(Extra) Exercise 301.4.5 - Build a Report (making a GSH script)

In this exercise, we are going create an automatically generated report of all memberships for something like a daily report to a security officer around campus who is interested in who has access to what for auditing purposes. GSH can help us make that pretty simple without having to write any sort of database query.  We will be generating that report using GSH, but also looking at how to run a script in GSH without having to copy and paste all of the lines of the script. 

Open your favorite text editor and copy in the following script:

session = GrouperSession.startRootSession();
group = GroupFinder.findByName(session, "basis:student:exchange_students", true);
effectiveMembers = group.getEffectiveMembers();
immediateMembers = group.getImmediateMembers();
PrintWriter writer = new PrintWriter("/tmp/out", "UTF-8");
writer.println(String.join("\t", "id", "name", "Effective", "Immediate"));
for (Member m: group.getMembers()) {
    writer.print(m.getSubject().getId() + "\t" + m.getSubject().getName() + "\t");
    writer.print(effectiveMembers.contains(m).toString() + "\t");
    writer.println(immediateMembers.contains(m).toString() + "\t");
}
writer.close();

Save your script (in this case called report.gsh) and here are two ways we can run the gsh script:

Copy the script in to the container and then run it:

$ docker cp report.gsh 301.4.1:/tmp/report.gsh

Open a shell into the container and switch to the tomcat user:

$ ./gte-shell

$ sudo -u tomcat /bin/bash

Run the report

$ bin/gsh.sh /tmp/report.gsh

Note that data in containers is not persistent and it would be better to mount in a directory where you keep your GSH scripts. 

The output shows very little other than the output for each command run:

groovy:000> :load '/opt/grouper/grouper.apiBinary/conf/groovysh.profile'
groovy:000> :gshFileLoad '/tmp/report.gsh'
===> dcf994127b6147afa86d6f3569d131a4,'GrouperSystem','application'
===> Group[name=basis:student:exchange_students,uuid=de101843359046ff987d1ff167e8301c]
===> []
===> ['ddavis232'/'person'/'ldap', 'jpeterson243'/'person'/'ldap', ... ]
===> java.io.PrintWriter@7e0883f3
===> null
===> null
===> null
groovy:000> :exit

We can see our output report by running the following:

$ docker exec -i 301.4.1 cat /tmp/out

...and we see our beautiful tab separated report we can now ship off to whoever requested it.

You could execute your recurring GSH scripts either by calling the docker commands from cron, or perhaps creating a container based on tier-grouper that runs crond instead of any of the grouper components. Your container would be built with the crontab and you would be able to schedule any functionality you need with grouper components like GSH. But that is an exercise left to the reader (for now).

(Extra) Exercise 301.4.6 - Creating and Running a Grouper Loader Job

For this exercise, we are going to create a very simple loader job that loads all subjects from the grouper members database table and run it. While the UI does provide a very easy to use mechanism to create and run loader jobs, there are times where you may find it convenient to use GSH instead. GSH can be helpful if there are a bunch of loader groups you need to create and do not want to click through the UI for each one. Much of how to create Grouper Loader jobs in GSH is documented on the Grouper Loader wiki page. 

First, create the group.

addStem("test","subfolder","subfolder")
addGroup("test:subfolder","loader_test","Test Loader Group")

Now assign it the attributes needed to make it a loader job. This time, instead of writing one line at a time, try copying the entire block below in to GSH:

groupAddType("test:subfolder:loader_test", "grouperLoader");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderDbName", "grouper");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderType", "SQL_SIMPLE");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderScheduleType", "CRON");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuartzCron", "0 * * * * ?");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuery", "select distinct subject_id as SUBJECT_ID, subject_source SUBJECT_SOURCE_ID from grouper.grouper_members where subject_source = 'ldap'");

GSH handled the line feeds between commands and you should see each command run followed by ‘true’ printed after each line. In the next exercise we will show you how to run a block of commands as a script, which is the preferred way over copy/paste many lines at a time.

Now let’s dry run the job to see what it would do. Because dry run requires a group object and not the group id as a string we are going to put a function to find the group in as the argument to dry run command:

loaderDryRunOneJob(GroupFinder.findByName(session,'test:subfolder:loader_test'), null)

/*
Group: test:subfolder:loader_test add Subject id: elewis86, sourceId: ldap
Group: test:subfolder:loader_test add Subject id: lmartinez31, sourceId: ldap
Group: test:subfolder:loader_test add Subject id: esmith814, sourceId: ldap
Group: test:subfolder:loader_test add Subject id: mvales228, sourceId: ldap
...
===> loader dry ran successfully, would have inserted 285 memberships, would have deleted 0 memberships, total membership count: 285, unresolvable subjects: 0
*/

Note that variable called ‘session’ that we set back in the first exercise is needed for the GroupFinder functions. It seems that what this loader job will do is fine, so let’s run the job:

loaderRunOneJob(GroupFinder.findByName(session,'test:subfolder:loader_test'))

/*
===> loader ran successfully, inserted 285 memberships, deleted 0 memberships, total membership count: 285, unresolvable subjects: 0
*/

Verify in the UI that the membership count matches what GSH reported. Also check out under More->Loader that the settings are the same as if you were to configure a loader job in the UI

(Extra) Exercise 301.4.7 - Burn it Down

Sometimes there are folders in Grouper that you simply want to destroy everything under including all history that it ever existed. A good example of this are course enrollment groups for a particular semester that are likely a lot of data and that changed often during the semester. After a certain amount of time, these groups will be meaningless and can be destroyed (including all history) in order to free up some storage. We do not have any course enrollments in this exercise, so let’s destroy all of our hard work from Exercise 301 instead.  

If you do not have GSH open, fire it up again and run the following:

obliterateStem("test", true, true);

/*
Would obliterate stem: test
Would obliterate stem: test:subfolder
Would be done deleting group: test:subfolder:loader_test
Would be done deleting group: test:subfolder:test_group
Would be done obliterating stem: test:subfolder
Would be done obliterating stem: test
===> true
*

The first argument specifies what stem we want to destroy. The second argument is set to true as a test only mode. The third argument specifies whether you want this also destroyed from all point in time data too. We see above that it would destroy all of our work today as expected. Let’s do it:

obliterateStem("test", false, true);
/*
Obliterating stem: test
Obliterating stem: test:subfolder
Done deleting group: test:subfolder:loader_test
Done deleting group: test:subfolder:test_group
Done obliterating stem: test:subfolder
Done obliterating stem: test
Waiting for Grouper Daemon to process before obliterating from point in time data. This is expected to take a few minutes. Be sure the Grouper Daemon is running.
Obliterating stem from point in time: test, ID=b8b1cbecf1bb4919822d0dd412c86c4a
Done deleting group from point in time: test:test, ID=fbdb9881c28e43f8b0d1756481d6651f
Obliterating stem from point in time: test:subfolder, ID=ac9a6978f1a94849a3760eb1bd606824
Done deleting group from point in time: test:subfolder:loader_test, ID=4597c45d2d9f454ca56793f6f6d56ddc
Done deleting group from point in time: test:subfolder:test_group, ID=9ca911618ff74ad2ac0d954fff8fa3c9
Done obliterating stem from point in time: test:subfolder, ID=ac9a6978f1a94849a3760eb1bd606824
Done obliterating stem from point in time: test, ID=b8b1cbecf1bb4919822d0dd412c86c4a
===> true*/

This may take a while if you have a very large set of folders/groups to delete. This can also be very dangerous since, as you can see, there was no confirmation asked when we ran the command above before it destroyed everything. We can look in the Grouper UI to verify that our work is no longer there:

Conclusion

You have now used GSH to manipulate folders, groups, and members in Grouper and even create a report that could be automated by running script on a regular basis through cron. Finally, we used GSH to efficiently destroy everything from a given folder. These were all basic examples, but we hope that you can begin to see the potential power for GSH commands in creating things like templates for new services, bulk adding/removing members, writing a script a script to bootstrap your development environment, etc. All of these training exercises are built out using GSH scripts. 

Here is an example from the training environment that bootstraps the environment by building out a complete tree of folders and groups, adds members, grants privileges, creates a loader job, add attributes, creates a composite group and assigns Grouper rules. 

https://github.internet2.edu/docker/grouper_training/blob/master/full-demo/container_files/demo.gsh