Exercise 301.4.1 - Getting Started
Start up your Grouper docker stack. You will want the UI running (log in as banderson) so you can see visually how the shell commands impact objects in Grouper.
GSH can initiated by running:
docker exec -it -u tomcat 301.4.1 /opt/grouper/grouperWebapp/WEB-INF/bin/gsh.sh
On start, you’ll see some helpful information about your grouper environment that will look something like this:
Using GROUPER_HOME: /opt/grouper/grouper.apiBinary
Using GROUPER_CONF: /opt/grouper/grouper.apiBinary/conf
Using JAVA: /usr/lib/jvm/zulu-8//bin/java
using MEMORY: 64m-750m
Grouper starting up: version: 2.4.0, build date: 2018/08/23 07:48:38, env: <no label configured>
grouperPatchStatus read from: /opt/grouper/grouper.apiBinary/grouperPatchStatus.properties
api patches installed: 0, 1, 2
pspng patches installed: 0
grouper.properties read from: /opt/grouper/grouper.apiBinary/conf/grouper.properties
Grouper current directory is: /opt/grouper/grouper.apiBinary
log4j.properties read from: /opt/grouper/grouper.apiBinary/conf/log4j.properties
Grouper is logging to file: /tmp/logpipe, at min level WARN for package: edu.internet2.middleware.grouper, based on log4j.properties
grouper.hibernate.properties: /opt/grouper/grouper.apiBinary/conf/grouper.hibernate.properties
grouper.hibernate.properties: root@jdbc:mysql://localhost:3306/grouper?CharSet=utf8&useUnicode=true&characterEncoding=utf8
subject.properties read from: /opt/grouper/grouper.apiBinary/conf/subject.properties
sources configured in: subject.properties
subject.properties ldap source id: ldap: demo
subject.properties groupersource id: g:gsa
subject.properties groupersource id: grouperEntities
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/grouper/grouper.apiBinary/lib/grouper/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/grouper/grouper.apiBinary/lib/custom/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Type help() for instructions
Groovy Shell (2.5.0-beta-2, JVM: 1.8.0_181)
Type ':help' or ':h' for help.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> :load '/opt/grouper/grouper.apiBinary/conf/groovysh.profile'
groovy:000>
Note that the current patch level is indicated, information on what the connection string to the Grouper database is, where various configuration files are read from, and available subject sources. If your subject source is based on a database instead of ldap, you will see information on the connection string to that database too:
subject.propertiees jdbc source id: jdbc: GROUPER_USR@jdbc:oracle:thin:@pdr.db.example.edu:1564:PersonDataRegistryDB
A friendly GroovyShell prompt greets us at the end (groovy:000>) and we are ready to start interacting with GSH. We’ll start by looking up information about the subject banderson.
groovy:000> subj = findSubject("banderson");
===> Subject id: banderson, sourceId: ldap, name: Bob Anderson
Note that you do not need to declare your variables in GSH ahead of time and that GSH will output information about the last run command. This is helpful in validating that the banderson we found is in fact Bob Anderson, but let’s make sure that this is really Bob Anderson, the Grouper administrator, and not some other imposter Bob Anderson.
groovy:000> getMembers("etc:sysadmin");
This time, we did not specify a variable at the end because we did not care to use the result for anything else. When we run that command though, we see an error:
groovy:000> getMembers("etc:sysadmin");
ERROR edu.internet2.middleware.grouper.exception.GroupNotFoundException:
Cannot find group with name: 'etc:sysadmin'
at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3GroupDAO.findByName (Hib3GroupDAO.java:1130)
at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3GroupDAO.findByName (Hib3GroupDAO.java:1092)
at edu.internet2.middleware.grouper.GroupFinder.findByNameNoCache (GroupFinder.java:586)
at edu.internet2.middleware.grouper.GroupFinder.findByName (GroupFinder.java:556)
at edu.internet2.middleware.grouper.GroupFinder.findByName (GroupFinder.java:519)
at edu.internet2.middleware.grouper.app.gsh.getMembers.invoke (getMembers.java:79)
at edu.internet2.middleware.grouper.app.gsh.getMembers$invoke.call (Unknown Source)
at groovysh_evaluate.getMembers (groovysh_evaluate:4)
GSH displays a friendly version of the problem (highlighted in red on most consoles) and we can quickly see that we just put in the wrong group name. The additional stack information is helpful if you ever need to report a problem out to the grouper-users email list or to the Grouper developers.
Let’s fix our mistake. Use the up arrow to go to your previous command to fix it up without having to retype the entire thing so it matches the command below:
groovy:000> getMembers("etc:sysadmingroup");
===> ['banderson'/'person'/'ldap']
There we go. We see the members of the Grouper sys admins group and verify that banderson is in fact the entity we want.
In this next exercise, we will use that subj we found to create a few folders, a group, and add some members.
Exercise 301.4.2 - Creating a Folder/Group Structure
Before we get started manipulating anything in Grouper, we will need to start a session. There are two ways to accomplish this, the first is starting a root session:
groovy:000> session = GrouperSession.startRootSession();
===> 7c847cea3822464dad3ccd12a22e3d2d,'GrouperSystem','application'
Or you can start a session as a specific subject and act as that subject:
groovy:000> session = GrouperSession.start(subj);
===> 08d1776e32614d718dfa534d56b45473,'banderson','person'
Note that we used subj from the above exercise where we looked up and found banderson. Let’s create a folder called test:
groovy:000> addRootStem("test","Test Folder at Root");
===> Stem[displayName=Test Folder at Root,name=test,uuid=4c8b68bbf3cf46fca37e5bf715f1a143,creator=ff8969d26d084674a1420a0d8c3f4a5e]
The arguments for that command are the folder id and then the friendly name of the folder. Now let’s create a folder within test:
groovy:000> addStem("test","subfolder","Test Subfolder");
===> Stem[displayName=Test Folder at Root:Test Subfolder,name=test:subfolder,uuid=9d2043e2cab1410f80c63bc68dd971df,creator=ff8969d26d084674a1420a0d8c3f4a5e]
We specified the folder to make this folder in, the id of the folder, and the friendly name of the folder. If you make a mistake, whether it is a missing argument, or that it cannot find something you referenced in an argument, GSH will help you out:
groovy:000> addStem("test","subfolder2");
ERROR groovy.lang.MissingMethodException:
No signature of method: groovysh_evaluate.addStem() is applicable for argument types: (String, String) values: [test, subfolder2]
Possible solutions: addStem(java.lang.String, java.lang.String, java.lang.String)
groovy:000> addStem("test2","subfolder2","fail subfolder");
ERROR edu.internet2.middleware.grouper.exception.StemNotFoundException:
Can't find stem by name: 'test2'
at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.findByName (Hib3StemDAO.java:2004)
at edu.internet2.middleware.grouper.StemFinder.findByName (StemFinder.java:230)
at edu.internet2.middleware.grouper.StemFinder.findByName (StemFinder.java:200)
at edu.internet2.middleware.grouper.app.gsh.StemHelper.addStem (StemHelper.java:73)
at edu.internet2.middleware.grouper.app.gsh.addStem.invoke (addStem.java:71)
at edu.internet2.middleware.grouper.app.gsh.addStem$invoke.call (Unknown Source)
at groovysh_evaluate.addStem (groovysh_evaluate:4)
IMPORTANT: You are often acting as a super user in GSH, so do be careful with any commands to ensure you do not accidentally destroy any data in your Grouper environment. GSH will not prompt you if it is OK to delete something once you hit enter on the command.
Take a look in the UI (you will want to login as banderson for these exercises). We will see our new folder and subfolder created:
Note that the creator of the folder is Bob Anderson, because we started the Grouper session as banderson. If you started a root session, the creator would instead be GrouperSysAdmin.
Click on More Actions -> View audit log.
Note that the audit log shows that this folder was created through a GSH command. This can be very valuable in determining whether it was Bob Anderson logged in through the UI creating folders or someone acting as Bob Anderson via GSH that created the folder instead. We’ll look at an example audit log with both when we create a group next.
Back in your GSH window, let’s now create a group:
groovy:000> addGroup("test:subfolder","test_group","Testing Group");
===> Group[name=test:subfolder:test_group,uuid=52af781176e54f56a72a5e32f5caf95c]
...and add a member to it:
groovy:000> addMember("test:subfolder:test_group","jsmith");
===> true
GSH prints out true because it successfully added the member ‘jsmith’ to ‘test:subfolder:test_group’. Here are a couple cases of what happens when it fails to add the membership:
groovy:000> addMember("test:subfolder:test_group","jsmith");
ERROR edu.internet2.middleware.grouper.exception.MemberAddAlreadyExistsException:
membership already exists,
Problem in HibernateSession: HibernateSession (703e5614): notNew, notReadonly, READ_WRITE_NEW, activeTransaction, session (261609a7), membership: group: test:subfolder:test_group, subject: jsmith, field: members, uuid: null, startDate: null, endDate: null,
, group name: test:subfolder:test_group, subject: Subject id: jsmith, sourceId: ldap, field: members,
Problem in HibernateSession: HibernateSession (1d3e5a05): new, notReadonly, READ_WRITE_NEW, notActiveTransaction, session (261609a7)
at edu.internet2.middleware.grouper.Membership.internal_addImmediateMembership (Membership.java:1264)
at edu.internet2.middleware.grouper.Group$4.callback (Group.java:1394)
at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession (HibernateSession.java:701)
at edu.internet2.middleware.grouper.Group.internal_addMember (Group.java:1366)
at edu.internet2.middleware.grouper.Group.addMember (Group.java:940)
at edu.internet2.middleware.grouper.Group.addMember (Group.java:901)
at edu.internet2.middleware.grouper.app.gsh.addMember.invoke (addMember.java:145)
at edu.internet2.middleware.grouper.app.gsh.addMember$invoke.call (Unknown Source)
at groovysh_evaluate.addMember (groovysh_evaluate:4)
at groovysh_evaluate.addMember (groovysh_evaluate:3)
groovy:000> addMember("test:subfolder:test_group","jsmith2");
ERROR edu.internet2.middleware.subject.SubjectNotFoundException:
subject not found: jsmith2
at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.thereCanOnlyBeOne (SourcesXmlResolver.java:486)
at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier (SourcesXmlResolver.java:527)
at edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier (CachingResolver.java:406)
at edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier (ValidatingResolver.java:203)
at edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier (SubjectFinder.java:316)
at edu.internet2.middleware.grouper.app.gsh.addMember.invoke (addMember.java:144)
at edu.internet2.middleware.grouper.app.gsh.addMember$invoke.call (Unknown Source)
at groovysh_evaluate.addMember (groovysh_evaluate:4)
at groovysh_evaluate.addMember (groovysh_evaluate:3)
In the first failure, we attempted to add jsmith to the group we had already added him to. GSH helpfully tells us that the membership already exists. In the second failure, we tried to add a subject that does not exist. Again, GSH points us in the direction of where we went wrong.
(As a reminder, the arguments for all of these commands are available on the GSH wiki page)
Let’s take a look in the UI at our new group we just created:
While you are in the UI, go ahead and also add ‘banderson’ to the group too as a member. Once you have done that, take a look at the audit log for the group:
Note how even though both group member adds were done as Bob Anderson, we can see that the second add was down through the UI while the first was done as a GSH command.
In the next exercise, we will create a group again, but this time use the Grouper Loader to populate the group instead of manually adding members.
Exercise 301.4.3 - Creating and Running a Grouper Loader Job
For this exercise, we are going to create a very simple loader job that loads all subjects from the grouper members database table and run it. While the UI does provide a very easy to use mechanism to create and run loader jobs, there are times where you may find it convenient to use GSH instead. GSH can be helpful if there are a bunch of loader groups you need to create and do not want to click through the UI for each one. Much of how to create Grouper Loader jobs in GSH is documented on the Grouper Loader wiki page.
First, create the group:
groovy:000> addGroup("test:subfolder","loader_test","Test Loader Group");
===> Group[name=test:subfolder:loader_test,uuid=de101843359046ff987d1ff167e8301c]
Now assign it the attributes needed to make it a loader job. This time, instead of writing one line at a time, try copying the entire block below in to GSH:
groupAddType("test:subfolder:loader_test", "grouperLoader");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderDbName", "grouper");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderType", "SQL_SIMPLE");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderScheduleType", "CRON");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuartzCron", "0 * * * * ?");
setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuery", "select distinct subject_id as SUBJECT_ID, subject_source SUBJECT_SOURCE_ID from grouper.grouper_members where subject_source = 'ldap'");
GSH handled the line feeds between commands and you should see each command run followed by ‘true’ printed after each line. In the next exercise we will show you how to run a block of commands as a script, which is the preferred way over copy/paste many lines at a time.
You should see output like this:
groovy:000> groupAddType("test:subfolder:loader_test", "grouperLoader");
===> true
groovy:000> setGroupAttr("test:subfolder:loader_test", "grouperLoaderDbName", "grouper");
===> true
groovy:000> setGroupAttr("test:subfolder:loader_test", "grouperLoaderType", "SQL_SIMPLE");
===> true
groovy:000> setGroupAttr("test:subfolder:loader_test", "grouperLoaderScheduleType", "CRON");
===> true
groovy:000> setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuartzCron", "0 * * * * ?");
===> true
groovy:000> setGroupAttr("test:subfolder:loader_test", "grouperLoaderQuery", "select distinct subject_id as SUBJECT_ID, subject_source SUBJECT_SOURCE_ID from grouper.grouper_members where subject_source = 'ldap'");
===> true
And now let’s dry run the job to see what it would do. Because dry run requires a group object and not the group id as a string we are going to put a function to find the group in as the argument to dry run command:
groovy:000> loaderDryRunOneJob(GroupFinder.findByName(session,'test:subfolder:loader_test'), null);
Group: test:subfolder:loader_test add Subject id: banderson, sourceId: ldap
Group: test:subfolder:loader_test add Subject id: jsmith, sourceId: ldap
===> loader dry ran successfully, would have inserted 2 memberships, would have deleted 0 memberships, total membership count: 2, unresolvable subjects: 0
Note that variable called ‘session’ that we set back in the first exercise is needed for the GroupFinder functions. It seems that what this loader job will do is fine, so let’s run the job:
groovy:000> loaderRunOneJob(GroupFinder.findByName(session,'test:subfolder:loader_test'));
===> loader ran successfully, inserted 2 memberships, deleted 0 memberships, total membership count: 2, unresolvable subjects: 0
...and let’s go look at it in the UI. While you are in the UI, check out the memberships to see that it did add the two members it was going to add and also check out under More->Loader that the settings are the same as if you were to configure a loader job in the UI
In the next exercise, we are going create an automatically generated report of all memberships for something like a daily report to a security officer around campus who is interested in who has access to what for auditing purposes. GSH can help us make that pretty simple without having to write any sort of database query.
Exercise 301.4.4 - Build a Report (making a GSH script)
For this exercise, we will be generating that report using GSH, but also looking at how to run a script in GSH without having to copy and paste all of the lines of the script.
Open your favorite text editor and copy in the following script:
session = GrouperSession.startRootSession();
group = GroupFinder.findByName(session, "test:subfolder:loader_test", true);
effectiveMembers = group.getEffectiveMembers();
immediateMembers = group.getImmediateMembers();
PrintWriter writer = new PrintWriter("/tmp/out", "UTF-8");
writer.println(String.join("\t", "id", "name", "Effective", "Immediate"));
for (Member m: group.getMembers()) {
writer.print(m.getSubject().getId() + "\t" + m.getSubject().getName() + "\t");
writer.print(effectiveMembers.contains(m).toString() + "\t");
writer.println(immediateMembers.contains(m).toString() + "\t");
}
writer.close();
Save your script (in this case called report.gsh) and here are two ways we can run the gsh script:
Copy the script in to the container and then run it:
$ docker cp report.gsh 301.4.1:/tmp/report.gsh
Open a shell into the container and switch to the tomcat user:
$ ./gte-shell
$ sudo -u tomcat /bin/bash
Run the report
$ bin/gsh.sh /tmp/report.gsh"
Note that data in containers is not persistent and it would be better to mount in a directory where you keep your GSH scripts.
The output shows very little other than the output for each command run:
groovy:000> :load '/opt/grouper/grouper.apiBinary/conf/groovysh.profile'
groovy:000> :gshFileLoad '/tmp/report.gsh'
===> dcf994127b6147afa86d6f3569d131a4,'GrouperSystem','application'
===> Group[name=test:subfolder:loader_test,uuid=de101843359046ff987d1ff167e8301c]
===> []
===> ['jsmith'/'person'/'ldap', 'banderson'/'person'/'ldap']
===> java.io.PrintWriter@7e0883f3
===> null
===> null
===> null
groovy:000> :exit
We can see our output report by running the following:
$ docker exec -i 304.4.1 cat /tmp/out
id name Effective Immediate
jsmith John Smith false true
banderson Bob Anderson false true
...and we see our beautiful tab separated report we can now ship off to whoever requested it.
You could execute your recurring GSH scripts either by calling the docker commands from cron, or perhaps creating a container based on tier-grouper that runs crond instead of any of the grouper components. Your container would be built with the crontab and you would be able to schedule any functionality you need with grouper components like GSH. But that is an exercise left to the reader (for now).
Exercise 301.4.5 - Burn it Down
Sometimes there are folders in Grouper that you simply want to destroy everything under including all history that it ever existed. A good example of this are course enrollment groups for a particular semester that are likely a lot of data and that changed often during the semester. After a certain amount of time, these groups will be meaningless and can be destroyed (including all history) in order to free up some storage. We do not have any course enrollments in this exercise, so let’s destroy all of our hard work from Exercise 301 instead.
If you do not have GSH open, fire it up again and run the following:
groovy:000> obliterateStem("test", true, true);
Would obliterate stem: test
Would obliterate stem: test:subfolder
Would be done deleting group: test:subfolder:loader_test
Would be done deleting group: test:subfolder:test_group
Would be done obliterating stem: test:subfolder
Would be done obliterating stem: test
===> true
The first argument specifies what stem we want to destroy. The second argument is set to true as a test only mode. The third argument specifies whether you want this also destroyed from all point in time data too. We see above that it would destroy all of our work today as expected. Let’s do it:
groovy:000> obliterateStem("test", false, true);
Obliterating stem: test
Obliterating stem: test:subfolder
Done deleting group: test:subfolder:loader_test
Done deleting group: test:subfolder:test_group
Done obliterating stem: test:subfolder
Done obliterating stem: test
Waiting for Grouper Daemon to process before obliterating from point in time data. This is expected to take a few minutes. Be sure the Grouper Daemon is running.
Obliterating stem from point in time: test, ID=b8b1cbecf1bb4919822d0dd412c86c4a
Done deleting group from point in time: test:test, ID=fbdb9881c28e43f8b0d1756481d6651f
Obliterating stem from point in time: test:subfolder, ID=ac9a6978f1a94849a3760eb1bd606824
Done deleting group from point in time: test:subfolder:loader_test, ID=4597c45d2d9f454ca56793f6f6d56ddc
Done deleting group from point in time: test:subfolder:test_group, ID=9ca911618ff74ad2ac0d954fff8fa3c9
Done obliterating stem from point in time: test:subfolder, ID=ac9a6978f1a94849a3760eb1bd606824
Done obliterating stem from point in time: test, ID=b8b1cbecf1bb4919822d0dd412c86c4a
===> true
This may take a while if you have a very large set of folders/groups to delete. This can also be very dangerous since, as you can see, there was no confirmation asked when we ran the command above before it destroyed everything. We can look in the Grouper UI to verify that our work is no longer there:
Conclusion
You have now used GSH to manipulate folders, groups, and members in Grouper and even create a report that could be automated by running script on a regular basis through cron. Finally, we used GSH to efficiently destroy everything from a given folder. These were all basic examples, but we hope that you can begin to see the potential power for GSH commands in creating things like templates for new services, bulk adding/removing members, writing a script a script to bootstrap your development environment, etc. All of these training exercises are built out using GSH scripts.
Here is an example from the training environment that bootstraps the environment by building out a complete tree of folders and groups, adds members, grants privileges, creates a loader job, add attributes, creates a composite group and assigns Grouper rules.
https://github.internet2.edu/docker/grouper_training/blob/master/full-demo/container_files/demo.gsh