View Source

In grouper 2.3.0 (unpatched) and previous versions, the subject API was configured with a sources.xml .

Configuration

Sample sources.xml config

<?xml version="1.0" encoding="utf-8"?>
<sources>
 <source adapterClass="edu.internet2.middleware.grouper.subj.GrouperJdbcSourceAdapter">
    <id>jdbc</id>
    <name>Example JDBC Source Adapter</name>
     <type>person</type>
     
     <init-param>
       <param-name>sortAttribute1</param-name>
       <param-value>LOGINID</param-value>
     </init-param>
     <init-param>
       <param-name>searchAttribute0</param-name>
       <param-value>searchAttribute0</param-value>
     </init-param>
     <internal-attribute>searchAttribute0</internal-attribute>

     <search>
         <searchType>searchSubject</searchType>
         <param>
             <param-name>sql</param-name>
             <param-value>
select
   s.subjectid as id, s.name as name,
   (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,
   (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,
   (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,
   (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email
from
   subject s
where
   {inclause}
            </param-value>
         </param>
         <param>
             <param-name>inclause</param-name>
             <param-value>
s.subjectid = ?
            </param-value>
         </param>
     </search>
   </source>
</sources>

This needs to be converted to config overlays so Grouper packaging can have defaults and overlays. Also so the Grouper team can internal sources without having to merge with a potentially changed file.

Sample subject.properties config:

#########################################
## Configuration for source id: jdbc
## Source configName: jdbc
#########################################
subjectApi.source.jdbc.id = jdbc

# this is a friendly name for the source
subjectApi.source.jdbc.name = Example JDBC Source Adapter

# type is not used all that much.  Can have multiple types, comma separate.  Can be person, group, application
subjectApi.source.jdbc.types = person

# the adapter class implements the interface: edu.internet2.middleware.subject.Source
# adapter class must extend: edu.internet2.middleware.subject.provider.BaseSourceAdapter
# edu.internet2.middleware.grouper.subj.GrouperJdbcSourceAdapter2  :  if doing JDBC this should be used if possible.  All subject data in one table/view.
# edu.internet2.middleware.grouper.subj.GrouperJdbcSourceAdapter   :  oldest JDBC source.  Put freeform queries in here
# edu.internet2.middleware.grouper.subj.GrouperJndiSourceAdapter   :  used for LDAP
subjectApi.source.jdbc.adapterClass = edu.internet2.middleware.grouper.subj.GrouperJdbcSourceAdapter

subjectApi.source.jdbc.param.jdbcConnectionProvider.value = edu.internet2.middleware.grouper.subj.GrouperJdbcConnectionProvider

subjectApi.source.jdbc.param.identifierAttributes.value = LOGINID

# subject identifier to store in grouper's member table.  this is used to increase speed of loader and perhaps for provisioning
# you can have up to max 1 subject identifier
subjectApi.source.jdbc.param.subjectIdentifierAttribute0.value = LOGINID

#searchSubject: find a subject by ID.  ID is generally an opaque and permanent identifier, e.g. 12345678.
#  Each subject has one and only on ID.  Returns one result when searching for one ID.

# sql is the sql to search for the subject by id should use an {inclause}
subjectApi.source.jdbc.search.searchSubject.param.sql.value = select    s.subjectid as id, s.name as name,    (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,    (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,    (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,    (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email from    subject s where    {inclause}

# inclause allows searching by subject for multiple ids or identifiers in one query, must have {inclause} in the sql query,
#    this will be subsituted to in clause with the following.  Should use a question mark ? for bind variable
subjectApi.source.jdbc.search.searchSubject.param.inclause.value = s.subjectid = ?


 
# internal attributes are used by grouper only not exposed to code that uses subjects.  comma separated
subjectApi.source.jdbc.internalAttributes = searchAttribute0

Note, the part that links the properties entries together is the part between "source" and the following dot. In this case "jdbc". This was automatically converted from the sources.xml file by looking at the source name and making sure there are no special chars (replace with underscore).

Use the xml configuration

If you want to use an xml configuration (generally you wont need to do this) set this in subject.properties (or remove to use the default):

########################################
## Use old sources.xml
########################################

# enter the location of the sources.xml.  Must start with classpath: or file:
# blank means dont use sources.xml, use subject.properties
# default is: classpath:sources.xml
# e.g. file:/dir1/dir2/sources.xml
subject.sources.xml.location = classpath:sources.xml

Upgrade via Patch

Get the latest grouperInstaller.jar
Backup your sources.xml file
Install the 2.3.0#40 API patch
If you have ever edited the subject.properties file (unlikely), move that file to another location and merge in changes later if applicable
Run the grouperInstaller, admin task, upgradeTasks, convert sources xml to properties
Follow the prompt to identify the location of the sources.xml file
This will generate the subject.properties file based on your sources.xml
Examine the subject.properties file. Eyeball it to see that your settings from sources.xml are there
Diff your sources.xml file with your sources.example.xml file and make sure you have never changed the settings for Grouper internal sources (not common), e.g. the g:gsa source, or g:isa, or grouperEntities. If you have, configure those properties in the subject.properties overlay. Maybe alert the Grouper team as well to see if it should be a new default for all.
Delete the sources.xml and sources.example.xml files (again, keep a backup somewhere for a while)
Install the patch in your other envs (UI, WS, daemon, etc), and copy the subject.properties to each env (no need to go through this same process unless you have different cache customizations in each env). And delete the sources.xml and sources.example in each env

Upgrade via upgrade

If you upgrade to 2.3.1+ it will walk you through the process of automatically converting your sources.xml to subject.properties

Example of using the latest grouperInstaller.jar

[appadmin@i2midev1 patchesAuto]$ rm grouperInstaller.jar
[appadmin@i2midev1 patchesAuto]$ wget --no-check-certificate https://software.internet2.edu/grouper/release/2.3.0/grouperInstaller.jar

Backup your sources.xml and sources.example.xml file

[appadmin@i2midev1 patchesAuto]$ cp /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/sources.xml /tmp
[appadmin@i2midev1 patchesAuto]$ cp /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/sources.example.xml /tmp

Install the 2.3.0#40 API patch (note: Force install that patch since it will say file mismatch)

Patch grouper_v2_3_0_api_patch_35 is low risk, is not a security patch
GRP-1417: migrate from grouper.ehcache.xml to hierarchical properties configuration
 - set property: grouper_v2_3_0_api_patch_35.date from: 2016/11/27 19:05:16 to: 2016/11/27 19:12:08
This patch requires all processes that user Grouper to be stopped.
  Please stop these processes if they are running and press <enter> to continue...

Problem applying patch since this patch old file:
  /opt/grouper/2.3/patchesAuto/tarballs/patches/grouper_v2_3_0_api_patch_35/old/classes/ehcache.xml
  is not the same as what the patch expects:
  /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.xml
  Do you want to force install this patch (t|f)? [f]: 
t
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.base.properties
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/misc/GrouperCheckConfig.class
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/misc/GrouperCheckConfig.java
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/cfg/GrouperCacheConfig.class
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/cfg/GrouperCacheConfig.java
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/cache/EhcacheController.java
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/edu/internet2/middleware/grouper/cache/EhcacheController.class
Applying file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.properties
Deleting file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.example.xml
Deleting file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.xml
Patch successfully applied: grouper_v2_3_0_api_patch_35
 - set property: grouper_v2_3_0_api_patch_35.state from: error to: applied

Run the grouperInstaller, admin task, upgradeTasks, convert sources xml to properties

Follow the prompt to identify the location of the sources.xml file

This will generate the subject.properties file based on your sources.xml edits

[appadmin@i2midev1 patchesAuto]$ java -jar grouperInstaller.jar
Do you want to 'install' a new installation of grouper, 'upgrade' an existing installation,
  'patch' an existing installation, 'admin' utilities, or 'createPatch' for Grouper developers
  (enter: 'install', 'upgrade', 'patch', 'admin', 'createPatch' or blank for the default) [install]: admin
What admin action do you want to do (manage, upgradeTask)? : upgradeTask
What upgrade task do you want to do (convertEhcacheXmlToProperties)? : convertEhcacheXmlToProperties
Note, you need to convert the ehcache.xml file for each Grouper runtime, e.g. loader, WS, UI.
Note, you need to be running Grouper 2.3.0 with API patch 35 installed.
Enter the location of the ehcache.xml file: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.xml
Enter the location of the grouper.cache.base.properties file [/opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.base.properties]: 
Enter the location of the grouper.cache.properties file (to be created)  [/opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.properties]: 
File was written: /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.properties

Examine the subject.properties file. Diff your sources.xml file with your sources.example.xml file and make sure those diffs are expected and listed in your subject.properties file

[appadmin@i2midev1 patchesAuto]$ more /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/grouper.cache.properties
# Copyright 2016 Internet2
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#
# Grouper Cache Configuration
#

# The grouper cache config uses Grouper Configuration Overlays (documented on wiki)
# By default the configuration is read from grouper.cache.base.properties
# (which should not be edited), and the grouper.cache.properties overlays
# the base settings.  See the grouper.cache.base.properties for the possible
# settings that can be applied to the grouper.cache.properties

cache.name.internal_dao_hib3_Hib3MemberDAO_FindBySubject.maxElementsInMemory = 10000

cache.name.internal_dao_hib3_Hib3MembershipDAO.maxElementsInMemory = 20000
cache.name.internal_dao_hib3_Hib3MembershipDAO.timeToIdleSeconds = 60
cache.name.internal_dao_hib3_Hib3MembershipDAO.timeToLiveSeconds = 60




[appadmin@i2midev1 patchesAuto]$ diff /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.xml /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.example.xml
90c90
<           maxElementsInMemory="10000"
---
>           maxElementsInMemory="5000"
101c101
<           maxElementsInMemory="20000"
---
>           maxElementsInMemory="10000"
103,104c103,104
<           timeToIdleSeconds="60"
<           timeToLiveSeconds="60"
---
>           timeToIdleSeconds="5"
>           timeToLiveSeconds="5"
[appadmin@i2midev1 patchesAuto]$

Delete the sources.xml and sources.example.xml files

[appadmin@i2midev1 patchesAuto]$ rm /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.xml
[appadmin@i2midev1 patchesAuto]$ rm /opt/tomcats/tomcat_f/webapps/grouper_v2_3/WEB-INF/classes/ehcache.example.xml

Install the patch in your other envs (UI, WS, daemon, etc), and copy the subject.properties to each env (no need to go through this same process unless you have different cache customizations in each env. Delete the sources.xml and sources.example.xml in other envs too

Logging

Grouper startup will show the subject.properties info

TODO