Grouper configuration in the database and UI

In a patch to Grouper 2.4, Grouper will allow configuration which is normally in config files to be stored in the database.  Configuration in the database will make the configuration consistent in an environment, otherwise the config files need to be kept in sync for the UI servers, WS servers, daemons, and GSH.   Also, editing the configuration in the UI will prevent configuration  problems and help users to configure Grouper easier and more reliably.

https://bugs.internet2.edu/jira/browse/GRP-1953

How it works

In the hierarchy section of the config file will be an option to use the database

e.g. in grouper.base.properties

OLD

grouper.config.hierarchy = classpath:grouper.base.properties, classpath:grouper.properties

NEW

grouper.config.hierarchy = classpath:grouper.base.properties, classpath:grouper.properties, database:grouper

This "database:grouper" will check the grouper database table: grouper_config for overlays.  This should be last so that when you edit something in the UI, it will take effect.

Note: if you have configs you want to differ in different components (e.g. ws/ui/daemon), you need to set those in the config files, and not in the database, which will span all components (e.g. UI/WS/Daemon) that use that database

Config files

1. grouper.properties
2. grouper-loader.properties
3. subject.properties
4. grouper.client.properties (if there is a grouper.hibernate.properties in the classpath)
5. grouper.cache.properties
6. grouper.hibernate.properties (note, the database connection information needs to be in the config file of course, since that database holds the configuration and it needs to be able to connect)

7. grouper-ui.properties

8. grouper-ws.properties

Edit the configuration in the UI

There will be an editor in the UI

• Grouper admins will be able to view/edit the UI

• The source IP address will need to configured or disabled
  
  

  # The configuration editor in grouper is very sensitive.  If you can only allow certain source IP addresses 
  # it will add another layer of security.  Otherwise allow 0.0.0.0/0 and all will be allowed
  # If this configuration item is not filled in, then none are allowed
  # you can configure multiple CIDR addresses or networks comma separated, e.g. 1.2.3.4/32, 2.3.4.5/24
  grouperUi.configurationEditor.sourceIpAddresses = 

• The UI will show where the config comes from in the hierarchy of config (e.g. could be base, override, database, etc)

We will use this facility to create wizards for various things like source config, pspng config, etc.

The UI will use comments in the config file to describe the configuration, and will also use configuration metadata (described below)

Import/export the config

You can import/export the configuration in the database.  You cannot import a "base" config file.

You can import from a config file.  This will only be a config file in the classpath or on the filesystem.  This is how to get started with this feature.  Note, you will need to remove the config from other config files as well, e.g. web service configs.

If we need to be able to export a properties file we can do that too.

You can export a consolidated XML file which has all config from the database.  This will export on the file system and will not be downloadable from the UI

You can import from a consolidated export from the filesystem.

Configuration metadata

There can be JSON configuration metadata in the config files to help the UI correctly display and validate the configs.  These metadata are in the "base" config files only.  This is an example:

#######################################
## inititalization and configuration settings
#######################################

#if grouper should auto init the registry if not initted (i.e. insert the root stem, built in fields, etc)
#defaults to true
# {valueType: "boolean", sensitive: false}
registry.autoinit = true

#auto-create groups (increment the integer index), and auto-populate with users 
#(comma separated subject ids) to bootstrap the registry on startup
#(note: check config needs to be on)
# {regex: "configuration.autocreate.group.name.[0-9]+", valueType: "group"}
#configuration.autocreate.group.name.0 = $$grouper.rootStemForBuiltinObjects$$:uiUsers
# {regex: "configuration.autocreate.group.description.[0-9]+", valueType: "string"}
#configuration.autocreate.group.description.0 = users allowed to log in to the UI
# {regex: "configuration.autocreate.group.subjects.[0-9]+", valueType: "subject", multiple: true}
#configuration.autocreate.group.subjects.0 = johnsmith

Note: just because there is validation doesnt mean a script cant be used.... e.g.

# group who can assign id index cols (also, wheel or root is allowed)
# {valueType: group}
grouper.tableIndex.groupWhoCanAssignIdIndex = $$grouper.rootStemForBuiltinObjects$$:canAssignIdIndex

Configuration file layout

1. Config file starts with license, which has comments, none of which start with 10 hashes
   
   

   #
   # Copyright 2014 Internet2
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   #
   
   # Grouper loader uses Grouper Configuration Overlays (documented on wiki)
   # By default the configuration is read from grouper-loader.base.properties
   # (which should not be edited), and the grouper-loader.properties overlays
   # the base settings.  See the grouper-loader.base.properties for the possible
   # settings that can be applied to the grouper.properties
   
   
   
   

2. Section header starts with a comment with at least 10 hashes, has documentation that starts with two hashes, and ends with a line with at least 10 hashes
   
   

   ########################################
   ## Config chaining hierarchy
   ## configure how configs are read and in what order
   ########################################
   
   

3. Blank line separates items and sections

4. Config file comment and item has documentation.  Comments start with a single hash.  Comments can contain json metadata.  Metadata can span several comment lines.  Item keys and values are separated by an equals and optional whitespace.  Metadata must follow comments, not the other way around.  After the comment or metadata is the property or whitespace
   
   

   # seconds between checking to see if the config files are updated
   # {valueType: "integer", required: true}
   grouper.config.secondsBetweenUpdateChecks = 60

5. You could comment out one sample value after the metadata which will be used an a sample value
   
   

   #auto-create groups (increment the integer index), and auto-populate with users 
   #(comma separated subject ids) to bootstrap the registry on startup
   #(note: check config needs to be on)
   # {regex: "configuration.autocreate.group.name.[0-9]+", valueType: "group", required: true}
   #configuration.autocreate.group.name.0 = $$grouper.rootStemForBuiltinObjects$$:uiUsers

6. df