Always available web services are WS that do not have a single point of failure. This includes the loadbalancer, database, and even the data center.
Grouper should be conducive to always available web services.
The plan is to allow readonly database replication to an off site database, and have a Grouper WS application server which runs in readonly mode against that database. Then a protocol needs to be designed and dcoumented that shows how clients should interact with the multiple WS servers. This protocol should be implemented in the Grouper client. Note that there might not be consistency between the read/write and readonly depending on how the database replication is implemented.
If you want highly available web services (though not always available), then you could use a DNS-based loadbalancer that polls the servers to see which ones are available. Though there could be a minute or two of downtime here or there until the loadbalancer catches up.
Note that you could do local active/active loadbalancing but you might have a point of failure on the load balancer, firewall, or data center itself.
WS discovery
If you are going to have multiple WS servers available for clients, and the load balancing managed on the client, then you might want to use a discovery service so the list of servers can be managed centrally. Here is an example config of a WS discovery service that would be HTTPS based:
# # Grouper client discovery configuration # ######################################## ## LDAP discovery connection settings ######################################## # urls of directory, including the base DN (distinguished name) # add more properties and increment the integer (.1, .2, etc) # e.g. ldap://server.school.edu/dc=school,dc=edu # e.g. ldaps://server.school.edu/dc=school,dc=edu grouperClient.discovery.ldap.0.url = # active/active or active/standby # active/active will pick a server randomly, # and will stick with it for as long as the affinity is set # active/standby will always use the first connection # if no errors, then try the second one etc. # if a connection has more errors and has a higher priority, # then it will not be tried again until the # takeConnectionOutOfPoolOnErrorForSeconds timeout # passes grouperClient.discovery.ldap.loadBalancing = active/active # if we are active/active, then the same connection will # be used for a certain number of seconds. If this is -1, then # always keep the same server (unless errors) grouperClient.discovery.ldap.affinitySeconds = 600 # if a connection has more errors than another, it will not be # used until this error timeout passes (unless the other is throwing errors # too) grouperClient.discovery.ldap.lowerConnectionPriorityOnErrorForSeconds = 180 # when a connection is attempted, this is the timeout that it will use before trying # another connection grouperClient.discovery.ldap.timeoutSeconds = 30 # after all connections have been attempted, it will wait for this long # to see if any finish grouperClient.discovery.ldap.extraTimeoutSeconds = 15 ######################################## ## Web service discovery Connection settings ######################################## # urls of web service, should include everything up to the first resource to access # this is for read or write operations # add more properties and increment the integer (.1, .2, etc) # e.g. http://groups.school.edu:8090/grouper-ws/servicesRest # e.g. https://groups.school.edu/grouper-ws/servicesRest grouperClient.discovery.webService.readWrite.0.url = # url of web service, should include everything up to the first resource to access # this is for only read operations # add more properties and increment the integer (.1, .2, etc) # e.g. http://groups.school.edu:8090/grouper-ws/servicesRest # e.g. https://groups.school.edu/grouper-ws/servicesRest grouperClient.discovery.webService.readOnly.0.url = # active/active or active/standby # active/active will pick a server randomly, # and will stick with it for as long as the affinity is set # active/standby will always use the first connection # if no errors, then try the second one etc. # if a connection has more errors and has a higher priority, # then it will not be tried again until the # takeConnectionOutOfPoolOnErrorForSeconds timeout # passes grouperClient.discovery.webService.loadBalancing = active/active # if you want to always try read/write before readOnly (i.e. if you are # worried about if you make a write and read right after each other) grouperClient.discovery.webService.preferReadWrite = true # if we are active/active, then the same connection will # be used for a certain number of seconds. If this is -1, then # always keep the same server (unless errors) grouperClient.discovery.webService.affinitySeconds = 600 # if a connection has more errors than another, it will not be # used until this error timeout passes (unless the other is throwing errors # too) grouperClient.discovery.webService.lowerConnectionPriorityOnErrorForSeconds = 180 # when a connection is attempted, this is the timeout that it will use before trying # another connection grouperClient.discovery.webService.timeoutSeconds = 60 # after all connections have been attempted, it will wait for this long # to see if any finish grouperClient.discovery.webService.extraTimeoutSeconds = 30
Here would be the grouper client configuration to find the discovery configs, and to set defaults or overrides:
######################################## ## Service discovery settings ######################################## # if you are using a discovery service, but a discovery properties # at a URL (preferably SSL with valid certificate) # you should have multiple discovery URLs hosted at independent locations # to add more, increment the integer grouperClient.urlOfDiscovery.0 = grouperClient.urlOfDiscovery.1 = # if you want to cache the discovery properties locally, put a directory here: # this is recommended especially if you are using the grouper client as a command # line application and the process is constantly restarting # note, this will be used for a failsafe cache if all discovery servers are unavailable # dot is the current directory... note, this directory must exist # or it will be created (attempted) grouperClient.cacheDirectoryForDiscoveryProperties = . # this will cache the discovery properties in memory or on disk grouperClient.cacheDiscoveryPropertiesForSeconds = 60 #### ## Below here are default values and override values for the discovery ## properties at your institution. Note: if the override keys are there ## with no value then it will blank out the discovery service value #### # default urls of directory, including the base DN (distinguished name) # add more properties and increment the integer (.1, .2, etc) # e.g. ldap://server.school.edu/dc=school,dc=edu # e.g. ldaps://server.school.edu/dc=school,dc=edu grouperClient.discoveryDefault.ldap.0.url = #grouperClient.discoveryOverride.ldap.0.url = # default active/active or active/standby # active/active will pick a server randomly, # and will stick with it for as long as the affinity is set # active/standby will always use the first connection # if no errors, then try the second one etc. # if a connection has more errors and has a higher priority, # then it will not be tried again until the # takeConnectionOutOfPoolOnErrorForSeconds timeout # passes grouperClient.discoveryDefault.ldap.loadBalancing = active/active #grouperClient.discoveryOverride.ldap.loadBalancing = active/active # if we are active/active, then the same connection will # be used for a certain number of seconds. If this is -1, then # always keep the same server (unless errors) grouperClient.discoveryDefault.ldap.affinitySeconds = 600 #grouperClient.discoveryOverride.ldap.affinitySeconds = 600 # if a connection has more errors than another, it will not be # used until this error timeout passes (unless the other is throwing errors # too) grouperClient.discoveryDefault.ldap.lowerConnectionPriorityOnErrorForSeconds = 180 #grouperClient.discoveryOverride.ldap.lowerConnectionPriorityOnErrorForSeconds = 180 # when a connection is attempted, this is the timeout that it will use before trying # another connection grouperClient.discoveryDefault.ldap.timeoutSeconds = 30 #grouperClient.discoveryOverride.ldap.timeoutSeconds = 30 # after all connections have been attempted, it will wait for this long # to see if any finish grouperClient.discoveryDefault.ldap.extraTimeoutSeconds = 15 #grouperClient.discoveryOverride.ldap.extraTimeoutSeconds = 15 # urls of web service, should include everything up to the first resource to access # this is for read or write operations # add more properties and increment the integer (.1, .2, etc) # e.g. http://groups.school.edu:8090/grouper-ws/servicesRest # e.g. https://groups.school.edu/grouper-ws/servicesRest grouperClient.discoveryDefault.webService.readWrite.0.url = #grouperClient.discoveryOverride.webService.readWrite.0.url = # url of web service, should include everything up to the first resource to access # this is for only read operations # add more properties and increment the integer (.1, .2, etc) # e.g. http://groups.school.edu:8090/grouper-ws/servicesRest # e.g. https://groups.school.edu/grouper-ws/servicesRest grouperClient.discoveryDefault.webService.readOnly.0.url = #grouperClient.discoveryOverride.webService.readOnly.0.url = # active/active or active/standby # active/active will pick a server randomly, # and will stick with it for as long as the affinity is set # active/standby will always use the first connection # if no errors, then try the second one etc. # if a connection has more errors and has a higher priority, # then it will not be tried again until the # takeConnectionOutOfPoolOnErrorForSeconds timeout # passes grouperClient.discoveryDefault.webService.loadBalancing = active/active #grouperClient.discoveryOverride.webService.loadBalancing = active/active # if you want to always try read/write before readOnly (i.e. if you are # worried about if you make a write and read right after each other) grouperClient.discoveryDefault.webService.preferReadWrite = true #grouperClient.discoveryOverride.webService.preferReadWrite = true # if we are active/active, then the same connection will # be used for a certain number of seconds. If this is -1, then # always keep the same server (unless errors) grouperClient.discoveryDefault.webService.affinitySeconds = 600 #grouperClient.discoveryOverride.webService.affinitySeconds = 600 # if a connection has more errors than another, it will not be # used until this error timeout passes (unless the other is throwing errors # too) grouperClient.discoveryDefault.webService.lowerConnectionPriorityOnErrorForSeconds = 180 #grouperClient.discoveryOverride.webService.lowerConnectionPriorityOnErrorForSeconds = 180 # when a connection is attempted, this is the timeout that it will use before trying # another connection grouperClient.discoveryDefault.webService.timeoutSeconds = 60 #grouperClient.discoveryOverride.webService.timeoutSeconds = 60 # after all connections have been attempted, it will wait for this long # to see if any finish grouperClient.discoveryDefault.webService.extraTimeoutSeconds = 30 #grouperClient.discoveryOverride.webService.extraTimeoutSeconds = 30
sdf
WS client failover protocol
Configure a read/write and a readonly URL.
Identify at least a primary and a secondary URL (note that if there are non-readonly queries, then the readwrite should probably be the primary URL). Or the client could rotate servers for all readonly, or a DNS load balancer could do this for one URL.
Try a server (e.g. the primary), and look at the result code. If it is a SUCCESS, or something like INVALID_QUERY, then trust the response. If there is no response, or it is an error, then if appropriate (e.g. if readonly and the secondary is readonly), try another URL (e.g. the secondary). Note there should be a configurable timeout.
LDAP as a back end
Having LDAP as a back end to web services is an interesting idea, but I think it would be difficult to pull off implementing the full WS API against LDAP (when everyone designs their LDAP structures differently). It is possible to have the grouper client have some operations that can go to WS and LDAP (primary and secondary or vise versa)... we should look into this possibility...