Always available web services are WS that do not have a single point of failure. This includes the loadbalancer, database, and even the data center.
Grouper should be conducive to always available web services.
The plan is to allow readonly database replication to an off site database, and have a Grouper WS application server which runs in readonly mode against that database. Then a protocol needs to be designed and dcoumented that shows how clients should interact with the multiple WS servers. This protocol should be implemented in the Grouper client. Note that there might not be consistency between the read/write and readonly depending on how the database replication is implemented.
WS client failover protocol
Configure a read/write and a readonly URL.
Identify at least a primary and a secondary URL (note that if there are non-readonly queries, then the readwrite should probably be the primary URL). Or the client could rotate servers for all readonly, or a DNS load balancer could do this for one URL.
Try a server (e.g. the primary), and look at the result code. If it is a SUCCESS, or something like INVALID_QUERY, then trust the response. If there is no response, or it is an error, then if appropriate (e.g. if readonly and the secondary is readonly), try another URL (e.g. the secondary). Note there should be a configurable timeout.
LDAP as a back end
Having LDAP as a back end to web services is an interesting idea, but I think it would be difficult to pull off implementing the full WS API against LDAP (when everyone designs their LDAP structures differently). It is possible to have the grouper client have some operations that can go to WS and LDAP (primary and secondary or vise versa)... we should look into this possibility...