Post PSP Provisioning
Following on from the discussion on the July 3, 2013 Grouper Dev call and the July 29, 2014 and the August 27, 2014 Grouper Dev call, this page captures thoughts about the future of the PSP and directions for provisioning strategy. General consensus:
- The PSP is weighted heavily towards trying to be all things to all people.
- Due to the above, it suffers horribly in the area of performance.
- The community believes that messaging and more specifically a queue is the future direction that they all need to move in
- Building on the message-queue concept, Grouper should create a generic interface and message format that would translate changelog events or events triggered by hooks in Grouper into generic messages.
- Grouper would also create an interface to read/validate messages
- The community and project would use this message format and interface to write provisioners.
- The two messaging systems that the Grouper project will launch with support for are ActiveMQ and Amazon SQS
- Grouper will also attempt to write an implementation of the interface that does not rely on an external queue so that adopters who do not wish to run AMQ or connect up to Amazon could still run the provisioners locally.
General Requirements
General requirements moving forward for message queue readers:
- Support incremental provisioning by reading events off the message queue
- Support bulk reconciliation. Note: they may not be the same executable, but should retain and reuse code and configs where possible/practical.
- When doing bulk-reconciliation the provisioner shall support optionally being pointed at a particular group or folder to reconcile
- Each module should be capable of being run in parallel with other modules
- All provisioning modules will follow the Grouper configuration paradigms with properties overlays and expression language.
- Modules shall provision based either on the location of a group in the tree as done today or using an attribute that will instruct the provisioner to provision the group and/or how to provision the group
Message Format
This still needs to be worked out.
Deployer | Message Format Documentation |
|---|---|
Washington | https://wiki.cac.washington.edu/display/infra/Groups+AWS+Event+Messaging |
Carnegie Mellon | |
Grouper ESB | link to grouper ESB |
- JSON will be the message format.
- Payload is JSON too
- Messages should always be signed (using JOSE JWS) – Config option to turn off signing, but leave on by default.
- Messages could be optionally encrypted (JOSE JWE)
- Messages will always carry their own metadata and consumers will not rely on the messaging system metadata but rather the grouper-produced metadata (this will keep our messages messaging-system-agnostic)
- When messages get too big, the message is chunked and a new message will be created for additional info
What's in a message
- All info about a group (name, desc, etc)
- Incremental information (adds / deletes)
First Implementations
Prospective first implementations for Grouper 2.3 are:
- LDAP
- Active Directory
Configuration
The provisioning modules will be configured via properties files using standard grouper configuration mechanisms. Modules will be activated by either calling them from GSH for batch/reconciliation functions or either a changelog consumer or Grouper hooks for incremental provisioning. Hooks will be investigated as a means of provisioning more quickly. The provisioners will support two kinds of configuration:
Global Configuration
Global configuration of the modules will be done by properties files specific to all instances of that module. Some of the envisioned properties set in these files include
- Hostnames / connection endpoints of provisioned systems
- Authentication information for provisioned systems
- Logging options
- Scheduling for running batch reconciliations
- Conflict handling
- Thresholds to prevent a provisioner from performing a change that would affect N% of users in a group
- Transformation instructions for turning group names or membership names into a format that the downstream system expects
Categorical Configuration
The idea behind categorical configuration is that rather than decorate a specific group to be provisioned to a specific target endpoint, we create an abstraction capability. The idea is that a group could receive the provisioning decoration of standard which would signal the downstream provisioners looking for standard to provision their targets accordingly. In this manner, a Group Admin, knowing that standard meant provision to LDAP, AD, and Google Apps, for instance, could just apply that one attribute & be done with configuring the provisioning. Categorical implementation would likely take the form of a group attribute with some metadata explaining which targets to hit.
Group-Level Configuration
This needs to be re-thought in light of the message consumers. Do we put the attributes into the message to inform the downstream consumers about it or do we require the consumers to call-back into grouper for additional data. My gut says we should do our level best to ensure a message has all the data a consumer would need to keep speed up and callbacks down.
Group level configuration would be handled by attributes placed upon groups or folders. The following standard attributes are envisioned:
- $NameProvision – when set on a group, that provisioner will provision the memberships to the target system. When set on a folder, all groups under that folder will be provisioned to the downstream system.
Specific provisioners may specify additional attributes they will use to determine how to provision that group's membership. Some examples are:
- Attribute to describe where in the LDAP DIT the group shall live. If this is set, it will override any global configuration.
- Attribute to describe the type of provisioning (members of this group shall represent Google Apps Organizations vs Google Apps Groups)
- Customized provisioning thresholds (for this group, refuse to provision if more than N% of the membership is affected)
The Grouper UIs will also be updated to facilitate managing of these attributes.
15 Comments
Michael Gettes
Just about all the requirements noted thus far can be achieved with ActiveMQ. The CMU GAP code has the ability to perform the bulk recon either for all or selected groups. Our code doesn't support provisioning based on attributes but that wouldn't be hard to add. It would be a shame to reinvent the message queue wheel when there are existing products solving the problem space in very reliable, elegant and even free ways!
David Langenberg (uchicago.edu)
Yeah, there's just two issues with the CMU stuff. Using it would require adding ActiveMQ to the dependency list. While how big of a burden this would be on a grouper deployment is debatable, we're hesitant to add something on that order as a dependency. The second problem is I believe your stuff requires the use of Perl, so not only would a deployer, as well as the project, have to deal with Java and all its dependency issues, they also descend into Perl dependency purgatory and have to ensure their grouper servers have Perl.
Michael Gettes
Frankly, I don't see your comments about Perl as an issue. Perl comes with most linux distros and the modules we use are all part of CPAN... easily gotten. The AMQ dependency question is more reasonable. Lots of products embed AMQ these days. I think there needs to be some discussion with those with actual experience instead of making assumptions about how one would deploy. I'd be happy to participate in such a discussion and I believe the folks from PSU would as well. This can all be done in a way which is far simpler than you make it out to be.
David Langenberg (uchicago.edu)
The problem with Perl is that, yes, it does come with Linux, but not Windows. So for windows shops that'd mean needing an additional skill-set (Perl wrangler) that the vast majority of Windows admins lack.
thompsow@lafayette.edu
These kinds of threads are better off on the dev list.
Seems more likely that a Windows Admin would have Perl than Java skills, no? In any case, I agree it doesn't make much sense for a Java based project to take on Perl as well. I do think it would be wise to evaluate supporting AMQ (or something like it) more or less OOTB. Perhaps the CMU Perl modules could be basis for a design or ported to Java/Groovy?
David Langenberg (uchicago.edu)
Probably so. I find the windows admins I run into are much more familiar with Java than Perl at least from a "how do I get this app running?" standpoint.
nsr11@psu.edu
As a former Windows admin, I can say that Java and Perl are equally foreign, and many enterprise Windows people are used to having to support them in some capacity. I had to learn to support Java in order to run an IdP. All our legacy provisioning code was Perl, which we slowly ported to PowerShell. Languages (scripting or otherwise) are all effectively the same, they all suck, and anyone who can learn one can learn another. There are Perl distros for Windows. Penn State is planning to use Michael's AMQ code, and we have found AMQ to be relatively easy to manage and install. I believe having some kind of AMQ equivalent along with Grouper OOTB, which can be used for provisioning, would be a boon to Grouper.
If support on Windows without foreign dependencies is truly something you want to try to do, then you'd have to write everything in .NET and Powershell. If not, Java, Perl, doesn't really matter what it is, it's not something a Windows admin is going to be trained to support unless they've had to have exposure to it before, which they likely have, if they are worth their salt.
David Langenberg (uchicago.edu)
Right, Nick, but since we have to make them deal with Java, let's not go crazy and force a second language on them.
nsr11@psu.edu
That's a very good point. Michael, is there a reason you went with Perl and not Java? Is it because it is easier to get in and change things without having to have a full dev environment?
Michael Gettes
This has absolutely nothing to do with our decision to use Perl. This has everything to do with not reinventing the message queue wheel. The requirements I see indicate reinventing the AMQ-like wheel and I feel this is a waste of limited resources. This is not about Java/Perl/whatever. If you want to, for example, include AMQ as part of Grouper and then write your own consumer to do what we have done with GAP in Java... I don't care. I already have a number of folks asking for our Perl code so I don't think there is the purest perspective you may think. I will note we provide java based mechanisms to perform fullsyncs and tree-level sync as well. We are in process of putting our code up under BSD license.
Likewise, requiring all configuration to be handled like the rest of grouper config, I don't think that should be a requirement either. Should config be easy? yes. Grouper config is NOT easy. Solve the problem of easy.
nsr11@psu.edu
While I like the idea of keeping message metadata messaging-system-agnostic, doing that will also prevent use of certain nice features enabled by keeping the messaging metadata in the message headers. Specifically, the ability to use JMS message selectors to provision specific types of groups to specific destinations would be nice.
David Langenberg (uchicago.edu)
I don't disagree, and I don't see why we couldn't / shouldn't also leverage what a messaging system provides in the way of metadata capabilities. The problem though is different systems will have different support for what they can / will carry in message metadata, so also keeping our own metadata will solve that problem for all messaging systems.
nsr11@psu.edu
Do all messages need to be signed? If we are just doing internal provisioning and the messages never leave the IAM infrastructure, I think signing would cause some overhead that could be unnecessary in some cases. Could it be optional?
David Langenberg (uchicago.edu)
It could be optional, but I think we'd ship it with a default of on. I assume in your message platform you very tightly control who can publish to a topic. In our deployment, we're looking at cloud for the messaging platform which, for us, means we want to be sure of the authenticity of stuff coming back from the cloud.
nsr11@psu.edu
Yep, we are fully in control of our internal provisioning messaging infrastructure, so not a concern for us (yet). Eventually it probably will be, but having an option would be good. Thanks!