(With thanks to Rob Carter)
Boyd: (Referencing forwarded PPT presentation) - Keith & I were talking about what we've done and our connector strategy – seemed like a potential for open-
sourcing (with Omnibond, LLC potentially offering commercial support as a side note). Focus today is on connectors and the overall architecture of the identity management infrastructure Clemson is using.
As depicted in the second slide of the deck, we have connectors developed for Unix/Linux, Blackboard, Banner and a collection of other target systems. The Kuali reference in the slide is due to Clemson's deploying Coeus (KC) for research administration.
In the third slide of the deck, note that the current Clemson architecture uses NetIQ as the basic identity store. There's a transform module for performing generic XSLT transformations and attribute mappings before data from the NetIQ store is passed to a proprietary SSL transport layer to the remote systems, where call-outs to either app-specific APIs or target connector scripts are used to actually deliver the data into target systems.
The next slide depicts the reverse direction of data flow, from SoRs into the registry. SoRs can generate events or make calls out to local scripts to pass data across the same proprietary SSL transport and back through the mapping and transformation layer before being loaded into the identity store.
Rob: Is this basically Novell's dirXML re-branded and extended by the new owners, or is the NetIQ core substantively different from the old Novell offering?
Boyd: It's still essentially the same thing.
Keith: What can you tell us about the transport mechanism and the event model?
Boyd: Essentially, in the NetIQ vault, there's an event capture mechanism that queues events to individual drivers. Each event is passed through an acceptance filter for the target driver to determine if the driver needs to receive it, and then triggers the driver to call out across the SSL channel (a proprietary remote loader protocol, which could be changed relatively easily) to push the data out.
Skipping to slide 5 in the deck, you can see there are three flavors of events – "add" events, "modify" events, and "delete" events. There can also be query and polling scripts available to be called when needed to retrieve data from other systems or retrieve additional data from the vault.
Back to slide 4… There's been some discussion of busses (in the ESB sense)… Our drivers can listen on a bus or something similar and pull events from it. There's still a queuing mechanism in NetIQ that needs to filter events for targets, one of which could probably be an ESB – processing in that sense could be essentially the same. Building an open source version of this, we'd probably replace the proprietary SSL protocol with some sort of HTTPS binding – the question is what folks might like to see in that regard.
Keith: Say you plugged into Penn State's CPR… They're using ActiveMQ for messaging… to plug bits of this architecture into the CPR, the message bus would essentially end up being ActiveMQ…
Boyd: Does ActiveMQ (and the other MQ mechanisms) keep track of what systems get which events, or does it just put them out on the bus and consumers sift them or not as they see fit?
Keith: It has a pub/sub model with the notion of channels
Rob: Yeah – it could easily mix with this sort of architecture, I think – channels would become the analog to target drivers in NetIQ parlance…
Keith: Is the data exchange similar to SCIM or something like SPML with call-outs? SCIM, for sure, has "add/update/delete" semantics in its operation set, as well as a way to represent the data that's been modified – does the transport in this case hold something similar?
Boyd: When I talk about event types and the transport… as long as we can map the event types onto standard forms of payload in the form of SCIM or SPML, I think it'd be fine. The current transport is just a proprietary XML document format, but so long as we're able to wrap whatever else we need in a different format, it should be a fairly straightforward thing.
Tom: What can you say about your Coeus connectors?
Boyd: Those are really new – just got started. Mostly, they're focused on getting users into and out of Coeus, now, but we'll see what else the Coeus folks decide they need.
Tom: Is this using RICE services? I ask because in the Coeus world, it seems that there are some attributes that are endemic to RICE and others that are specific to Coeus.
Boyd: I was just talking to the developer for our Coeus connector at lunch, and I forget exactly, but I think it's mostly using RICE services.
Tom: I'll share CMU's experience as I figure out more about what's going on here.
Boyd: Going back up the slides… There are windows , Linux/Unix versions in C, HTTP-bound web services, Remote powershell scripts, and all the obligatory Perl and Python bindings exist for the underlying libraries. There are script packages that are extensible to talk to Exchange, Windows Domain Controllers, Listserv… The Blackboard connector leverages Java classes for user/password/group provisioning in Blackboard… We have Banner connections in development using BEIS for users and Banner's LIS V2 implementation for course information.
Our main interest now is in figuring out how to connect to and what to use for the big "repository" space…
Keith: Question about IMS for learners and courses… It specifies a kind of add/modify/delete semantics for learners and course offerers?
Boyd: Yes – we're using the Java API because Blackboard doesn't expose quite enough functionality directly in their Bb API – the LIS V2 interface gives us everything we need, though.
Keith: Yeah – LIS V2 is of interest if we can figure out how to turn it into a Leggo brick, which is something you seem to be indicating a willingness to work on.. Can we discussion how that might look and what the requirements might look like? Which parts of this are interesting to folks on the call an which parts should be pulled out separately and made available in an open-source fashion?
All: <silence>
Keith: Ok.. So, when you think about this, Boyd, how would you carve out the interesting bits of the puzzle?
Boyd: I think it goes back to the core add/modify/delete constructs for identities… we could pretty much plug this into anything as long asyou could get the event through and queue up the next event – pretty easy on the app or system side to plug that in… We could look at Active MQ, if that's the right thing to use for a bus… what to do within that for payload to describe the event is an interesting question – is it SPML, SCIM, or whatever else that goes on the wire? Just gathering that from a few places and looking for a core set of requirements might be a good starting place.
Rob: How separable would you say the business logic associated with your current drivers and remote loaders is – that is, could you easily swap out NetIQ for something else, or swap out a connector, and retain he business logic (which would be the part that would be unique to a given site's implementation)?
Boyd: Many of these things are used across multiple sites already via NetIQ and Omnibond, so a the driver level, it's really pretty generic. The business logic is pretty well separated, and most of it lies in the XSLT transform policies, so the connectors themselves should be highly reusable. There are going to be things that hit, for example, the ActiveMQ channel that will need to be either protected of modified – something needs to understand the business logic – the event filtering and XSLT transformation layer – before events get into channels on the bus. We might have to build in a little extra support for the bus that we don't need for direct feed connectors, but for that added cost, we get dramatically better fan-out capabilities.
Rob: That seems reasonably well centralized, which probably makes it easier to change out between sites.
Keith: Any further questions for Boyd and Jeremy? No? I take this, then, to be the second of a series of engagements with you guys, and the next logical step may be more of a conceptual architecture discussion with some of us to come up with where, if the message bus integration is what needs real work, we might get started with that. Another bit might be understanding a bit better the stable of connectors and what you're wiling to put into open-source (with Omnibond's agreement). I understand that is something you guys are making a business of, and we're very appreciative of your willingess to discuss open sourcing some of the effort! We'll put our heads together and get back to you, but is that everyone else's sense of where we are with this? On ethnic I could imagine would be taking this and trying to use it to solve one of our provisioning user cases as PoC – that would be more ambitious an undertaking, but it might be more useful than anything else in he long run…
With that, Muhammad – do you want to talk about provisioning in the world of OpenRegisry at Rutges?
(2) Rutgers provisioning with OpenRegistry
Muhammad: We're using Camel at both the consumer and producer level. We use Camel to create events in our database table, each tagged with an affected person and an event type – one of "add person", "update person" or "delete person". After that, there's a consumer that listens on the event queue and provisions (currently) two endpoints – a legacy database endpoint and an LDAP directory. AS an event comes into the database queue, it's posted to a provisioning endpoint. In our roadmap, we plan to have ActiveMQ act as a consumer, so that Camel can put events into an MQ message channel, and then anyone that wants to can consume it from there. In Camel, we have about 70+ protocols – they call them components – we have plans to add JMS and ActiveMQ components to the mix. I didn't get time before the call to send around the diagram of the architecture, but I'll forward that along separately.
Keith: I know you guys are busy with ongoing Rutgers stuff plus you're on the CIFER registry calls – appreciate your help with the diagram if you get a chance to send it along. Just coincidentally, I happened to be sitting in on a FuseSource webinar about Camel, and began to believe that it might be possible for mere mortals to run Camel and configure it. When you mentioned the 70 components – if you're familiar with the EIP book (Enterprise Integration Patterns), the Leggo blocks in the book correspond to the components in Camel
Muhammad: We're using a database table now as the queue for Camel purposes – the queue is sorted by creation date, so Camel can treat the table as FIFO queue.
Keith: Sounds like a changelog polling model in the rough terms we've been using, with Camel generating the log on the producer side and Camel polling from the consumer side for messages. Very interesting. You're pretty much able, then, to take delivered components from Camel and connect them to LDAP and your legacy system - the CPD? Just a matter of using canned components, is it?
Muhammad - We use JPA to talk to the database, and then there's a Camel LDAP component that can write arbitrary data into an LDAP.
Rob: Where does Active MQ fit in the future model, then? Replacing the changelog table in the database, or…?
Muhammad: We'll leave the database table in its current role – we consider that to be part of the registry itself – ActiveMQ will be a second level consumer for some external system – it will be another endpoint for SAML.
Keith: I know there's been come conversation between the OR folks and Penn State's folks working on the CPR project. Jimmy V., in particular. Were you part of any of those conversations?
Muhammad: No, that must have been other folks.
Keith: That would be something interesting to look at – some interesting convergence with both Rutgers and Penn State supporting an ActiveMQ approach.
I asked Jimmy if they were perhaps using any of the Camel componentry, and they didn't seem to be – mostly they're using pretty simple connectors hung off the "far" side of the ActiveMQ message queue, but not doing lots of fancy Leggo building stuff with it yet. Might be interesting to drill into the confluence there…
Rob: I'm trying to figure out where the interface might get built between the registry and our P&I stuff – where's the cleavage plane where we could build the "plug and play" API to allow multiple different registries (eg., CPR vs. OR) to share common integration and provisioning code bidirectionally with other systems.
Keith: Yes – if we could define a cleavage plane and then agree to an API for it, we could mix-and-match connectors, scripts, etc. with different compliant registries.
Rob: Maybe it's a pipe-dream, but it seems like something worth trying for, at least.
Keith: Actually, that's one of the items in the 3-6 month work prospectus I'd posted – III-a-4 is "vetting emerging CIFER APIs against P&I Use Cases"… I put call information for the CIFER API calls into the wiki, btw). I think we can now say a bit more about what that might look like – with the APIs defined for OR and CPR, eg., we can do what we're talking about… What we've' heard today is different from the API approach we've been hearing on the other calls – the definition or scope of the apis might need to be expanded to incorporate this, but we I think Rob and I both agree that we (in the P&I workstream) are the ones most likely to want to stress-test the APIs as they're devised.
Rob: Yeah – I keep thinking that the restful interfaces the API folks are shooting for designing lean toward the transport layer , but there's a tie-in point needed somehow to the static data representation in the repository and/or to the bus interface…
Keith: If you guys (Boyd, Jeremy, Muhammad) are willing to have another phone call or Skype session, maybe we can push forward?
Boyd: Jeremy and I were just talking on mute about the Active MQ confluence – maybe there's how you connect to the bus (using open wire-level protocols) – we can probably easily integrate with a bus here, but the other piece is that either CPR or OR has to somehow put events onto the bus to make it all fit together.. We need to come up with a common event description – maybe SPML or SCIM or something (so w don't have to re-invent our own)… We've built a whole network registration system out of this stuff where we get DHCP events and consolidate that data with registration information for computers… My point is that this stuff doesn't have to be solely for people data – it can be used for other "identity" data as well… It would be nice to have whatever's on the bus as a data format be flexible enough to look at people and groups but also other non-person things – computers and such.
Keith: Exactly – the whole notion of integration patters is that you can integrate data beyond just people – I like that idea for sure. Also like the notion that ActiveMQ may be a common point of reference for discussion whether it's a common point or architecture or not (which is still up for discussion). Camel may help, also, in that it's basically EIP implemented as a Leggo set… and yes, I think you're right about SCIM/SPML – there's interesting work to be done around what goes into the message schema..far too much for one brain and one call, I'm sure. I always come onto these calls trying to get other people to take on action items, though, so…
AIs:
- Keith: Set up second talk with Boyd & Co. about next steps with their provisioning model
- Rob: Carry over the AI I have to get Gary from Cal Davis in to discuss their use case(s)
- Rob: Take a note to try and get Omer or Muhammad and Jimmy V. (re)introduced and see about getting a discussion going about ActiveMQ and the confluence there.
- Keith: Talk to Celeste about next steps with their provisioning model
(3) 3-6 month work plans
Keith: Item 3 on the agenda is 3-6 month work plans – Under the January 31 heading (note that I've already moved everything back one month already)…
With a long list of use cases to develop demo solutions from – I've got UNC's Improv provisioning mechanism on there, Celeste, if you're wiling to do another call to nail down the use case you've got and what we want to document there?
Celeste: Yes – sounds good.