Page History

 
See email thread "grouper ActiveMQ integration"
 
Here is something I have been working on, let me know any thoughts, perhaps we could discuss at the dev call on wed. Anyone else interested in using this?
https://spaces.at.internet2.edu/display/Grouper/Grouper+ActiveMQ+integration
  
BTW, I hope Penn and Penn State (and Brown?) can document some lessons learned on general ActiveMQ configuration / loadTesting / configuration / etc.
ActiveMQ is an open source application messaging middleware service where applications can send messages to other applications (or groups of consumers) in real-time without polling.
  
Grouper can be used to manage ActiveMQ authorizations. University of Pennsylvania and Penn State are interested in this component.
  
ActiveMQ users can send/receive messages to topics/queues, and also can dynamically create topics/queues. Grouper permissions can be exported and kept in sync (cron'ed and real-time) with a local permissions store. An ActiveMQ plugin reads the local permissions and enforces the rules.
  
The Grouper-ActiveMQ component is a jar that you can add to the ActiveMQ system (along with a few other jars), and some configuration that allows authorization managed from Grouper.
  
Thanks,
Chris

 
At UW we've used ActiveMQ for group change notices for several years.
It has generally worked well enough, but for many reasons all of our
new group messaging activities are using Amazon's SNS/SQS messaging.
 
ActiveMQ is expensive. You have to get and maintain hardware,
install and document the software, monitor it, be on-call during the
night, administer it. It might be open source, but it isn't free.
 
If you are lucky enough to have an existing ActiveMQ service
operated by another department, and they are willing to let use use
it mostly for free (our situation), hope they will continue
to give you wonderful service after they have migrated their own
stuff to RabbitMQ, or OpenMQ, or some other MQ and you're the
only one left on the ActiveMQ system.
 
Connections can hang. We use the CMS client, for c++ with an
interface library for plain c. (You know there's trouble with an
interface when the first thing you absolutely need is to find an
interface package for your language.) This interface uses persistent
connections and certain conditions on the server will cause these
connections to hang. This is a difficult situation to monitor.
Generally an external process has to send test messages and report
when none has been processed for some timeout period.
 
Amazon's messaging service, by comparison, is nearly free, is
exceptionally easy to use (there are client libraries for many
languages, including both our present favorites: java and python,
but I had no trouble writing a c client from scratch), easy to
configure (a web client make most administration self-service),
and is reliable. It is also RESTful.
 
We have, as do most universities nowadays, initiatives to use cloud
services in lieu of a 'running everything ourselves' mentality.
Our people are not so fearful of losing control of even core
services. It is the future.
 
There are a couple of advantages ActiveMQ has over Amazon SNS.
 
ActiveMQ is an order of magnitude faster than SNS. Is that important?
For our network team, monitoring thousands of routers and whatnot,
and sending millions of messages per day, it is important. For our
group service, not so much. Messaging for course enrollment,
operating since summer, now totals about 300,000 adds or drops.
All the messages get to our consumers in a few seconds. And a few
seconds is a short time for this application.
 
ActiveMQ always delivers messages in order (I think). With SNS/SQS
they can get out of order. Something to deal with.
 
Jim

Some questions regarding SQS...
 
1. Is there a way to do a topic as opposed to a queue? i.e. I want to clear an authz cache which is on 10 app servers. I want to send the message to one topic, and whoever is subscribed will get the message.
 
2. I don't really like the polling. I want to send a message and the receiver will get it in real time. With SQS I have to poll every minute, and even then, it might not be ready and it will be there (hopefully) the next time after that...
 
3. The polling doesn't seem like it scales... if every need to get a message is a poll (every minute? every 15 seconds?) then that are a lot of SSL web service requests as opposed to a socket which is just waiting for traffic. If you have a lot of apps on a machine or a cluster it seems like this polling can add up.
 
4. Whoops, I think the last time I estimated the cost of SQS I was calculating polling every second, which is expensive. But I think for one endpoint to poll SQS every minute, it costs 50 cents per year ok, that is pretty cheap...
 
Thanks,
Chris