Initial Investigations: Analysis of Grouper Logs using the Free and Open Source ELK stack (Elasticsearch, Logstash and Kibana)
Current work: Testing the ELK stack for Grouper log aggregation, exploration and visualization; The Grouper Demo Server is serving as the data source.
The ELK stack brings together
- Logstash for aggregation of monitoring data: Logstash can consume data from multiple sources and transport modes:
|
- Elasticsearch for searching, persistence and analysis; Uses https://lucene.apache.org/ as its search engine.
- Kibana for data exploration, analysis and visualization
This whole field is in an explosive phase of growth, and it is well on its way to becoming a first-order discipline of its own. Since TIER architecture is premised on distributed components that are loosely coupled, the ELK stack is an extremely useful tool for addressing TIER monitoring needs.
One of the deep challenges for TIER will be determining which specific sorts of data and data analysis will give us the greatest degree of visibility into the physiological processes and state of health of the distributed TIER infrastructure.
4 Comments
Nicole Roy
Have the huge problems with the inability to provide authentication for the Kibana frontend without breaking its backend calls to Elasticsearch been figured out? We were running ELK at Penn State IDS and stopped using it because of this enormous security problem.
Nicole Roy
Looks like there is now a basic AuthN plugin that can be backed by LDAP and AD (so, LDAP and LDAP). https://www.elastic.co/products/shield
Brian Savage (bc.edu)
Shield seems to require an Elastic commercial subscription. Some are opting for an alternative like Graylog that still uses Elasticsearch for indexing, but comes with access control security (can tie to ldap) and some other features. Employing it brings a decision about whether to use its feed mechanisms or make use of the power of logstash. Tradeoffs.
James Babb (wisc.edu)
The way we had run it at my previous employer (and how I more or less had the demo working on Keith's server) was everything was restricted to listening only internally on localhost including Kibana. Apache sat as a proxy between the outside world and Kibana. We then could use any apache auth methods (basic, digest, shib, etc) to provide authentication for Kibana. This seemed to work pretty well.