Summary
Rsyslog is used in the CommIT environment to enable centralized logging. This allows the instances to use ephemeral storage as well as providing an copy of all logs on a different server in the event of a security breach. This document details how rsyslog is deployed in this environment.
Installation
rsyslog is installed by default on Amazon Linux and presumably most other Linux variations in EC2. The current deployed version is 5.8.10. This is important as the syntax of the configuration file, /etc/rsyslog.conf, has changed dramatically over the later versions. The current version is 8.X and the default documentation on http://www.rsyslog.com is for that version. Documentation for the 5.X version can be found here: http://www.rsyslog.com/doc/v5-stable/index.html.
Since we are using version 5.X, we use what are now called Legacy configuration directives.
An rsyslog server instance has been deployed and given an Elastic IP of 54.214.22.10.
Server Configuration
The rsyslog server is configured to accept syslog messages from remote systems. This functionality is configured in /etc/rsyslog.conf. The important lines in that file are as follows.
# Provides TCP syslog reception
$ModLoad imtcp
...
$RuleSet REMOTE_Ruleset
$template REMOTE_Messages,"/var/log/remote-syslogs/%FROMHOST%/messages"
$template REMOTE_Secure,"/var/log/remote-syslogs/%FROMHOST%/secure"
$template REMOTE_Maillog,"/var/log/remote-syslogs/%FROMHOST%/maillog"
$template REMOTE_Cron,"/var/log/remote-syslogs/%FROMHOST%/cron"
$template REMOTE_Catalina,"/var/log/remote-syslogs/%FROMHOST%/catalina.out"
if \
$syslogseverity <= '6' \
and $syslogfacility-text != 'mail' \
and $syslogfacility-text != 'authpriv' \
and $syslogfacility-text != 'cron' \
and $syslogfacility-text != 'local1' \
then ?REMOTE_Messages
if \
$syslogfacility-text == 'authpriv' \
then ?REMOTE_Secure
if \
$syslogfacility-text == 'mail' \
then -?REMOTE_Maillog
if \
$syslogfacility-text == 'cron' \
then ?REMOTE_Cron
if \
$syslogfacility-text == 'local1' \
then ?REMOTE_Catalina
# These two lines must be defined *after* the ruleset it is defined (above)
$InputTCPServerBindRuleset REMOTE_Ruleset
$InputTCPServerRun 10514
$RuleSet RSYSLOG_DefaultRuleset
Note, the rsyslog server was configured to run in TCP mode and listen on port 10514 for connections from remote servers.
After making these changes, the rsyslogd process on the server must be restarted using /etc/init.d/rsyslog restart
Client Configuration
Clients must be configured to fork syslog messages to the rsyslog server. This functionality is configured in /etc/rsyslog.conf. The important lines in that file are as follows.
# Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 ... $WorkDirectory /var/lib/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@54.214.22.10:10514
The IP address, namely 54.214.22.10, is the elastic IP of the rsyslog server configured above. Note, that the TCP protocol, as denoted by two '@@' versus one for UDP, is used and the communication takes place over port 10514.
It may seem odd that we've also configured the client to listen for UDP communication on port 514. This is necessary to allow the log4j syslog appender to talk to the rsyslog daemon on the client instance which then forwards the logging requests over to the rsyslog server. This is used to support remote logging of Tomcat's catalina.out. This is configured in Tomcat as follows.
The instructions below assume that the Tomcat 6.0.39 is installed. Two "extra"
Common Commands
Starting and Stopping Rsyslog
[root@ip-10-252-20-14 ~]# /etc/init.d/rsyslog start
[root@ip-10-252-20-14 ~]# /etc/init.d/rsyslog stop
Enabling Debugging
export RSYSLOG_DEBUG=Debug
rsyslogd -d -n
rsyslogd -d -n