Summary
Rsyslog is used in the CommIT environment to enable centralized logging. This allows the instances to use ephemeral storage as well as providing an copy of all logs on a different server in the event of a security breach. This document details how rsyslog is deployed in this environment.
Installation
rsyslog is installed by default on Amazon Linux and presumably most other Linux variations in EC2. The current deployed version is 5.8.10. This is important as the syntax of the configuration file, /etc/rsyslog.conf, has changed dramatically over the later versions. The current version is 8.X and the default documentation on http://www.rsyslog.com is for that version. Documentation for the 5.X version can be found here: http://www.rsyslog.com/doc/v5-stable/index.html.
Since we are using version 5.X, we use what are now called Legacy configuration directives.
An rsyslog server instance has been deployed and given an Elastic IP of 54.214.22.10.
Server Configuration
The rsyslog server is configured to accept syslog messages from remote systems. This functionality is configured in /etc/rsyslog.conf. The important lines in that file are as follows.
# Provides TCP syslog reception
$ModLoad imtcp
...
$RuleSet REMOTE_Ruleset
$template REMOTE_Messages,"/var/log/remote-syslogs/%FROMHOST%/messages"
$template REMOTE_Secure,"/var/log/remote-syslogs/%FROMHOST%/secure"
$template REMOTE_Maillog,"/var/log/remote-syslogs/%FROMHOST%/maillog"
$template REMOTE_Cron,"/var/log/remote-syslogs/%FROMHOST%/cron"
$template REMOTE_Catalina,"/var/log/remote-syslogs/%FROMHOST%/catalina.out"
if \
$syslogseverity <= '6' \
and $syslogfacility-text != 'mail' \
and $syslogfacility-text != 'authpriv' \
and $syslogfacility-text != 'cron' \
and $syslogfacility-text != 'local1' \
then ?REMOTE_Messages
if \
$syslogfacility-text == 'authpriv' \
then ?REMOTE_Secure
if \
$syslogfacility-text == 'mail' \
then -?REMOTE_Maillog
if \
$syslogfacility-text == 'cron' \
then ?REMOTE_Cron
if \
$syslogfacility-text == 'local1' \
then ?REMOTE_Catalina
# These two lines must be defined *after* the ruleset it is defined (above)
$InputTCPServerBindRuleset REMOTE_Ruleset
$InputTCPServerRun 10514
$RuleSet RSYSLOG_DefaultRuleset
Note, the rsyslog server was configured to run in TCP mode and listen on port 10514 for connections from remote servers.
After making these changes, the rsyslogd process on the server must be restarted using /etc/init.d/rsyslog restart
Client Configuration
Clients must be configured to fork syslog messages to the rsyslog server. This functionality is configured in /etc/rsyslog.conf on the client instances. The important lines in that file are as follows.
# Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 ... $WorkDirectory /var/lib/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@54.214.22.10:10514
The IP address, namely 54.214.22.10, is the elastic IP of the rsyslog server configured above. Note, that the TCP protocol, as denoted by two '@@' versus one '@' for UDP, is used and the communication takes place over port 10514.
It may seem odd that we've also configured the client to listen for UDP communication on port 514. This is because the log4j syslog appender, which is used to support remote logging of Tomcat's catalina.out, connects via UDP on 514 by default and it doesn't appear to be a way to override that. This is configured in Tomcat as follows.
The instructions below assume that the Tomcat 6.0.39 is installed in /opt/apache-tomcat-6.0.39. Two "extra" jar files are required. These can be downloaded from the 'extras' area for the specific Tomcat version.
cd mkdir tmp cd tmp wget http://mirrors.gigenet.com/apache/tomcat/tomcat-6/v6.0.39/bin/extras/tomcat-juli-adapters.jar wget http://mirrors.gigenet.com/apache/tomcat/tomcat-6/v6.0.39/bin/extras/tomcat-juli.jar wget http://www.carfab.com/apachesoftware/logging/log4j/1.2.17/log4j-1.2.17.tar.gz tar xzf log4j-1.2.17.tar.gz cd apache-tomcat-6.0.39/ cp ~/tmp/apache-log4j-1.2.17/log4j-1.2.17.jar libcd /opt/apache-tomcat-6.0.39/ cp ~/tmp/apache-log4j-1.2.17/log4j-1.2.17.jar lib/ cp ~/tmp/tomcat-juli-adapters.jar lib/ mv bin/tomcat-juli.jar bin/tomcat-juli.jar.orig cp ~/tmp/tomcat-juli.jar bin/ mv conf/logging.properties conf/logging.properties.orig cp ~/tmp/tomcat-juli-adapters.jar lib mv bin/tomcat-juli.jar bin/tomcat-juli.jar.origcp ~/tmp/tomcat-juli.jar binmv conf/logging.properties conf/logging.properties.ori
Finally, a /opt/apache-tomcat-6.0.39/lib/log4j.properties file should be created and populated with the following:
log4j.rootLogger=INFO, CATALINA, SYSLOG
# Define all the appenders
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=localhost
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.Facility=LOCAL1
log4j.appender.CATALINA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.CATALINA.File=${catalina.base}/logs/catalina.log
log4j.appender.CATALINA.Append=true
log4j.appender.CATALINA.Encoding=UTF-8
# Roll-over the log once per day
log4j.appender.CATALINA.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout
log4j.appender.CATALINA.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.LOCALHOST=org.apache.log4j.DailyRollingFileAppender
log4j.appender.LOCALHOST.File=${catalina.base}/logs/localhost.log
log4j.appender.LOCALHOST.Append=true
log4j.appender.LOCALHOST.Encoding=UTF-8
log4j.appender.LOCALHOST.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
log4j.appender.LOCALHOST.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.MANAGER=org.apache.log4j.DailyRollingFileAppender
log4j.appender.MANAGER.File=${catalina.base}/logs/manager.log
log4j.appender.MANAGER.Append=true
log4j.appender.MANAGER.Encoding=UTF-8
log4j.appender.MANAGER.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.HOST-MANAGER=org.apache.log4j.DailyRollingFileAppender
log4j.appender.HOST-MANAGER.File=${catalina.base}/logs/host-manager.log
log4j.appender.HOST-MANAGER.Append=true
log4j.appender.HOST-MANAGER.Encoding=UTF-8
log4j.appender.HOST-MANAGER.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Encoding=UTF-8
log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
# Configure which loggers log to which appenders
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=INFO, LOCALHOST
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager]=INFO, MANAGER
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager]=INFO, HOST-MANAGER
Note the 'log4j.appender.SYSLOG.Facility=LOCAL1' bolded text in the configuration above. This configures the log4j syslog appender to use the 'local1' syslog facility. When these messages are sent to the remote rsyslog server, they are handled by the following section of '/etc/rsyslog.conf on the rsyslog server:
*if *
* $syslogfacility-text == 'local1' *
then ?REMOTE_Catalina
and thus sent to /var/log/remote-syslogs/%FROMHOST%/catalina.out.
Common Commands
Starting and Stopping Rsyslog
[root@ip-10-252-20-14 ~]# /etc/init.d/rsyslog start
[root@ip-10-252-20-14 ~]# /etc/init.d/rsyslog stop
Enabling Debugging
export RSYSLOG_DEBUG=Debug
rsyslogd -d -n
rsyslogd -d -n