EXTRA - rsyslog, syslog-ng

rsyslog - syslog-ng

Rsyslog (Remote syslog)

The main benefit of the Rsyslog utility is that it can forward usual syslog messages to a remote server. It implements the basic syslog protocol, and extends it with content-based filtering, queued operations (to handle log sending if daemon is offline), flexible configuration options and adds features such as using TCP for transport.

It can be downloaded from IBM AIX Web download programs page:
https://www-01.ibm.com/marketing/iwm/iwm/web/pickUrxNew.do?source=aixbp

After untar (install.images.ryslog-8.4.2.3.tar ) there will be 2 files, which can be installed with installp:
-rw-------    1 332313   563716      1930240 Mar 15 08:35 rsyslog.base
-rw-------    1 332313   563716       278528 Mar 15 08:35 rsyslog.license

# installp -Yd . rsyslog.base rsyslog.license


an example configuration file:
/etc/rsyslog.conf:
aso.notice /var/log/aso/aso.log
aso.info /var/log/aso/aso_process.log
aso.debug /var/log/aso/aso_debug.log
caa.debug /var/adm/ras/syslog.caa
mail.debug /var/log/mail
user.debug /var/log/user
kern.debug /var/log/kern
syslog.debug /var/log/syslog
daemon.debug /var/log/daemon
auth.debug /var/log/secure
local2.debug /var/log/sudo
kern.*,daemon.*,auth.* @logserver.example.domain.com:514;RSYSLOG_SyslogProtocol23Format


After that we can switch between syslog and rsyslog:
syslog_ssw -r                  <-- switch to rsyslog
syslog_ssw -s                  <-- switch to usual syslog


-----------------------------------------
-----------------------------------------
-----------------------------------------

syslog-ng

Syslog-ng is a system logging application, which can be a replacement of the default syslog. With syslog-ng, the log messages can be sent in an ecrypted/secure channel to a remote server. If the central log server or the network connection becomes unavailable syslog-ng will store messages on the local hard disk. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted. (Another possibility to send those messages to a secondary server.)

syslog-ng can filter log messages and select only the ones matching certain criteria, but it cannot interpret and analyze the meaning behind the messages. It can receive messages from files, remote hosts, and other sources, and these are sent to one or more destinations (files, remote hosts..),

It has a server - client model, here only syslog-ng client informations will be described (syslog-ng server has not been tested, only client, which were sending messages to a remote server (qradar).)

(One missing feature of syslog-ng, that it cannot rotate logs by itself. For log rotation an external tool needs to be used.)

-----------------------------------------

Configuration

The syslog-ng application is configured by editing the syslog-ng.conf file. The syslog-ng.conf and license.txt files are located in the /opt/syslog-ng/etc/ directory.
The configuration file is similar to an Object oriented programming language, where different objects are defined with specific parameters.

The main body of the configuration file consists of object definitions:

object_type object_id {<options>};

Main object types can be: sources, filters, destinations, log paths, which together define which log messages are received and where they are sent. Object definitions end with a semicolon (;)

-----------------------------------------

Log path

Log paths are the final statements (usually at the end of the conf. file), which joins together the different objects (sources, destinations...). Log paths determine what happens with the incoming log messages. Messages coming from the sources and matching all the filters are sent to the listed destinations.


Here is a basic configuartion (without any filters):

source s_local { system(); internal(); };                    <--source definition (lists which sources will be used)
destination d_local { file("/var/log/messages"); };          <--destination definition (shows which file to log incoming messages)
log { source(s_local); destination(d_local); };              <--log path (it combines sources and destinations)


-----------------------------------------

Sources

A source is where syslog-ng receives log messages from.

These "drivers" (categories) define where and how messages are received:
file()       Opens the specified file and reads messages.
internal() Messages generated internally in syslog-ng.
network() Receives messages from remote hosts
program() Opens the specified application and reads messages from its standard output.
system() collects the native log messages of AIX (same function as original syslog daemon)


file(): 
If messages has to be collected from a text file, "file()" parameter (driver) needs to be used. When syslog-ng is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this position after the restart. The filename (but not the pathname) may include wildcard characters (for example *). Note that when using wildcards in filenames, always set how often syslog-ng should check the file for new messages using the follow-freq() parameter. (because it can generate extra load)

internal():
All messages generated internally by syslog-ng use this special source. To collect warnings, errors and notices from syslog-ng itself, include this source in one of your source statements.

system():
This provides the same function as the native syslogd. Usually system() and internal() sources are configured together in the beginning of the configuration file.

-----------------------------------------

Filters

Filters are narrowing the selection of all the incoming messages. With the help of the filters, we can select specific messages to be logged to one destination, and other messages to another destination.

For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

Main filter categories:
facility() Filter messages based on the sending facility (these can be used: kern, user, mail, daemon, auth, syslog...local0-local7)
host()       Filter messages based on the sending host.
level()      Filter messages based on their priority. (these can be used: emerg, alert, crit, err, warning, notice, info, debug)
match()      Use a regular expression to filter messages based on a specified header or content field.
message() Use a regular expression to filter messages based on their content.
source() Select messages of the specified syslog-ng source statement.


facility():
facility () filter accepts these facilities:

level():
level() filter accepts these levels: emerg, alert, crit, err, warning, notice, info, debug

For example, to select every message of error or higher level, use this in filter: level(err..emerg)
-----------------------------------------

Destination

A destination is where a log message is sent, if the filtering rules match. Similarly to sources, destinations consist of one or more drivers (categories), each defining where and how messages are sent.

Main destination drivers:
file()         Writes messages to the specified file.
network() Sends messages to a remote host
program() Forks and launches the specified program, and sends messages to its standard input.
usertty() Sends messages to the terminal of the specified user, if the user is logged in.

-----------------------------------------

syslog-ng.conf file

All sources, filters, destinations and log paths are configured in syslog-ng.conf file.
Usual naming convention:
sources:       s_<any_name>
destinations:  d_<any_name>
filters:       f_<any_name>


!!!Important note:
Sources and destinations are initialized only when they are used in a log statement (log path). For example, syslog-ng starts listening on a port or starts polling a file only if the source is used in a log statement.

# cat syslog-ng.conf

@version: 5.0                                                <--syslog-ng conf. file must begin with a line containing the version information
#Default configuration file for syslog-ng.
...

@include "scl.conf"                                          <--the syslog-ng Source Configuration Library (SCL) must be included as well

options { };                                                 <--global options can be defined here (for example to use dns-cache or not)

######
# sources
source s_local { internal(); system(); };                    <--captures syslog-ng internal messages and default system logs (like syslogd)
source s_myapp { file("/usr/log/*.txt" follow-freq(10)); };  <--specific txt files are configured as well for sources


######
# destinations                                                <--the usual AIX syslog files are defined for destination
destination d_syslog { file("/var/adm/syslog/syslog.log"); };
destination d_kern { file("/var/adm/syslog/kern.log"); };
destination d_user { file("/var/adm/syslog/user.log"); };
destination d_mail { file("/var/adm/syslog/mail.log"); };
destination d_daemon { file("/var/adm/syslog/daemon.log"); };
destination d_auth { file("/var/adm/syslog/auth.log"); };
destination d_lpr { file("/var/adm/syslog/lpr.log"); };
destination d_news { file("/var/adm/syslog/news.log"); };
destination d_uucp { file("/var/adm/syslog/uucp.log"); };
destination d_all_err { file("/var/adm/syslog/problem.log"); };
destination d_local1 { file("/var/adm/syslog/local1.log"); };

#destination d_qradar { network("remotehost.abc.com" port(514) transport("udp") log-fifo-size(1000) ); };
destination d_qradar { network("remotehost.abc.com" port(6514) transport("tls") log-fifo-size(1000) tls(peer-verify(optional-untrusted)) ); };

#cluster
destination d_console { file("/dev/console"); };                 <--destination will be the console here
destination d_cluster { file("/var/cluster.log"); };
destination d_caa { file("/var/syslog.caa"); };


######
# filters
filter f_syslog { facility(syslog); };                           <--these are filters by facility
filter f_kern { facility(kern); };
filter f_user { facility(user); };
filter f_mail { facility(mail); };
filter f_daemon { facility(daemon); };
filter f_auth { facility(auth); };
filter f_lpr { facility(lpr); };
filter f_news { facility(news); };
filter f_uucp { facility(uucp); };
filter f_local1 { facility(local1); };

filter f_err { level(err..emerg); };                             <--these are filters by severity
filter f_debug { level(debug..emerg); };
filter f_warn { level(warn..emerg); };

#cluster
filter f_console { facility(local0) and level(crit..emerg); };   <--combining facility and level with boolean operator (it means: local0.crit)
filter f_cluster { (facility(local0) and level(info..emerg)) or (facility(user,daemon) and level(notice..emerg)) };  <--it means:
                                                                                                       local0.info;user.notice;daemon.notice
filter f_caa { match("caa:" value("MESSAGE")) };             <--it filters for the word "caa:" in the header or in the message of the log


######
# log paths
log { source(s_local); filter(f_syslog); filter(f_debug);  destination(d_syslog); };   <--log paths to files
log { source(s_local); filter(f_kern); filter(f_debug); destination(d_kern); };
log { source(s_local); filter(f_user); filter(f_debug); destination(d_user); };
log { source(s_local); filter(f_mail); filter(f_debug); destination(d_mail); };
log { source(s_local); filter(f_daemon); filter(f_debug); destination(d_daemon); };
log { source(s_local); filter(f_auth); filter(f_debug); destination(d_auth); };
log { source(s_local); filter(f_lpr); filter(f_debug); destination(d_lpr); };
log { source(s_local); filter(f_news); filter(f_debug); destination(d_news); };
log { source(s_local); filter(f_uucp); filter(f_debug); destination(d_uucp); };
log { source(s_local); filter(f_err); destination(d_all_err); };
log { source(s_local); filter(f_local1); filter(f_warn); destination(d_local1); };

log { source(s_local); destination(d_qradar); };                                      <--log paths to remote server
log { source(s_myapp); destination(d_qradar); };

#cluster
log { source(s_local); filter(f_console); destination(d_console); };
log { source(s_local); filter(f_cluster); destination(d_cluster); };
log { source(s_local); filter(f_caa); filter(f_info); destination(d_caa); };

-----------------------------------------

/opt/syslog-ng/sbin/syslog-ng -V                                        <--it will show syslog-ng verion, settings
/opt/syslog-ng/sbin/syslog-ng --syntax-only                             <--before activating conf file syntax can be checked for correctness
/opt/syslog-ng/sbin/syslog-ng -f /opt/syslog-ng/etc/syslog-ng.conf      <--it will activate (reload) conf. file
stopsrc -s syslog-ng; sleep 5; startsrc -s syslog-ng                    <--restarting fully syslog-ng

/opt/syslog-ng/sbin/syslog-ng -Fevd                                     <--syslog-ng will show in foreground debug messages about its work
logger -p auth.err "auth.err test syslog"                               <--test logging (to see if message will appear in syslog-ng)

-----------------------------------------

The filename (not the path) can include UNIX-style wildcard characters (*, ?). When using wildcard characters, syslog-ng will include every matching file.

When including a directory, syslog-ng will try to include every file from the directory, except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive

-----------------------------------------

The following filter statement selects the messages that contain the word deny and come from the host "example":
filter demo_filter { host("example") and match("deny" value("MESSAGE")) };

(Here a macro "MESSAGE" was used, which contain the text contents of the log message .)

-----------------------------------------

Install/Uninstall


Install:
After downloading the syslog-ng installer package, we have 2 options
- run ./syslog-ng-<edition>-<version>-<OS>-<platform>.run script, or
- install as an rpm package: rpm -i syslog-ng-premium-edition-<version>-<OS>-<arch>.rpm

During install, the default syslogd will be replaced automatically by syslog-ng (no parallel operation is possible).


Uninstall:
If the .run installer has been used: /opt/syslog-ng/bin/uninstall.sh (The uninstall script will automatically restore the syslog daemon used before installing syslog-ng.)
If the .rpm package has been used: rpm -e syslog-ng-premium-edition (with rpm, it does not restore the syslog daemon used before syslog-ng).

-----------------------------------------