A utility that locates and opens files in CentOS 7 is
Log files are files that contain messages about the system, including the kernel, services, and applications running on it. There are different log files for different information. For example, there is a default system log file, a log file just for security messages, and a log file for cron tasks. Show
Log files can be very useful when trying to troubleshoot a problem with the system such as trying to load a kernel driver or when looking for unauthorized login attempts to the system. This chapter discusses where to find log files, how to view log files, and what to look for in log files. Some log files are controlled by a daemon called Log files can also be managed by the By default, these two logging tools coexist on your system. The 23.1. Locating Log Files A list of log files maintained by You may notice multiple files in the 23.2. Basic Configuration of Rsyslog The main configuration file for rsyslog is 23.2.1. Filters A rule is specified by a filter part, which selects a subset of syslog messages, and an action part, which specifies what to do with the
selected messages. To define a rule in your rsyslog offers various ways to filter syslog messages according to selected properties. The available filtering methods can be divided into Facility/Priority-based, Property-based, and Expression-based filters. Facility/Priority-based filters The most used and well-known way to filter syslog messages is to use the facility/priority-based filters which filter syslog messages based on two conditions: facility and priority separated by a dot. To create a selector, use the following syntax: FACILITY.PRIORITY where:
Property-based filters let you filter syslog messages by any property, such as Property-based filter must start with a colon ( :PROPERTY, [!]COMPARE_OPERATION, "STRING" where:
Expression-based filters select syslog messages according to defined arithmetic, Boolean or string operations. Expression-based filters use rsyslog's own scripting language called RainerScript to build complex filters. The basic syntax of expression-based filter looks as follows: if EXPRESSION then ACTION else ACTION where:
See the section called “Online Documentation” for more examples of various expression-based filters. RainerScript is the basis for rsyslog's new configuration format, see Section 23.3, “Using the New Configuration Format” 23.2.2. ActionsActions specify what is to be done with the messages filtered out by an already defined selector. The following are some of the actions you can define in your rule: Saving syslog messages to log files The majority of actions specify to which log file a syslog message is saved. This is done by specifying a file path after your already-defined selector: FILTER PATH where FILTER stands for user-specified selector and PATH is a path of a target file. For instance, the following rule is
comprised of a selector that selects all cron syslog messages and an action that saves them into the cron.* /var/log/cron.log By default, the log file is synchronized every time a syslog message is generated. Use a dash mark ( FILTER -PATH Note that you might lose information if the system terminates right after a write attempt. However, this setting can improve performance, especially if you run programs that produce very verbose log messages. Your specified file path can be either static or dynamic. Static files are represented by a fixed file path as shown in the example above. Dynamic file paths can differ according to the received message. Dynamic file paths are represented by a template and a question mark ( FILTER ?DynamicFile where DynamicFile is a name of a predefined
template that modifies output paths. You can use the dash prefix ( If the file you specified is an existing terminal or rsyslog allows you to send and receive syslog messages over the network. This feature allows you to administer syslog messages of multiple hosts on one machine. To forward syslog messages to a remote machine, use the following syntax: @( where:
Output channels are primarily used to specify the
maximum size a log file can grow to. This is very useful for log file rotation (for more information see Section 23.2.5, “Log Rotation”). An output channel is basically a collection of information about the output action. Output channels are defined by the $outchannel NAME, FILE_NAME, MAX_SIZE, ACTION where:
, ). To send messages to every user that is currently logged on, use an asterisk (* ). Executing a
program
rsyslog lets you execute a program for selected syslog messages and uses the FILTER ^EXECUTABLE; TEMPLATE Here an output of the FILTER condition is processed by a program represented by EXECUTABLE. This program can be any valid executable. Replace TEMPLATE with the name of the formatting template. Example 23.6. Executing a Program In the following example, any syslog message with any priority is selected, formatted with the . ^test-program;template When accepting messages from any host, and using the shell execute action, you may be vulnerable to command injection. An attacker may try to inject and execute commands in the program you specified to be executed in your action. To avoid any possible security threats, thoroughly consider the use of the shell execute action. Storing syslog messages in a databaseSelected syslog messages can be directly written into a database table using the database writer action. The database writer uses the following syntax: :PLUGIN:DB_HOST,DB_NAME,DB_USER,DB_PASSWORD;TEMPLATE where:
To discard your selected messages, use The discard action is mostly used to filter out messages before carrying on any further processing. It can be effective if you want to omit some repeating messages that would otherwise fill the log files. The results of discard action depend on where in the configuration file it is specified, for the best results place these actions on top of the actions list. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines. For instance, the following rule discards all messages that matches the local5.* stop In the following example, any cron syslog messages are discarded: cron.* stop With versions prior to rsyslog 7, the tilde character ( Specifying Multiple ActionsFor each selector, you are allowed to specify multiple actions. To specify multiple actions for one selector, write each action on a separate line and precede it with an ampersand (&) character: FILTER ACTION & ACTION & ACTION Specifying multiple actions improves the overall performance of the desired outcome since the specified selector has to be evaluated only once. Example 23.7. Specifying Multiple Actions In the following example, all kernel syslog messages with the critical priority ( kern.=crit user1 & ^test-program;temp & @192.168.0.1 Any action can be followed by a template that formats the message. To specify a template, suffix an action with a semicolon ( A template must be defined before it is used in an action, otherwise it is ignored. In other words, template definitions should always precede rule definitions in 23.2.3. Templates Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates. To create a template use the following syntax in template(name=”TEMPLATE_NAME” type=”string” string="text %PROPERTY% more text" [option.OPTION="on"]) where:
Generating Dynamic File NamesTemplates can be used to generate dynamic file names. By specifying a property as a part of the file path, a new file will be created for each unique property, which is a convenient way to classify syslog messages. For example, use the template(name=”DynamicFile” type=”list”) { constant(value=”/var/log/test_logs/”) property(name=”timegenerated”) constant(value”-test.log”) } Keep in mind that the . ?DynamicFile Properties Properties defined inside a template (between two percent signs ( %PROPERTY_NAME:FROM_CHAR:TO_CHAR:OPTION% where:
The following are some examples of simple properties:
Template ExamplesThis section presents a few examples of rsyslog templates. Example 23.8, “A verbose syslog message template” shows a template that formats a syslog message so that it outputs the message’s severity, facility, the time stamp of when the message was received, the host name, the message tag, the message text, and ends with a new line. Example 23.8. A verbose syslog message template template(name=”verbose” type=”list”) { property(name="syslogseverity”) property(name="syslogfacility”) property(name="timegenerated”) property(name="HOSTNAME”) property(name="syslogtag”) property(name="msg”) constant(value=”\n") } Example 23.9, “A wall message template” shows a template that resembles a traditional wall message (a
message that is send to every user that is logged in and has their Example 23.9. A wall message template template(name=”wallmsg” type=”list”) { constant(value="\r\n\7Message from syslogd@”) property(name="HOSTNAME”) constant(value=” at ") property(name="timegenerated”) constant(value=" ...\r\n ”) property(name="syslogtag”) constant(value=” “) property(name="msg”) constant(value=”\r\n”) }
Example 23.10, “A database formatted message template” shows a template that formats a syslog message so that it can be used as a database query. Notice the use of the Example 23.10. A database formatted message template template(name="dbFormat" type="list" option.sql="on") { constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") constant(value=" values ('") property(name="msg") constant(value="', ") property(name="syslogfacility") constant(value=", '") property(name="hostname") constant(value="', ") property(name="syslogpriority") constant(value=", '") property(name="timereported" dateFormat="mysql") constant(value="', '") property(name="timegenerated" dateFormat="mysql") constant(value="', ") property(name="iut") constant(value=", '") property(name="syslogtag") constant(value="')") } rsyslog also contains a set of predefined templates identified by the
A special format used for troubleshooting property problems. template(name=”RSYSLOG_DebugFormat” type=”string” string="Debug line with all properties:\nFROMHOST: '%FROMHOST%', fromhost-ip: '%fromhost-ip%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nescaped msg: '%msg:::drop-cc%'\nrawmsg: '%rawmsg%'\n\n") RSYSLOG_SyslogProtocol23Format The format specified in IETF’s internet-draft ietf-syslog-protocol-23, which is assumed to become the new syslog standard RFC. template(name=”RSYSLOG_SyslogProtocol23Format” type=”string” string="%PRI%1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n ") RSYSLOG_FileFormat A modern-style logfile format similar to TraditionalFileFormat, but with high-precision time stamps and time zone information. template(name="RSYSLOG_FileFormat" type="list") { property(name="timestamp" dateFormat="rfc3339") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag") property(name="msg" spifno1stsp="on" ) property(name="msg" droplastlf="on" ) constant(value="\n") } RSYSLOG_TraditionalFileFormat The older default log file format with low-precision time stamps. template(name="RSYSLOG_TraditionalFileFormat" type="list") { property(name="timestamp") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag") property(name="msg" spifno1stsp="on" ) property(name="msg" droplastlf="on" ) constant(value="\n") } RSYSLOG_ForwardFormat A forwarding format with high-precision time stamps and time zone information. template(name="ForwardFormat" type="list") { constant(value="<") property(name="pri") constant(value=">") property(name="timestamp" dateFormat="rfc3339") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag" position.from="1" position.to="32") property(name="msg" spifno1stsp="on" ) property(name="msg") } RSYSLOG_TraditionalForwardFormat The traditional forwarding format with low-precision time stamps. template(name="TraditionalForwardFormat" type="list") { constant(value="<") property(name="pri") constant(value=">") property(name="timestamp") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag" position.from="1" position.to="32") property(name="msg" spifno1stsp="on" ) property(name="msg") } 23.2.4. Global Directives Global directives are configuration options that apply to the global(localHostname=”machineXY”) The default size defined for this directive (10,000 messages) can be overridden by specifying a different value (as shown in the example above). You can define multiple directives in your 23.2.5. Log Rotation The
following is a sample # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # uncomment this if you want your log files compressed compress All of the lines in the sample configuration file define global options that apply to every log file. In our example, log files are rotated weekly, rotated log files are kept for four weeks, and all rotated log files are compressed by gzip into the You may define configuration options
for a specific log file and place it under the global options. However, it is advisable to create a separate configuration file for any specific log file in the The following is an example of a configuration file placed in the /var/log/messages { rotate 5 weekly postrotate /usr/bin/killall -HUP syslogd endscript } The configuration options in this file are specific for the The following is a list of some of the directives you can specify in your logrotate configuration file:
For the full list of directives and various configuration options, see the 23.2.6. Increasing the Limit of Open Files Under certain circumstances the To increase the limit of open files in the Create the [Service] LimitNOFILE=16384 23.3. Using the New Configuration Format In
rsyslog version 7, installed by default for Red Hat Enterprise Linux 7 in the rsyslog package, a new configuration syntax is introduced. This new configuration format aims to be more powerful, more intuitive, and to prevent common mistakes by not permitting certain invalid constructs. The syntax enhancement is enabled by the new configuration processor that relies on RainerScript. The legacy format is still
fully supported and it is used by default in the RainerScript is a scripting language designed for processing network events and configuring event processors such as rsyslog. RainerScript was first used to define expression-based filters, see
Example 23.3, “Expression-based Filters”. The version of RainerScript in rsyslog version 7 implements the Compare the configuration written with legacy-style parameters: $InputFileName /tmp/inputfile $InputFileTag tag1: $InputFileStateFile inputfile-state $InputRunFileMonitor and the same configuration with the use of the new format statement: input(type="imfile" file="/tmp/inputfile" tag="tag1:" statefile="inputfile-state") This significantly reduces the number of parameters used in configuration, improves readability, and also provides higher execution speed. For more information on RainerScript statements and parameters see the section called “Online Documentation”. 23.3.1. Rulesets Leaving special directives aside, rsyslog handles messages as defined by rules that consist of a filter condition and an action to be performed if the condition is true. With a traditionally written However, rules can be grouped into sequences called rulesets. With rulesets, you can limit the effect of certain rules only to selected inputs or enhance the performance of rsyslog by defining a distinct set of actions bound to a specific input. In other words, filter conditions that will be inevitably
evaluated as false for certain types of messages can be skipped. The legacy ruleset definition in $RuleSet rulesetname rule rule2 The rule ends when another rule is defined, or the default ruleset is called as follows: $RuleSet RSYSLOG_DefaultRuleset With the new configuration format in rsyslog 7, the ruleset(name="rulesetname") { rule rule2 call rulesetname2 … } Replace rulesetname with an
identifier for your ruleset. The ruleset name cannot start with After creating a ruleset, you need to specify what input it will apply to: input(type="input_type" port="port_num" ruleset="rulesetname"); Here you can identify an input message by input_type, which is an input module that gathered the message, or by port_num – the port number. Other parameters such as file or tag can be specified for You can also use the legacy format to define rulesets, for more information see the section called “Online Documentation”.
Example 23.11. Using rulesets The following rulesets ensure different handling of remote messages coming from different ports. Add the following into ruleset(name="remote-6514") { action(type="omfile" file="/var/log/remote-6514") } ruleset(name="remote-601") { cron.* action(type="omfile" file="/var/log/remote-601-cron") mail.* action(type="omfile" file="/var/log/remote-601-mail") } input(type="imtcp" port="6514" ruleset="remote-6514"); input(type="imtcp" port="601" ruleset="remote-601"); Rulesets shown in the above example define log destinations for the remote input from two ports, in case of port 23.3.2. Compatibility with sysklogd The compatibility mode specified via the For more information on various 23.4. Working with Queues in RsyslogQueues are used to pass content, mostly syslog messages, between components of rsyslog. With queues, rsyslog is capable of processing multiple messages simultaneously and to apply several actions to a single message at once. The data flow inside rsyslog can be illustrated as follows: Figure 23.1. Message Flow in Rsyslog Whenever rsyslog receives a message, it passes this message to the preprocessor and then places it into the main message queue. Messages wait there to be dequeued and passed to the rule processor. The rule processor is a parsing and filtering engine. Here, the rules defined in Only one queue per action is possible. Depending on configuration, the messages can be sent right to the action processor without action queuing. This is the behavior of direct queues (see below). In case the output action fails, the action processor notifies the action queue, which then takes an unprocessed element back and after some time interval, the action is attempted again. To sum up, there are two positions where queues stand in rsyslog: either in front of the rule processor as a single main message queue or in front of various types of output actions as action queues. Queues provide two main advantages that both lead to increased performance of message processing:
Apart from this, queues can be configured with several directives to provide optimal performance for your system. These configuration options are covered in the following sections. If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until
it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing 23.4.1. Defining Queues Based on where the
messages are stored, there are several types of queues: direct, in-memory, disk, and disk-assisted in-memory queues that are most widely used. You can choose one of these types for the main message queue and also for action queues. Add the following into object(queue.type=”queue_type”) By adding this you can apply the setting for:
Replace queue_type with one of The default setting for a main message queue is the FixedArray queue with a limit of 10,000 messages. Action queues are by default set as Direct queues. Direct QueuesFor many simple operations, such as when writing output to a local file, building a queue in front of an action is not needed. To avoid queuing, use: object(queue.type=”Direct”) Replace object with Disk Queues Disk
queues store messages strictly on a hard drive, which makes them highly reliable but also the slowest of all possible queuing modes. This mode can be used to prevent the loss of highly important log data. However, disk queues are not recommended in most use cases. To set a disk queue, type the following into object(queue.type=”Disk”) Replace object with object(queue.size=”size”) where size represents the specified size of disk queue part. The defined size limit is not restrictive, rsyslog always writes one complete queue entry, even if it violates the size limit. Each part of a disk queue matches with an individual file. The naming directive for these files looks as follows: object(queue.filename=”name”) This sets a name prefix for the file followed by a 7-digit number starting at one and incremented for each file. object(queue.maxfilesize=”size”) Disk queues are written in parts, with a default size 1 MB. Specify size to use a different value. In-memory Queues With in-memory queue, the enqueued messages are held in memory which makes the process very fast. The queued data is lost if the computer is power cycled or shut down.
However, you can use the
In general, use LinkedList queues when in doubt. Compared to FixedArray, it consumes less memory and lowers the processing overhead. Use the following syntax to configure in-memory queues: object(queue.type=”LinkedList”) object(queue.type=”FixedArray”) Replace object with Disk-Assisted In-memory Queues Both disk and in-memory queues have their advantages and rsyslog lets you combine them in disk-assisted in-memory
queues. To do so, configure a normal in-memory queue and then add the The disk queue is activated if the in-memory queue is full or needs to persist after shutdown. With a disk-assisted queue, you can set both disk-specific and in-memory specific configuration parameters. This type of queue is probably the most commonly used, it is especially useful for potentially long-running and unreliable actions. To specify the functioning of a disk-assisted in-memory queue, use the so-called watermarks: object(queue.highwatermark=”number”) object(queue.lowwatermark=”number”) Replace object with Example 23.12. Reliable Forwarding of Log Messages to a Server Rsyslog is often used to maintain a centralized logging system, where log messages are forwarded to a server over the network. To avoid
message loss when the server is not available, it is advisable to configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues are not configurable for connections using the Forwarding To a Single Server Suppose the task is to forward log messages from the system to a server with host name example.com, and to configure an action queue to buffer the messages in case of a server outage. To do so, perform the following steps:
Forwarding To Multiple Servers The process of forwarding log messages to multiple servers is similar to the previous procedure:
23.4.2. Creating a New Directory for rsyslog Log Files Rsyslog runs as the Creating a New Working Directory
23.4.3. Managing QueuesAll types of queues can be further configured to match your requirements. You can use several directives to modify both action queues and the main message queue. Currently, there are more than 20 queue parameters available, see the section called “Online Documentation”. Some of these settings are used commonly, others, such as worker thread management, provide closer control over the queue behavior and are reserved for advanced users. With advanced settings, you can optimize rsyslog's performance, schedule queuing, or modify the behavior of a queue on system shutdown. Limiting Queue SizeYou can limit the number of messages that queue can contain with the following setting: object(queue.highwatermark=”number”) Replace object with Disk assisted queues are unlimited by default and cannot be restricted with this directive, but you can reserve them physical disk space in bytes with the following settings: object(queue.maxdiskspace=”number”) Replace
object with Discarding MessagesWhen a queue reaches a certain number of messages, you can discard less important messages in order to save space in the queue for entries of higher priority. The threshold that launches the discarding process can be set with the so-called discard mark: object(queue.discardmark=”number”) Replace object with object(queue.discardseverity=”number”) Replace number with one of the following numbers for respective priorities:
Using TimeframesYou can configure rsyslog to process queues during a specific time period. With this option you can, for example, transfer some processing into off-peak hours. To define a time frame, use the following syntax: object(queue.dequeuetimebegin=”hour”) object(queue.dequeuetimeend=”hour”) With hour you can specify hours that bound your time frame. Use the 24-hour format without minutes. Configuring Worker ThreadsA worker thread performs a specified action on the enqueued message. For example, in the main message queue, a worker task is to apply filter logic to each incoming message and enqueue them to the relevant action queues. When a message arrives, a worker thread is started automatically. When the number of messages reaches a certain number, another worker thread is turned on. To specify this number, use: object(queue.workerthreadminimummessages=”number”) Replace number with a number of messages that will trigger a supplemental worker thread. For example, with number set to 100, a new worker thread is started when more than 100 messages arrive. When more than 200 messages arrive, the third worker thread starts and so on. However, too many working threads running in parallel becomes ineffective, so you can limit the maximum number of them by using: object(queue.workerthreads=”number”) where number stands for a maximum number of working threads that can run in parallel. For the main message queue, the default limit is 1 thread. Once a working thread has been started, it keeps running until an inactivity timeout appears. To set the length of timeout, type: object(queue.timeoutworkerthreadshutdown=”time”) Replace time with the duration set in milliseconds. Specifies time without new messages after which the worker thread will be closed. Default setting is one minute. Batch DequeuingTo increase performance, you can configure rsyslog to dequeue multiple messages at once. To set the upper limit for such dequeueing, use: $object(queue.DequeueBatchSize= ”number”) Replace number with the maximum number of messages that can be dequeued at once. Note that a higher setting combined with a higher number of permitted working threads results in greater memory consumption. Terminating QueuesWhen terminating a queue that still contains messages, you can try to minimize the data loss by specifying a time interval for worker threads to finish the queue processing: object(queue.timeoutshutdown=”time”) Specify time in milliseconds. If after that period there are still some enqueued messages, workers finish the current data element and then terminate. Unprocessed messages are therefore lost. Another time interval can be set for workers to finish the final element: object(queue.timeoutactioncompletion=”time”) In case this timeout expires, any remaining workers are shut down. To save data at shutdown, use: object(queue.saveonshutdown=”on”) If set, all queue elements are saved to disk before rsyslog terminates. 23.4.4. Using the New Syntax for rsyslog queues In the new syntax available in rsyslog 7, queues are defined inside the action(type="action_type "queue.size="queue_size" queue.type="queue_type" queue.filename="file_name" Replace action_type with the name of
the module that is to perform the action and replace queue_size with a maximum number of messages the queue can contain. For queue_type, choose Example 23.13. Defining an Action Queue To configure the output action with an asynchronous linked-list based action queue which can hold a maximum of 10,000 messages, enter a command as follows: action(type="omfile" queue.size="10000" queue.type="linkedlist" queue.filename="logfile") The rsyslog 7 syntax for a direct action queues is as follows: . action(type="omfile" file="/var/lib/rsyslog/log_file ) The rsyslog 7 syntax for an action queue with multiple parameters can be written as follows: . action(type="omfile" queue.filename="log_file" queue.type="linkedlist" queue.size="10000" ) The default work directory, or the last work directory to be set, will be used. If required to use a different work directory, add a line as follows before the action queue: global(workDirectory="/directory") Example 23.14. Forwarding To a Single Server Using the New Syntax The following example is based on the procedure
Forwarding To a Single Server in order to show the difference between the traditional sysntax and the rsyslog 7 syntax. The Use the following configuration in . action(type="omfwd"
queue.type="linkedlist"
queue.filename="example_fwd"
action.resumeRetryCount="-1"
queue.saveOnShutdown="on"
target="example.com" port="6514" protocol="tcp"
) Where:
23.5. Configuring rsyslog on a Logging Server The The ~]# yum install rsyslog The default protocol and port for syslog traffic is Other ports are sometimes used in examples, however SELinux is only configured to allow sending and receiving on the following ports by default: ~]# semanage port -l | grep syslog syslog_tls_port_t tcp 6514, 10514 syslog_tls_port_t udp 6514, 10514 syslogd_port_t tcp 601, 20514 syslogd_port_t udp 514, 601, 20514 The ~]# yum install policycoreutils-python In addition, by default the SELinux type for ~]# semanage port -l | grep 514
output omitted
rsh_port_t tcp 514
syslogd_port_t tcp 6514, 601
syslogd_port_t udp 514, 6514, 601 For more information on SELinux, see Red Hat Enterprise Linux 7 SELinux User’s and Administrator’s Guide. Perform the steps in the following procedures on the system that you intend to use as your logging server. All steps in these procedure must be made as the Configure SELinux to Permit rsyslog Traffic on a Port If required to use a new port for
See the Configuring firewalld Configure ~]# firewall-cmd --zone=zone --add-port=10514/tcp success Where zone is the zone of the interface to use. Note that these changes will not persist after the next system start. To make permanent changes to the firewall, repeat the commands adding the
Configuring rsyslog to Receive and Sort Remote Log Messages
Your log server is now configured to receive and store log files from the other systems in your environment. 23.5.1. Using The New Template Syntax on a Logging ServerRsyslog 7 has a number of different templates styles. The string template most closely resembles the legacy format. Reproducing the templates from the example above using the string format would look as follows: template(name="TmplAuthpriv" type="string" string="/var/log/remote/auth/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) template(name="TmplMsg" type="string" string="/var/log/remote/msg/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) These templates can also be written in the list format as follows: template(name="TmplAuthpriv" type="list") { constant(value="/var/log/remote/auth/") property(name="hostname") constant(value="/") property(name="programname" SecurePath="replace") constant(value=".log") } template(name="TmplMsg" type="list") { constant(value="/var/log/remote/msg/") property(name="hostname") constant(value="/") property(name="programname" SecurePath="replace") constant(value=".log") } This template text format might be easier to read for those new to rsyslog and therefore can be easier to adapt as requirements change. To complete the change to the new syntax, we need to reproduce the module load command, add a rule set, and then bind the rule set to the protocol, port, and ruleset: module(load="imtcp") ruleset(name="remote1"){ authpriv.* action(type="omfile" DynaFile="TmplAuthpriv") *.info;mail.none;authpriv.none;cron.none action(type="omfile" DynaFile="TmplMsg") } input(type="imtcp" port="10514" ruleset="remote1") 23.6. Using Rsyslog ModulesDue to its modular design, rsyslog offers a variety of modules which provide additional functionality. Note that modules can be written by third parties. Most modules provide additional inputs (see Input Modules below) or outputs (see Output Modules below). Other modules provide special functionality specific to each module. The modules may provide additional configuration directives that become available after a module is loaded. To load a module, use the following syntax: module(load=”MODULE”) where MODULE represents your desired module. For example, if you want to load the Text File Input Module ( module(load=”imfile”) rsyslog offers a number of modules which are split into the following main categories:
A comprehensive list of all available modules and their detailed description can be found at http://www.rsyslog.com/doc/rsyslog_conf_modules.html. Note that when rsyslog loads any modules, it provides them with access to some of its functions and data. This poses a possible security threat. To minimize security risks, use trustworthy modules only. 23.6.1. Importing Text Files The Text File Input Module, abbreviated as module(load=”imfile” PollingInterval=”int”) It is sufficient to load To identify the text files to import, use the following syntax in # File 1 input(type="imfile" File="path_to_file" Tag="tag:" Severity="severity" Facility="facility") # File 2 input(type="imfile" File="path_to_file2") ... Settings required to specify an input text file:
Apart from the required directives, there are several other settings that can be applied on the text input. Set the severity of imported messages by replacing severity with an appropriate keyword. Replace facility with a keyword to define the subsystem that produced the message. The keywords for severity and facility are the same as those used in facility/priority-based filters, see Section 23.2.1, “Filters”. Example 23.15. Importing Text Files The Apache HTTP server creates log files in text format. To apply the processing capabilities of rsyslog
to apache error messages, first use the module(load=”imfile”) input(type="imfile" File="/var/log/httpd/error_log" Tag="apache-error:") 23.6.2. Exporting Messages to a Database Processing of log data can be faster and more convenient when performed in a database rather than with text files. Based on the type of DBMS used, choose from various output modules such as Example 23.16. Exporting Rsyslog Messages to a Database To store the rsyslog messages in a MySQL database, add the following into module(load=”ommysql”)
. action(type”ommysql”
server=”database-server”
db=”database-name”
uid=”database-userid”
pwd=”database-password”
serverport=”1234”) First, the output module is loaded, then the communication port is specified. Additional information, such as name of the server and the database, and authentication data, is specified on the last line of the above example. 23.6.3. Enabling Encrypted TransportConfidentiality and integrity in network transmissions can be provided by either the TLS or GSSAPI encryption protocol. Transport Layer Security (TLS) is a cryptographic protocol designed to provide communication security over the network. When using TLS, rsyslog messages are encrypted before sending, and mutual authentication exists between the sender and receiver. For configuring TLS, see the section called “Configuring Encrypted Message Transfer with TLS”. Generic Security Service API (GSSAPI) is an application programming interface for programs to access security services. To use it in connection with rsyslog you must have a functioning Kerberos environment. For configuring GSSAPI, see the section called “Configuring Encrypted Message Transfer with GSSAPI”. Configuring Encrypted Message Transfer with TLSTo use encrypted transport through TLS, you need to configure both the server and the client.
Configuring Encrypted Message Transfer with GSSAPIIn rsyslog, interaction with GSSAPI is provided by the imgssapi module. To turn on the GSSAPI transfer mode:
The Example 23.17. Using GSSAPI The following configuration enables a GSS server on the port 1514 that also permits to receive plain tcp syslog messages on the same port. $ModLoad imgssapi $InputGSSServerPermitPlainTCP on $InputGSSServerRun 1514 23.6.4. Using RELPReliable Event Logging Protocol (RELP) is a networking protocol for data logging in computer networks. It is designed to provide reliable delivery of event messages, which makes it useful in environments where message loss is not acceptable. Configuring RELP To configure RELP, you need to configure both the server and the client using the
Configuring RELP with TLS To configure RELP with TLS, you need to configure authentication. Then, you need to configure both the server and the client using the
23.7. Interaction of Rsyslog and JournalAs mentioned above, Rsyslog and Journal, the two logging applications present on your system, have several distinctive features that make them suitable for specific use cases. In many situations it is useful to combine their capabilities, for example to create structured messages and store them in a file database (see Section 23.8, “Structured Logging with Rsyslog”). A communication interface needed for this cooperation is provided by input and output modules on the side of Rsyslog and by the Journal's communication socket. By default, As an alternative, configure module(load="imuxsock" SysSock.Use="on" SysSock.Name="/run/systemd/journal/syslog") You can also output messages from Rsyslog to Journal
with the module(load="omjournal") action(type="omjournal") For instance, the following configuration forwards all received messages on tcp port 10514 to the Journal: module(load="imtcp") module(load="omjournal") ruleset(name="remote") { action(type="omjournal") } input(type="imtcp" port="10514" ruleset="remote") 23.8. Structured Logging with RsyslogOn systems that produce large amounts of log data, it can be convenient to maintain log messages in a structured format. With structured messages, it is easier to search for particular information, to produce statistics and to cope with changes and inconsistencies in message structure. Rsyslog uses the JSON (JavaScript Object Notation) format to provide structure for log messages. Compare the following unstructured log message: Oct 25 10:20:37 localhost anacron[1395]: Jobs will be executed sequentially with a structured one: {"timestamp":"2013-10-25T10:20:37", "host":"localhost", "program":"anacron", "pid":"1395", "msg":"Jobs will be executed sequentially"} Searching structured data with use of key-value pairs is faster and more precise than searching text files with regular expressions. The structure also lets you to search for the same entry in messages produced by various applications. Also, JSON files can be stored in a document database such as MongoDB, which provides additional performance and analysis capabilities. On the other hand, a structured message requires more disk space than the unstructured one. In rsyslog, log messages with meta data are pulled
from Journal with use of the The Lumberjack project aims to add structured logging to rsyslog in a backward-compatible way. To identify a structured message, Lumberjack specifies the @cee: string that prepends the actual JSON structure. Also, Lumberjack defines the list of standard field names that should be used for entities in the JSON string. For more information on Lumberjack, see the section called “Online Documentation”. The following is an example of a lumberjack-formatted message: @cee: {"pid":17055, "uid":1000, "gid":1000, "appname":"logger", "msg":"Message text."} To build this structure inside Rsyslog, a template is used, see
Section 23.8.2, “Filtering Structured Messages”. Applications and servers can employ the 23.8.1. Importing Data from Journal The To import data from Journal to Rsyslog, use the following configuration in module(load=”imjournal” PersistStateInterval=”number_of_messages” StateFile=”path” ratelimit.interval=”seconds” ratelimit.burst=”burst_number” IgnorePreviousMessages=”off/on”)
You can use module(load”imjournal”) module(load”imuxsock” SysSock.Use=”off” Socket="/run/systemd/journal/syslog") You can translate all data and meta data stored by Journal into structured messages. Some of these meta data entries are listed in
Example 23.19, “Verbose journalctl Output”, for a complete list of journal fields see the 23.8.2. Filtering Structured MessagesTo create a lumberjack-formatted message that is required by rsyslog's parsing module, use the following template: template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n") This template prepends the ($!hostname == "hostname" && $!UID== "UID") 23.8.3. Parsing JSON The These messages can come from Journal or from other input sources, and must be formatted in a
way defined by the Lumberjack project. These messages are identified by the presence of the @cee: string. Then, To parse lumberjack-formatted JSON messages with module(load”mmjsonparse”)
. :mmjsonparse: In this example, the 23.8.4. Storing Messages in the MongoDBRsyslog supports storing JSON logs in the MongoDB document database through the ommongodb output module. To forward log messages into MongoDB, use the following syntax in the module(load”ommongodb”)
. action(type="ommongodb" server="DB_server" serverport="port" db="DB_name" collection="collection_name" uid="UID" pwd="password")
You can shape the form of the final database output with use of templates. By default, rsyslog uses a template based on standard lumberjack field names. 23.9. Debugging Rsyslog To run
With this command, export RSYSLOG_DEBUGLOG="path"
export RSYSLOG_DEBUG="Debug" Replace path with a desired location for the
file where the debugging information will be logged. For a complete list of options available for the RSYSLOG_DEBUG variable, see the related section in the To check if syntax used in the
Where 23.10. Using the Journal The Journal is a component of systemd that is responsible for viewing and management of log files. It can be used in parallel, or in place of a traditional syslog daemon, such as Logging data is collected, stored, and processed by the Journal’s 23.10.1. Viewing Log Files To access the journal logs, use the journalctl tool. For a basic view of the logs type as
An output of this command is a list of all log files
generated on the system including messages generated by system components and by users. The structure of this output is similar to one used in
Example 23.18. Example Output of journalctl The following is an example output provided by the journalctl tool. When called without parameters, the listed entries begin with a time stamp, then the host name and application that performed the operation is mentioned followed by the actual message. This example shows the first three entries in the journal log: # journalctl -- Logs begin at Thu 2013-08-01 15:42:12 CEST, end at Thu 2013-08-01 15:48:48 CEST. -- Aug 01 15:42:12 localhost systemd-journal[54]: Allowing runtime journal files to grow to 49.7M. Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpuset Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpu [...] In many cases, only the latest entries in the journal log are relevant. The simplest way to reduce
Replace Number with the number of lines to be shown. When no number is specified, The
Replace form with a keyword specifying a desired form of output. There are several options, such as Example 23.19. Verbose journalctl Output To view full meta data about all entries, type: # journalctl -o verbose [...] Fri 2013-08-02 14:41:22 CEST [s=e1021ca1b81e4fc688fad6a3ea21d35b;i=55c;b=78c81449c920439da57da7bd5c56a770;m=27cc _BOOT_ID=78c81449c920439da57da7bd5c56a770 PRIORITY=5 SYSLOG_FACILITY=3 _TRANSPORT=syslog _MACHINE_ID=69d27b356a94476da859461d3a3bc6fd _HOSTNAME=localhost.localdomain _PID=562 _COMM=dbus-daemon _EXE=/usr/bin/dbus-daemon _CMDLINE=/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation _SYSTEMD_CGROUP=/system/dbus.service _SYSTEMD_UNIT=dbus.service SYSLOG_IDENTIFIER=dbus SYSLOG_PID=562 _UID=81 _GID=81 _SELINUX_CONTEXT=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 MESSAGE=[system] Successfully activated service 'net.reactivated.Fprint' _SOURCE_REALTIME_TIMESTAMP=1375447282839181 [...] This example lists fields that identify a single log entry. These meta data can be used for message filtering as shown in
the section called “Advanced Filtering”. For a complete description of all possible fields see the 23.10.2. Access Control By default,
Journal users without
Here, replace username with a name of the user to be added to the adm group. This user then receives the same output of the 23.10.3. Using The Live View When called without parameters,
This command returns a list of the ten most current log lines. The journalctl utility then stays running and waits for new changes to show them immediately. 23.10.4. Filtering Messages The output of the Filtering by PriorityLog messages are often used to track erroneous behavior on the system. To view only entries with a selected or higher priority, use the following syntax:
Here, replace priority with one of the following keywords (or with a number): Example 23.20. Filtering by Priority To view only entries with error or higher priority, use:
Filtering by TimeTo view log entries only from the current boot, type:
If you reboot your system just occasionally, the
With Example 23.21. Filtering by Time and Priority Filtering options can be combined to reduce the set of results according to specific requests. For example, to view the warning or higher priority messages from a certain point in time, type:
Advanced Filtering
Example 23.19, “Verbose journalctl Output” lists a set of fields that specify a log entry and can all be used for filtering. For a complete description of meta data that To view a list of unique values that occur in a specified field, use the following syntax:
Replace fieldname with a name of a field you are interested in. To show only log entries that fit a specific condition, use the following syntax:
Replace fieldname with a name of a field and value with a specific value contained in that field. As a result, only lines that match this condition are returned. As the number of meta data fields stored by
and press the Tab key two times. This shows a list of available field names. Tab completion based on context works on field names, so you can type a distinctive set of letters from a field name and then press Tab to complete the name automatically. Similarly, you can list unique values from a field. Type:
and press Tab two times. This serves as an alternative to You can specify multiple values for one field:
Specifying two matches for the same field results in a logical Also, you can specify multiple field-value pairs to further reduce the output set:
If two matches for different field names are specified, they will be combined with a logical With use of the
+ symbol, you can set a logical journalctl fieldname1=value + fieldname2=value ... This command returns entries that match at least one of the conditions, not only those that match both of them. Example 23.22. Advanced filtering To display entries created by journalctl Since there are two values set for
the You can apply the aforementioned filtering also in the live-view mode to keep track of the latest changes in the selected group of log entries:
23.10.5. Enabling Persistent Storage By default,
Journal stores log files only in memory or a small ring-buffer in the Enabled persistent storage has the following advantages
Persistent storage has also certain disadvantages:
To enable persistent storage for Journal, create the journal directory manually as shown in the following example. As
Then, restart
23.11. Managing Log Files in a Graphical EnvironmentAs an alternative to the aforementioned command-line utilities, Red Hat Enterprise Linux 7 provides an accessible GUI for managing log messages. 23.11.1. Viewing Log Files Most log files are stored in plain text format. You can view them with any text editor such as
To view system log files in an interactive, real-time application, use the System Log. In order to use the System Log, first ensure the gnome-system-log package is
installed on your system by running, as ~]# yum install gnome-system-log For more information on installing packages with Yum, see Section 9.2.4, “Installing Packages”. After you have installed the gnome-system-log package, open the System Log by clicking → → , or type the following command at a shell prompt: ~]$ The application only displays log files that exist; thus, the list might differ from the one shown in Figure 23.2, “System Log”. Figure 23.2. System Log The System Log application lets you filter any existing log file. Click on the button marked with the gear symbol to view the menu, select
menu:[ Figure 23.3. System Log - Filters Adding or editing a filter lets you define its parameters as is shown in Figure 23.4, “System Log - defining a filter”. Figure 23.4. System Log - defining a filter When defining a filter, the following parameters can be edited:
When you have at least one filter defined, it can be selected from the Figure 23.5. System Log - enabling a filter When you select the 23.11.2. Adding a Log File To add a log file you want to view in the list, select → → → . This will display the Figure 23.6. System Log - adding a log file Click on the Open button to open the file. The file is immediately added to the viewing list where you can select it and view its contents. The System Log also allows you to open log files zipped
in the 23.11.3. Monitoring Log Files System Log monitors all opened logs by default. If a new line is added to a monitored log file, the log name appears in bold in the log list. If the log file is selected or displayed, the new lines appear in bold at the bottom of the log file.
Figure 23.7, “System Log - new log alert” illustrates a new alert in the Figure 23.7. System Log - new log alert 23.12. Additional Resources For
more information on how to configure the Installed Documentation
Installable Documentation
~]# yum install rsyslog-doc Online DocumentationThe rsyslog home page offers additional documentation, configuration examples, and video tutorials. Make sure to consult the documents relevant to the version you are using:
See Also
What is the default number of workspaces in CentOS 7?By default, CentOS configures four workspaces.
When you delete files from a folder where do they go choose all that apply?The recycle bin is a holding place for deleted files. To permanently delete a file, you can delete it again from the Recycle Bin, or you can empty the Recycle Bin. It is recommended you empty the Recycle Bin every once in a while to free up space.
Which of the following is not located on the taskbar?Answer: Start Button. Explanation: Follow me for more help.
What was developed to simplify the connection of peripheral devices?USB was designed to standardize the connection of peripherals to personal computers, both to communicate with and to supply electric power. It has largely replaced interfaces such as serial ports and parallel ports, and has become commonplace on a wide range of devices.
|