What is the logging severity level?

 This topic discusses the syslog reporting feature of the Delphix Engine, along with severity levels.

Syslog is a widely used standard for message logging. It permits the separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Delphix makes use of syslog as one of the standard mechanisms, along with SNMP and email, to distribute important user and system events, such as alerts, faults, and audits. In the case of Delphix, each Delphix Engine acts as a syslog client which propagates the events to a centralized syslog server.

Every syslog message is attached to a severity level. As the name suggests, the severity level describes the severity of the event in question. 

Audit records are Informational syslog messages.  If you would like to forward Audit records, choose Severity Level Informational.

Every syslog message is attached to a severity level number. Delphix defines the severity of syslog messages in accordance with RFC 3164. There are eight severity levels available, as follows:

Numerical         Severity
Code

           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages

When setting up the syslog settings for your Delphix Engine, you have the ability to choose what alerts to report. The severity levels above are available for users to select. Once you select a severity level, the Delphix Engine will send messages of the same or higher severity (i.e., the same or lower number) to your syslog server. Therefore, there is no reason to select more than one severity. For example, if the "Notice" severity level is selected, all events less severe than Notice (Informational and Debug) will not be reported. If you want all events to be reported via syslog, the Debug severity level should be chosen.

Log levels are labels that indicate the severity or urgency of a log entry. Their primary purpose is to separate messages that are merely informational (meaning that the system is working normally) from those that describe a problem or potential problem, such as when recurring errors are detected in the system. Log levels also provide a way to dynamically control your application's volume of log output (more on this later).

In this article, we will discuss the following concepts that should help you get a handle on what log levels are and how to use them to log more effectively.

The history of log levels

Syslog, a logging solution initially developed for the Sendmail project, first introduced the concept of log levels in the 1980s. It came with severity levels that are attached to each log entry to describe the severity of the event in question:

1. Emergency (emerg): system is unusable.
2. Alert (alert): immediate action required.
3. Critical (crit): critical conditions.
4. Error (error): error conditions.
5. Warning (warn): warning conditions.
6. Notice (notice): normal but significant conditions.
7. Informational (info): informational messages.
8. Debug (debug): messages helpful for debugging.

In the following years, Syslog was adopted by various software applications and eventually became a standard for message logging on Unix-like systems. Its severity levels were also adapted and refined by various application logging frameworks such as log4net and log4j, evolving into the various log levels that are commonplace today.

Common log levels and their use cases

The log levels available to you will vary depending on the programming language, framework, or service in use. Still, most will include some or all of the following levels: FATAL, ERROR, alert0, alert1, alert2, and alert3. You can usually override the defaults in your logging framework of choice with custom log levels, but we recommend sticking to the ones discussed below. They are arranged in decreasing order of urgency:

FATAL

The FATAL log level annotates messages with the greatest severity. It usually means that something critical is broken, and the application cannot continue to do any more useful work without the intervention of an engineer. Typically, such entries are logged before the application is shut down (with exit code alert5) to prevent further data corruption. If you use a log management service, you can configure it such that you get instant alerts when such entries are logged so that someone can react to them as quickly as possible.

Examples of situations that may be logged as FATAL errors include the following:

• Crucial configuration information is missing without fallback defaults.
• Unable to connect to a service crucial to the application's primary function (such as the database).
• Running out of disk space on the server.

ERROR

The ERROR log level is used to represent error conditions in an application that prevent a specific operation from running, but the application itself can continue working even if it is at a reduced level of functionality or performance. Generally, ERROR logs should be investigated as soon as possible but they don't carry the same urgency as FATAL messages since the application can continue working.

The occurrence of an error condition in the application does not necessarily mean that it should be logged at the ERROR level. For example, if an exception is expected behavior and does not indicate degradation in application functionality or performance, it can be logged as alert1. Also, errors with a possibility of recovery (such as network connectivity errors) can be labeled as alert1 if an automatic recovery strategy is in place (e.g retries). Such conditions can be promoted to the ERROR level if recovery isn't possible after a predetermined time.

Logging significant error conditions is also useful for generating metrics such as Mean Time Between Failures (MTBF) which can be used to assess the quality of the application or to compare different systems or designs. Examples of situations that are typically logged at the ERROR level include the following:

• A persistent connection failure to some external resource (after automated recovery attempts have failed).
• Failure to create or update a resource in the system.
• An unexpected error (e.g failed to decode a JSON object).

WARN

Messages logged at the alert0 level typically indicate that something unexpected happened, but the application can recover and continue to function normally. It is mainly used to draw attention to situations that should be addressed soon before they pose a problem for the application.

Events that may be logged at the alert0 level include the following:

• The disk usage on the server is above a configured threshold.
• Memory usage is above a configured threshold.
• The application is taking longer than usual to complete some important tasks (degraded performance).

INFO

alert1-level messages indicate events in the system that are significant to the business purpose of the application. Such events are logged to show that the system is operating normally. For example, a service was started or stopped, some resource was created, accessed, updated, or deleted in the database, and so on. Production systems typically default to logging at this level so that a summary of the application's normal behavior is visible to anyone reading the logs.

Other events that are typically logged at the alert1 level include the following:

• The state of an operation has changed (e.g from "PENDING" to "IN PROGRESS").
• The application is listening on a specific port.
• A scheduled job was completed successfully.

DEBUG

The alert2 level is used for logging messages that help developers find out what went wrong during a debugging session. While the specifics of what messages to log at the alert2 level is dependent on your application, you generally want to include detailed information that can help developers troubleshoot an issue quickly. This can include variable state in the surrounding scope, or relevant error codes. Unlike alert3 (below), alert2 level logging can be turned on in production without making the application unusable, but it should not be left on indefinitely to ensure optimal performance of the system.

TRACE

The alert3 level is used for tracing the path of code execution in a program. For example, you may use it to trace the processing of a incoming request or an algorithm's steps to solve a problem. Generally, alert3 is used for showing the flow of the program, and to provide a detailed breakdown of the sequence of events that led to a crash, a silent failure, an error, or some other event logged at a different level. Concrete examples of messages that should be logged at the alert3 level include the following:

• Entered or exited a function or method, perhaps with the processing duration.
• Calculation x + y produced output z.
• Starting or ending an operation and any intermediate state changes.

As you can see, the information logged at this level generally tries to capture every possible detail about the program's execution. Therefore, alert3 logging should only be enabled for short periods due to the significant performance degradation that it often causes. You will typically enable it only in development and testing environments.

Log levels are the primary way to control your application's volume of log entries. Once you select your default level, all log entries that are labeled with a severity lower than the default will not be recorded. For example, logging at the alert0 level will cause alert1, alert2 and alert3 messages to be ignored.

As you go down in default severity, the number of entries that are produced will increase, so it's a good idea to turn on only what is necessary to avoid being flooded with too much information. A typical default for production environments is alert1, which records messages logged at the alert1 level or higher priority (alert0, ERROR and FATAL). You can change this to alert0 if you only want to record events that indicate problems or potential problems.

When troubleshooting a problem in production, you might want to reduce the default severity of recorded messages to alert2. This level will typically produce a voluminous output filled with enough context that will help developers debug the issue, but it should be turned off afterward to prevent flooding the system with irrelevant log entries during normal operation of the application.

The alert3 level produces even more logs than alert2 so it shouldn't be used in production for sustained periods. It's better utilized in a development or testing environment where system performance degradation isn't a critical consideration.

Control your default log level is best done through an environmental variable so that you can change it without modifying the code. However, you might need to restart the application each time the log level needs to be updated. There are also several ways to update the log level at runtime, but the specific technique will depend on the application environment and framework used. Ensure to thoroughly investigate the options available if this is something that interests you.

How to use log levels for monitoring and analysis

After you've configured your application to produce logs with the severity levels included, you might be wondering how to use the recorded labels to make sense of the log messages. The three main ways to use log levels for post-logging analysis are discussed below:

1. Filtering

Log levels allow you to quickly sift your logs such that only the relevant ones are displayed. If you use a cloud log management service like Logtail, it's easy to specify filters that display only the ERROR level entries that occurred in a time period.

2. Alerting

Another useful way to use log levels is for creating alerts in various scenarios. You can notify relevant members of your team if a notable event occurs on the system, or if an expected event didn't occur within a specified time frame. The example below sends an alert to configured email addresses when more than five ERROR entries are logged within a 30 second period.

Aside from sending alerts to email addresses, you can configure various integrations so that you can receive alerts in Slack or other services in your stack.

3. Calculating various metrics

Log levels are also a useful tool for generating various metrics about the application, especially those that help gauge its reliability. For example, the number of ERROR or FATAL entries recorded in a specific period is valuable data that could help inform if some sort of "bug squashing sprint" should be next up on the calendar.

Final thoughts

Using the right log level is a crucial step for effective log management. If your log levels are sound, it will be easy to filter your logs by priority, and you can create alerts for notable events. We hope this article has provided enough information to help you understand log levels and when to use them. For more details on logging techniques and practices to follow, check out the other articles in our logging guide.

Thanks for reading, and happy logging!

Logs tell stories.
Read them.

Experience SQL-compatible
structured log management.

Centralize all your logs into one place.

Analyze, correlate and filter logs with SQL.

Create actionable
dashboards.

Share and comment with built-in collaboration.

Got an article suggestion? Let us know

6 Factors to Consider When Choosing a Logging Framework

Logging frameworks are tools that help you standardize logging in your application. This article will guide you through the process of choosing a suitable logging framework for your application

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

What are the levels of logging?

What is log level severe?

What is the highest level of logging?

What should be the logging level in production?