Tips & tricks for systemd and journald on NixOS

This document contains a list of tips and tricks for working with systemd, journalctl, and related tools.

SysVinit vs Upstart vs Systemd

The simplest cheatsheet:

SysVinit Upstart Systemd
/etc/init.d/service start start service systemctl start service
/etc/init.d/service stop stop service systemctl stop service
/etc/init.d/service restart restart service systemctl restart service
/etc/init.d/service status status service systemctl status service

Systemd Unit Types

Systemd has the following unit types you might be concerned with:

Systemd has other types of units but the above list is a good starting point. For more information please consult man systemctl.

The following commands can be used to query information about systemd units:

# List dependencies for a unit
$ systemctl list-dependencies UNITNAME

# List sockets
$ systemctl list-sockets

# List active systemd jobs
$ systemctl list-jobs

# List all units and their respective states
$ systemctl list-unit-files

# List all loaded or active units
$ systemctl list-units

Systemd Services

Most of the time we will be concerned with systemd services. Below are a list of useful commands for working with these:

# Need to have sudo privileges to stop/start/restart services
$ sudo systemctl stop SERVICE
$ sudo systemctl start SERVICE
$ sudo systemctl restart SERVICE

# Query commands anyone can run
$ systemctl status SERVICE
$ systemctl is-active SERVICE
$ systemctl show SERVICE

You can also run systemctl commands remotely like so:

$ systemctl -H hostname status SERVICE

This works for systemctl commands other than just systemd service specific commands.

Log Accessibility By journalctl

For all services that need logs accessed via journalctl you should log to the console from a systemd unit.

For example, Elasticsearch logging configuration can be set as so:

rootLogger: INFO, console
logger:
  action: INFO
  com.amazonaws: WARN
appender:
  console:
    type: console
    layout:
      type: consolePattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Then you will be able to query logs from the elasticsearch service unit by using:

$ journalctl -f -u elasticsearch

Accessing Logs Via journalctl

# tail "follow" all log messages for elasticsearch unit/service
$ journalctl -f -u elasticsearch

# show last 1000 error messages for elasticsearch unit/service (command
# terminates without ^C)
$ journalctl -fen1000 -u elasticsearch

# only show kernel messages in tail "follow" mode:
$ journalctl -k -f

# only show log messages for service BLA since last "boot"
$ journalctl -b -u BLA

# show all error log messages from all log sources since last "boot"
$ journalctl -xab

Many more permutations of options are available on journalctl. Please consult man journalctl for more information.

User Access To journalctl Logs

All users that are in the systemd-journal group should be able to query logs via journalctl. Ensure your SSH user is in this group via groups USERNAME.

NixOS Configuration for journald

The NixOS expression for a node’s configuration contains the following settings that are worth tuning on servers with high frequency events being logged.

As of NixOS 16.03, the defaults for services.journald.rateLimitBurst and services.journald.rateLimitInterval are worth evaluating for your needs:

$ sudo nixos-option services.journald.rateLimitBurst
Value:
100

Default:
100

Description:

Configures the rate limiting burst limit (number of messages per
interval) that is applied to all messages generated on the system.
This rate limiting is applied per-service, so that two services
which log do not interfere with each other's limit.

...

And:


$ sudo nixos-option services.journald.rateLimitInterval
Value:
"10s"

Default:
"10s"

Description:

Configures the rate limiting interval that is applied to all
messages generated on the system. This rate limiting is applied
per-service, so that two services which log do not interfere with
each other's limit. The value may be specified in the following
units: s, min, h, ms, us. To turn off any kind of rate limiting,
set either value to 0.

This means on this system journald will rate limit events per service after 100 messages within 10s. For many servers this is low, and you will want to adjust it with values like the following:

  services.journald.rateLimitBurst = 1000;
  services.journald.rateLimitInterval = 1s;

The above will rate limit services to logging 1000 messages per second.

You can also turn off rate limiting in journald with the following:

  services.journald.rateLimitInterval = 0;