Systemd or How I learned to stop worrying and love the newness

Big Data Engineering is such a field of work where there is a lot of cogs lying around waiting to be integrated. In other words, DevOps and, in this particular case, Linux are inevitable and inseparable parts of Big Data Engineering. This is a story about one of those cogs.
Featured story Blog
Systemd or How I learned to stop worrying and love the newness

Working in the IT world where things are not yet fully connected, integrated and automated leaves a plenty of opportunities for engineers to get their hands dirty by hard DevOps work. Big Data Engineering is such a field of work where there is a lot of cogs lying around waiting to be integrated. In other words, DevOps and, in this particular case, Linux are inevitable and inseparable parts of Big Data Engineering. This is a story about one of those cogs.

Systemd is a relatively new addition to the Linux ecosystem. It is a system and service manager that is compatible with SysV and LSB init scripts and can work as a drop-in replacement for sysvinit. Systemd’s inception was followed by a huge controversy. The flame war is not yet fully extinguished but systemd has become a part of many Linux distributions and, in my opinion, is here to stay. This post describes my encounter with systemd.


The story started naively, as it usually does – another day, another DevOps task. This time, creating an Ansible role for deploying OS metrics, one of our tools at SmartCat Labs, was needed.

OS metrics is one of the goodies in SmartCat’s monitoring toolbox. It is an OS metrics data shipper or, in project terms, a reporter that sends collected data as a stream of events to Riemann server. More about the monitoring stack we use can be found in a separate blog post – Monitoring stack for distributed systems.

So far, so good.


OS metrics is a python project. To start it manually, a python command like the following has to be executed.

As we want the OS metrics running as a system service, we need to provide an extra step to ‘daemonize’ it. The recommended way to do this in Ansible is to use the service module. It is an Ansible core module responsible for controlling services on remote hosts that supports a variety of init systems (BSD init, OpenRC, SysV, Solaris SMF, systemd, upstart).
The service module provides a nice abstracted way of service handling. An Ansible task for starting could look like this:

But this is not enough to have a service up and running. Depending on the underlay init system, a service script or definition is needed.
Even without a conciseness thought, the reflex was to provide a SysV init script. Old habits die slowly and the/etc/init.d/ mantra is carved deeply into the gray matter. So I extended the task like this:

Then, I started looking for an init script, suitable for starting/stopping python process. One can have a favorite SysV init script, carefully tailored over the years and used in many situations (my preciousss!). I, on the other hand, belong to those that look for it out there every time. The thing is that you can never find a perfect one – it’s so elusive. You could end up with a SystemV init script template like this or this and boldly go where no man has gone before. But in the end it could resemble something like this (as it was in my case):

Your mileage may vary depending on the distribution, distribution version, personal preferences and so on. But in the end, it looks very familiar and, what’s most important, it is simple and easily understandable in every of its incarnations. Right? In some cases, yes. In some other cases, you are supposed to meditate and live an ascetic life for several years in order to achieve the spiritual level needed for understanding it.

Once a decent init script was in place, I proceeded with the Ansible role using an CentOS 7 minimal Vagrant box as the test platform (CentOS 7 was the target platform) . But it started complaining right away that the service unit file had been changed and that systemctl daemon-reload has to be executed.


It didn’t take me long to realize that CentOS 7 is using systemd to manage system services. Even if interacting with the init V script directly, /etc/init.d/functions provides a way for systemd to hijack the process.

A close inspection of /etc/init.d/functions revealed that SYSTEMCTL_SKIP_REDIRECT variable can be used to avoid systemd completely. For example, you could add a line like this

into your init script before executing . /etc/init.d/functions and you should be good.

But I decided to embrace the newness and to give it a try to prove itself, despite all the horrible things that could be found googling for systemd. Most of these accusing posts were published in 2014 when systemd was introduced in some of the popular Linux distributions. Since then, systemd has gained a wider acceptance. Now there are plenty of overviews, documentation and how-to resources. For example, this DigitalOcean community tutorial or RedHat System Administration Guide provide very good introductions on the subject, as well as basic usage instructions.


Systemd is a system and service manager for Linux operating systems. Except for daemons, it can control different system tasks such as file system mount points, timers, socket and D-Bus activation. Systemd introduced the concept of units (systemd units). Each unit represents a single systemd responsibility/task. The list of available unit types can be found here. These units are defined by unit configuration files. A systemd unit file consists of configuration directives that describe the unit and define its characteristics. Unit file names have the following form:

and are located in /etc/systemd/system/. Unit files typically follow the following structure (a simple service unit example):

Where the unit file sections have the following function:

  • [Unit] — contains generic options that are not dependent on the type of the unit. These options provide unit description, specify the unit’s behavior, and set dependencies to other units.
  • [Service] — this section, named after the unit type (hence [Service]), contains unit type-specific directives.
  • [Install] — information about unit installation options used by systemctl commands.

The above unit file is an example of a simple network-related service. The After specifies a list of services or targets that the service should be started after. ExecStart specifies commands or scripts to be executed when the unit is started. Type configures the unit process startup type that affects the functionality of ExecStart and related options. In this example, simple describes a service where the process started with ExecStart is the main process of the service. When systemd starts a system service, it stores the ID of its main process (PIDFile) in order to keep track of it. The systemctl utility then uses this PID to query and manage the service.

Knowing all this, an Ansible template for the osmetrics unit file (osmetrics.service.j2) could look like this:

where {{ osmetrics_install_dir }}/osmetrics is a convenient shell script for starting the osmetrics process:

The osmetrics unit file should be placed in the /etc/systemd/system/ directory with correct file permissions using proper Ansible tasks:

And that’s pretty much it. Now compare the init V script with the osmetrics unit file! Pure Zen, right?


In my opinion, the most significant benefit of introducing systemd in Linux is the regularization/unification of service-related administration tasks. For certain orthodox Linux users, it is precisely the differences between distributions that represent totems and tokens of superiority. But if working with different Linux distributions on a daily basis is your job description, the systemd’s approach could really be a lifesaver.

Systemd promotes a declarative style (as an opposite of the imperative used in InitV, for example) for describing/controlling services. Avoiding imperative steps, that is, removing the need to describe a service through a series of commands that defines how and instead stating what needs to be done, reduces the complexity and makes things familiar and easy to maintain. For me, this is a proven way to ensure good maintainability and to reduce administration burden. Take the Ansible vs Puppet case, or Maven vs Ant, for example.

Replacing well established and old habits, such as the init system, can be hard enough by itself. Also, systemd is a relatively new project and, as such, is far from being flawless. But, having all its advantages in mind and its increasing adoption among major distributions, it seems well worth of exiting the comfort zone.