Practice Use drills for recall and labs for real operating judgment.

LAB-HW-03 - Systemd Service Architecture

Master systemctl to start, stop, enable, and inspect the foundational background services (daemons) that keep Linux running.

HW System Lifecycle & Hardware

Systemd Service Architecture

Master systemctl to start, stop, enable, and inspect the foundational background services (daemons) that keep Linux running.

30 min INTERMEDIATE LINUX Curriculum-reviewed
Prerequisites
Lab LAB-FS-02 - Inside /etc - The Config Vault
Success criteria
  • Master systemctl to start, stop, enable, and inspect the foundational background services (daemons) that keep Linux running.
  • Repeat the workflow without copy-paste or step-by-step prompting.
Safety notes
  • Use read-only discovery commands unless the lab explicitly tells you to change hardware or firmware state.

Part A: The Field Guide

🎯 What & Why

When your Linux machine boots, the Kernel wakes up the hardware. But who wakes up the network? Who starts the SSH server so you can log in? Who mounts the hard drives?

On 95% of modern Linux distributions, this job belongs to the absolute dictator of the operating system: systemd.

systemd is Process ID 1. It is the very first software program the Kernel launches. systemd then reads hundreds of configuration files and explicitly launches every other background process (called Daemons or Services) required to make the computer functional.

We command systemd using the systemctl (System Control) tool.

🧠 Mental Model: The Orchestra Conductor

Think of systemd as the Conductor of an orchestra. The individual musicians (Nginx, SSH, MySQL, Cron) are the “Services”.

There are two separate concepts the Conductor manages:

  1. The Current Performance (State): Is the musician playing right now? (Start, Stop, Restart).
  2. The Employment Contract (Enablement): Will this musician be hired to play tomorrow when the orchestra performs again? (Enable, Disable).

You can start a web server today, but if you do not enable it, it will not turn on automatically when the server reboots next week!

📖 Command Reference

Controlling the Present

Live State Control

$ # Ask the conductor for a detailed dossier on a specific musician $ sudo systemctl status ssh

$ # Tell the musician to begin playing immediately $ sudo systemctl start ssh

$ # Tell the musician to stop playing immediately $ sudo systemctl stop ssh

$ # Shoot the musician, hire a clone, and force them to start from the beginning $ sudo systemctl restart ssh

Note: You do not need the .service extension. systemctl restart ssh and systemctl restart ssh.service are exactly identical.

Controlling the Future

Boot Enablement

$ # Write the employment contract so the service auto-starts on the next reboot $ sudo systemctl enable ssh

$ # The Power Move: Enable it for tomorrow AND Start it today simultaneously! $ sudo systemctl enable —now ssh

$ # Fire the musician so they never auto-start again $ sudo systemctl disable ssh

Investigating the Orchestra

What if the server is running slowly, and you want to see every single service currently alive?

Listing Services

$ # Show every single active service running on the machine $ systemctl list-units —type=service —state=running

🌍 Real Scenarios

Scenario 1: The Subtle Reload vs Restart You change the configuration file for your Nginx web server. To apply the change, you could run systemctl restart nginx. However, restart completely kills the master process. Any user currently downloading a file from your website gets instantly disconnected. Instead, professionals use systemctl reload nginx. This sends a polite signal to the master process: “Finish serving your current users, but load the new config file for any users who arrive starting right now.” Zero downtime.

Scenario 2: The Failed Boot Mystery A junior tech reboots the server. The web server doesn’t come back online. You frantically SSH in. You run systemctl status nginx. It says Active: failed (Result: exit-code). Wait, why did it fail? The output of status automatically prints the last 10 log lines generated by the service! You read those 10 lines and instantly see: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use). You know exactly what to fix.

⚠️ Gotchas & Pitfalls

  1. Masking (The Super-Disable) A junior admin disables (systemctl disable mysql) the database because it’s being decommissioned. Tomorrow, another administrator runs systemctl start mysql. It starts perfectly, because disable only stops auto-starting. To make it physically impossible for anyone (even root) to start the service manually without breaking the seal, you must Mask it: sudo systemctl mask mysql. This links the service file into the /dev/null black hole.
  2. “Failed to connect to bus” Error If you try to use systemctl inside a basic Docker container or WSL1, it will throw a terrifying error about D-Bus. This is because basic containers intentionally do not run an init system like systemd to save memory. They are not full operating systems.

Part B: The Drill Deck

Terminal Required: Open a full Linux terminal (Ubuntu/Debian/RHEL). WSL2 works fine, but WSL1 will fail. You need a system running systemd.

G
Guided Step by step - type exactly this and compare the result
>

Exercise G1: The Initial Status Report

Let’s investigate the service that allows you to be logged into your terminal right now!

  1. Run: systemctl status ssh (If on RedHat, the service is named sshd).
  2. Read the Loaded: line. This shows you the absolute path to the actual text file (/lib/systemd/system/ssh.service) that defines how the service works!
  3. Read the Active: line. It should be colored green, saying active (running).

Exercise G2: Playing God

  1. Warning: Do not do this over a remote SSH connection. We will practice on the cron scheduling daemon instead.
  2. Verify it is running: systemctl status cron (or crond on RedHat).
  3. Stop it: sudo systemctl stop cron
  4. Check its status again. The green text is now white/grey inactive (dead).
  5. Start it back up! sudo systemctl start cron

Exercise G3: The Contract

  1. Check if cron is scheduled to boot automatically. Look at the Loaded: line from the status output again. It likely says ...cron.service; enabled;...
  2. Fire the daemon: sudo systemctl disable cron
  3. Notice the output! It physically removed a “symlink” file from your hard drive to break the boot chain.
  4. Hire it back: sudo systemctl enable cron. It will recreate the symlink, proving the contract is signed for the next reboot.
S
Solo Task described, hints available - figure it out
>

Exercise S1: Who is playing right now?

  1. Use the massive list command to view the entire orchestra: systemctl list-units --type=service --state=running
  2. Scroll through the dozens of core services keeping your PC alive (NetworkManager, systemd-journald, udev).
  3. Press q to quit the list view.

Exercise S2: Reading the Code

You know that systemd just reads text files to figure out how to start a program. Let’s read one of those text files!

  1. Find the path to the SSH service file by running systemctl status ssh (or sshd) and looking at the Loaded: row. (e.g., /lib/systemd/system/ssh.service).
  2. Use cat to print the contents of that entire file to your screen.
  3. Look at the [Service] block. You will see ExecStart=... this is literally the exact bash command the Conductor types in the background to launch the program!
M
Mission Real scenario - no hints, combine multiple skills
>

Mission M1: The Failing Service

We are going to intentionally break a service and use systemctl to diagnose the failure, just like a real sysadmin investigating an outage.

  1. You will need a dummy service to break. We will use the chrony time-sync daemon. Install it: sudo apt install chrony (or sudo dnf install chronyd).
  2. Verify it runs: sudo systemctl status chrony (or chronyd).
  3. Now, we break it. Open its configuration file and delete everything. Run: sudo bash -c "echo 'BOGUS_GARBAGE' > /etc/chrony/chrony.conf" (Ubuntu) (Or /etc/chrony.conf on RedHat).
  4. Restart the service to apply the “updated” configuration: sudo systemctl restart chrony

Your Mission:

  1. Run a status check on the service.
  2. Observe the terrifying red Failed state.
  3. Your mission is to locate the exact syntax error the daemon vomited out before it died. Look at the bottom 10 log lines provided by systemctl status. You should explicitly see the word BOGUS_GARBAGE listed as an unrecognized command, perfectly diagnosing the crash!
  4. (Bonus: Fix the file by emptying it > /etc/chrony... and restart the service so it turns green again!)