Signals and killing processes properly

What we're doing

The VM has two staged services running: order-worker (shuts down cleanly on SIGTERM) and legacy-sync (ignores SIGTERM, so it needs SIGKILL). We find them, stop the polite one cleanly, and force the stubborn one.

Watch the video first, then run these as we read. Both services run as root, so we use sudo to signal them. To redo the demo, re-arm with sudo systemctl start <name>.

What kill really does

kill does not directly stop anything. It sends a signal to a process, and for most signals the process decides what to do (run a handler, ignore it, or take the default). The exception is SIGKILL, which the kernel carries out directly, so it cannot be caught or ignored.

• SIGTERM (signal terminate, 15): the polite default. "Please shut down." A well-behaved process cleans up and exits.
• SIGKILL (signal kill, 9): forced. The kernel stops it at once, with no cleanup. Last resort.
• SIGHUP (signal hangup, 1): often means "reload config" for daemons.
• SIGINT (signal interrupt, 2): what Ctrl+C sends to the foreground process.

Step 1: find the process (its PID)

pgrep -a order-worker      # find PIDs by name; -a also shows the command line

1487 /usr/bin/python3 /usr/local/sbin/order-worker

PID (process ID) is 1487. Plain pgrep order-worker prints just the number.

Step 2: stop it politely (SIGTERM)

sudo kill 1487             # no signal named = SIGTERM (15), the polite "please stop"
pgrep order-worker         # empty = it stopped
systemctl status order-worker --no-pager

○ order-worker.service ... Active: inactive (dead)
   order-worker[1487]: caught SIGTERM, finishing up and shutting down cleanly
   systemd[1]: order-worker.service: Deactivated successfully.

It caught the signal and shut down cleanly. That is a graceful stop.

Step 3: a stubborn process, then force it (SIGKILL)

pgrep legacy-sync
sudo kill 1492             # polite SIGTERM again...
pgrep legacy-sync          # ...still there: it ignores SIGTERM
sudo kill -9 1492          # -9 = SIGKILL (9), the kernel forces it; cannot be caught
pgrep legacy-sync          # empty = gone

1492
1492
(after kill -9: nothing)

SIGTERM did nothing because legacy-sync ignores it. SIGKILL stopped it instantly, but with no clean-shutdown message, it got no chance to clean up. systemctl status legacy-sync shows Active: failed (Result: signal), signal=KILL.

Step 4: signal by name (pkill / killall)

sudo systemctl start legacy-sync   # re-arm
sudo pkill -9 legacy-sync          # signal every process named legacy-sync (here: SIGKILL)

pkill skips the PID lookup and signals by name. Without -9 it sends SIGTERM. killall is the same idea. Careful: they hit every match, so check with pgrep first.

Step 5: the signal list

kill -l                    # list all signal names and numbers

SIGHUP (1) often means "reload config"; SIGINT (2) is Ctrl+C (why it didn't stop our background job last tutorial, the signal only reaches the foreground).

Cheat sheet

pgrep -a NAME              # find PID(s) by name (+ command line)
sudo kill PID              # SIGTERM (15): ask it to stop cleanly  <- try this first
sudo kill -9 PID           # SIGKILL (9): force it (no cleanup)     <- last resort
sudo pkill NAME            # SIGTERM by name;  pkill -9 NAME = SIGKILL by name
kill -l                    # list signals

The one thing to remember: kill sends a signal, it does not force anything. Send SIGTERM first (plain kill) so a well-behaved process can shut down cleanly; give it a few seconds; reach for SIGKILL (kill -9) only if it ignores us, because it skips all cleanup. Ask first, force last.

Next tutorial: the OOM killer, what happens when the kernel runs low on memory and starts killing processes on its own.