What's eating my CPU and memory?

What we're doing

The VM has two staged processes running: report-builder (using a full CPU core) and image-cache (holding ~460 MB). We find each one with the process tools and learn which column to trust. This VM has 2 cores and ~3.9 GB RAM.

Watch the video first, then run these as we read. Everything here only reads what is already running, so we do not need sudo.

Understanding the two resources

CPU and Memory are separate resources, and so are the processes using them. We find the CPU one by sorting on %CPU, and the memory one by sorting on %MEM / RES. The two memory columns are not the same:

• VIRT / VSZ (Virtual memory Size): everything the process mapped (code, libraries, reserved memory). Often very large, mostly not in RAM. Misleading on its own.
• RES / RSS (Resident Set Size, "resident" = present in RAM): the memory actually in RAM right now. This is the real cost.

Step 1: how many cores

nproc                  # number of CPU cores

2

A process at 100% CPU is using one full core. On 2 cores that is half the machine.

Step 2: the live view with top

top                    # live view; sorts by %CPU by default; q to quit

%Cpu(s): 50.0 us,  0.0 sy,  0.0 ni, 50.0 id, ...
MiB Mem :   3914.9 total,  2222.2 free,  934.2 used,  991.0 buff/cache

    PID USER   PR  NI    VIRT    RES   SHR S  %CPU  %MEM     TIME+ COMMAND
   3308 root   20   0   18620  10948  6936 R 100.0   0.3   0:32.85 report-builder
      1 root   20   0   22580  13924  9608 S   0.0   0.3   0:03.68 systemd

report-builder is at 100.0 %CPU, one full core, which is the 50.0 system %Cpu(s) on 2 cores. The summary is for the whole machine; the rows are per process.

In top:

• M = sort by memory → image-cache moves up, 11.7 %MEM, RES ~460 MB.
• P = sort by CPU (the default).
• q = quit.

Step 3: the snapshot with ps

ps aux --sort=-%mem | head   # snapshot, sorted by memory (largest first)
ps aux --sort=-%cpu | head   # same, sorted by CPU

USER         PID %CPU %MEM    VSZ    RSS STAT COMMAND
root        3360  0.6 11.7 479424 471756 Ss   python3 /usr/local/sbin/image-cache
root        1540  0.0  2.0 2014560  82988 Ssl  /usr/bin/dockerd -H fd:// ...
root         877  0.1  1.1 1793224  45912 Ssl  /usr/bin/containerd

• aux = all processes, with the useful columns. --sort=-%mem = by memory, - = largest first. | head = first few lines.
• image-cache is the memory one: 11.7 %MEM, RSS ~460 MB.
• ps calls the columns VSZ/RSS (the same as top's VIRT/RES), and COMMAND shows the full command line (the python3 path), where top showed the short name.
• We use ps when we want a snapshot to search, save, or use in a script.

Step 4: read RES, not VIRT

Look at dockerd above: VSZ ~2 GB but RSS only ~81 MB. VSZ/VIRT counts everything mapped; only RSS/RES is really in RAM. So image-cache is the real memory user, because its RES is ~460 MB, while dockerd's 2 GB is almost all virtual.

Step 5: htop (easier to read, if installed)

htop                   # one bar per core + a memory bar; F6 sort, arrows move, F10/q quit

Same numbers as top, easier to read: one core's bar is full, the other almost empty.

Cheat sheet

nproc                          # core count (helps read %CPU)

# live
top                            # P = sort by CPU, M = sort by memory, q = quit
htop                           # F6 = sort, F10/q = quit

# snapshot (for scripts)
ps aux --sort=-%cpu | head     # most CPU first
ps aux --sort=-%mem | head     # most memory first

The one thing to remember: CPU and memory are separate, so we read separate columns. Sort by %CPU for the CPU process (100% = one core, check nproc), and by %MEM/RES for the memory process, where RES (not VIRT) is the memory really in RAM.

Next tutorial: load average, the three numbers after load average: on top's first line, and what they mean.