Inodes, links, and disk space

What we're doing

You'll first see what a file really is, an inode plus names. Then you'll meet two traps that follow from it:

1. A deleted-but-open file: a process holds a deleted file open, so its space is still used but du can't find it. You'll locate it and reclaim the space without restarting the process.
2. Inode exhaustion: a filesystem runs out of inodes (file slots) while it still has free bytes. You'll cause it and diagnose it.

Watch the video first, then run these as you read. You're root on this VM, so use sudo where shown. The staged VM has two small loopback filesystems: /mnt/lab (a deleted file is held open here) and /mnt/inodes (formatted with very few inodes).

Heads up: both filesystems are loopback files, so they show up as /dev/loop0 and /dev/loop1. Everything here, ls -li, lsof, df -i, works exactly the same on real disks.

The idea: a file is an inode plus names

A file is two separate things:

• The name (a path like /var/log/app.log), which is just a pointer.
• The inode (short for index node), the real thing: the data blocks plus metadata (owner, size, dates) and a link count, the number of names pointing at it.

The data lives in the inode, not the name. An inode is freed only when two things are true: its link count is 0 (no name points at it) and no process has it open. Everything below comes from that one rule.

Step 1: see the inode and a hard link

echo hello > /tmp/note.txt
ls -li /tmp/note.txt                 # -i adds the inode number as the first column

1310721 -rw-r--r-- 1 root root 6 Jun 12 10:00 /tmp/note.txt

1310721 is the inode number. The 1 after the permissions is the link count (one name).

ln /tmp/note.txt /tmp/note-2.txt     # hard link: a second name for the SAME inode
ls -li /tmp/note.txt /tmp/note-2.txt

1310721 -rw-r--r-- 2 root root 6 Jun 12 10:00 /tmp/note-2.txt
1310721 -rw-r--r-- 2 root root 6 Jun 12 10:00 /tmp/note.txt

Same inode number, one file with two names. Link count is now 2.

rm /tmp/note.txt                     # removes a NAME, not the data
cat /tmp/note-2.txt                  # still "hello"
ls -li /tmp/note-2.txt               # link count back to 1

rm removed one pointer. The data survives because a name still points at the inode. It frees only at link count 0, plus nothing holding it open.

Step 2: find a deleted-but-open file

A service, telemetry-agent, wrote a 150 MB buffer, opened it, then deleted it while keeping it open. Look at its filesystem:

df -h /mnt/lab          # how full the filesystem says it is
sudo du -sh /mnt/lab    # how much du can actually find

/dev/loop0      287M  158M  111M  59% /mnt/lab
20K	/mnt/lab

df says 158 MB used, du finds 20 KB. The missing ~150 MB is a deleted file still held open. Find it:

sudo lsof +L1 /mnt/lab     # +L1 = open files with link count < 1 (deleted but open)

COMMAND     PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK  NODE NAME
telemetry  1423 root    3u   REG    7,0 157286400     0    12 /mnt/lab/telemetry.buf (deleted)

• PID 1423, FD 3u: process 1423 has it open on file descriptor 3 (read-write).
• NLINK 0: no name points at it.
• NAME ... (deleted): the name is gone, the file isn't.

See it from the process side too:

pid=$(systemctl show -p MainPID --value telemetry-agent)
sudo ls -l /proc/$pid/fd/3      # the open handle, as a /proc symlink

lrwx------ 1 root root 64 Jun 12 10:05 /proc/1423/fd/3 -> '/mnt/lab/telemetry.buf (deleted)'

Step 3: reclaim the space without restarting

Stopping the process would free it (fd closes, link count 0, nothing holds it). The no-downtime way is to empty the file through the open descriptor:

sudo truncate -s 0 /proc/$pid/fd/3    # /proc/.../fd/3 IS the deleted file
df -h /mnt/lab

/dev/loop0      287M  4.2M  269M   2% /mnt/lab

Space back, process still running.

Stopping telemetry-agent (sudo systemctl stop telemetry-agent) would free it too. truncate through the fd is the trick when you can't take the service down.

Step 4: run out of inodes (not bytes)

/mnt/inodes was formatted with very few inodes. Fill them with empty files:

cd /mnt/inodes
sudo bash -c 'i=0; while touch f$i 2>/dev/null; do i=$((i+1)); done; echo "stopped after $i files"'
sudo touch /mnt/inodes/onemore        # now watch it fail

stopped after 2037 files
touch: cannot touch '/mnt/inodes/onemore': No space left on device

"No space left," with empty files. Check both meters:

df -h /mnt/inodes      # bytes: nearly empty
df -i /mnt/inodes      # inodes: full

/dev/loop1       58M  2.1M   52M   4% /mnt/inodes
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop1       2048  2048     0  100% /mnt/inodes

4% of bytes, 100% of inodes. df -h can't show this; df -i does. Each file, however small, costs one inode, and the pool is fixed at format time.

Find the directory with the most files, then clear it:

sudo du --inodes -x -d1 /mnt/inodes | sort -n | tail   # count inodes per dir, not bytes
sudo rm /mnt/inodes/f*
df -i /mnt/inodes

/dev/loop1       2048    11  2037   1% /mnt/inodes

Inodes freed, filesystem usable again.

Cheat sheet

# a file is an inode + names
ls -li FILE                 # inode number + link count
ln FILE FILE2               # hard link: another name, same inode
stat FILE                   # inode number, links, blocks

# deleted-but-open file (df > du)
df -h /mnt/lab ; sudo du -sh /mnt/lab     # the disagreement
sudo lsof +L1 /mnt/lab                    # find files deleted but still open
sudo ls -l /proc/<PID>/fd                 # the process's open handles
sudo truncate -s 0 /proc/<PID>/fd/<FD>    # reclaim, no restart

# out of inodes (No space left, bytes free)
df -h PATH                  # bytes meter
df -i PATH                  # inode meter (the real story here)
sudo du --inodes -x -d1 PATH | sort -n | tail   # which dir has the most files

The one thing to remember: a file is an inode plus names, and it's freed only when no name points at it and no process holds it open. So a "deleted" file can still eat space (df sees it, du can't, lsof +L1 finds it), and a disk has two budgets, bytes and inodes, so it can be full on one with the other empty (df -i shows the inode one).

That completes the disk-full toolkit: df -h and du for big files, df -i for inodes, lsof +L1 for the deleted-but-open ghost.