LVM
The Disk That Almost Took Down Our Deployment Pipeline
It was a Tuesday. Not a special Tuesday. The kind where you have three PRs open, a standup in ten minutes, and absolutely no reason to expect anything to go wrong.
Then Slack lit up.
Our Jenkins build server had stopped accepting jobs. No error in the pipeline itself — just silence. Builds queuing, nothing running. I SSH'd in expecting some rogue process eating CPU. Instead:
bash
df -h
/dev/sda1 50G 50G 0 100% /
Full. Completely full. The build artifacts had been quietly accumulating for weeks and nobody had set up a cleanup job. Classic.
The obvious fix — delete old artifacts, free up space, move on — would have worked. But we'd hit this wall before, and I knew we'd hit it again in six weeks. The root partition was a fixed 50GB, carved out at VM provisioning time by someone who underestimated how fast Jenkins eats disk.
Resizing a partition on a running VM without LVM? You're looking at downtime, snapshot restores, praying to whatever cloud console you're in. Not ideal at 11 AM with deploys queued.
But this server had LVM underneath. I hadn't set it up — it came that way, provisioned by someone on the infra team who had the foresight to use a volume group with unallocated space sitting there, unused, like an emergency fund nobody told me about.
bash
vgdisplay
--- Volume group ---
VG Name ubuntu-vg
VG Size 200.00 GiB
Alloc PE / Size 50.00 GiB / 50.00 GiB
Free PE / Size 150.00 GiB / 150.00 GiB
150GB free. Just sitting there.
Three commands later:
bash
lvextend -L +50G /dev/ubuntu-vg/ubuntu-lv
resize2fs /dev/ubuntu-vg/ubuntu-lv
No reboot. No downtime. Builds started running again before my standup ended. I extended a live filesystem on a running server in under two minutes.
I sent a message to the infra engineer who'd provisioned it: "Whoever set this up with LVM — thank you." He replied with a thumbs up. Didn't even seem like a big deal to him.
That's when it clicked for me. LVM isn't a feature you use every day. It's the thing you set up before you need it, like a spare tire. You don't think about it until you're on the side of the road — and then it's the only thing you care about.
What actually happened under the hood
LVM sits between your physical disks and your filesystem. The stack looks like this:
Physical Volumes (PVs) — the actual disks or partitions (
/dev/sda,/dev/sdb)Volume Groups (VGs) — a pool that combines one or more PVs
Logical Volumes (LVs) — what your OS actually mounts; carved out of the VG pool
The magic is that LVs aren't tied to physical disk boundaries. You can grow them into free VG space, add new disks to the VG, or even span an LV across multiple disks — all without touching the filesystem mount point or rebooting.
In a DevOps context, this matters more than people realize:
Kubernetes nodes running out of container storage mid-workload — extend the LV, resize the filesystem, no node drain needed
CI/CD build servers accumulating artifacts — grow the volume instead of scrambling to clean up at 11 AM
Database servers where stopping Postgres to resize a partition is genuinely not an option
Monitoring stacks where Prometheus TSDB fills up and your alerting goes blind right when you need it most
The pattern is always the same: fixed partitions get full at the worst possible time, and LVM is the thing that lets you fix it without anyone noticing.
The random thing
The Volume Group on that server was named ubuntu-vg. I only know this because I once spent 20 minutes convinced I was on the wrong server — the hostname said one thing, the VG name said another, and I was about to extend the wrong volume. Always check hostname before you touch anything. Basic stuff. I know.
Set it up before you need it
If you're provisioning VMs — for Jenkins, for Prometheus, for anything that writes to disk over time — use LVM from the start. Leave unallocated space in the VG. It costs nothing and it will save you at least once.
bash
# At provisioning time — leave headroom in the VG
pvcreate /dev/sdb
vgcreate data-vg /dev/sdb
lvcreate -L 50G -n data-lv data-vg
mkfs.ext4 /dev/data-vg/data-lv
mount /dev/data-vg/data-lv /var/lib/jenkins
# The remaining space in data-vg sits ready for when you need it
Future you will SSH into a full server at the worst possible time, run vgdisplay, see free space, and feel an emotion that is difficult to describe but resembles relief mixed with gratitude toward past you.
Set it up right the first time. Leave the headroom. The disk will fill up. It always does.