I tried booting up into Debian and got all sorts of systemd breakages apparently because my /var partition was full. That's fair, but the pain started when Debian frustrated any attempt to free up space. I'm wondering if this is a 'feature' that needs removing or if there might be a bug in the underlying filesystem. I really don't want anymore finite hours of my life (or anyone else's) lost in this problem if I can find the cause.
One of the culprits in my full /var partition was a 3 gig syslog file which has only been getting bigger since January despite running logrotate -f. I try to run it this time but I'm told that it can't rotate anything because there's no space left of the device. OK, Plan B. Another thing that the interwebs say is to run apt-get clean to sweep out downloaded packages, of which I collected hundreds of megabytes. Again, this command failed because there was no space left on the device. Since there's almost no documentation as to what can be safely rm'd in /var without breaking your system, I decide the least risky choice is to sudo rm -rf the offending 3-gig syslog file from single-user mode and the systemd debug shell. But *THIS* command failed because there was 'no space left on the device'. Is this right? Does rm need space on a drive to free other space? If so, how on earth can you fix a full partition if you can't remove anything from it?! Since Debian can't delete files from its own partitions, I have to boot from a Ubuntu DVD. I'm able to rm -rf the syslog file from that, but when I reboot into Debian, I get the same 'no space left on device' errors. That's weird, so I df -h to figure out what's going on and df correctly reports a 5G var partition, of which under 3G are now used and avail space is 0G. Whoa, wait, what?!?! How can 5G - 3G = 0G?! I start blindly casting whatever btrfs spells I can find on the Internet to fix 'no space left on device' errors. One of them eventually works and df -h correctly reports the free space in my /var partition and Debian boots normally again. My question, therefore, is whether this is a btrfs bug that got triggered by the full /var partition or whether Debian is designed to break irrecoverably when /var fills up. Any ideas of what happened?
