[Linux] “No space left on the device” but there is
Background
We sometimes see an error “No space left on device”,
# touch /tmp/testfile.txt
unable to create `/tmp/testfile.txt': No space left on device
but when we do a df -h, we will see that there is space on the disk. Eg,
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 9.5G 1.9G 7.7G 20% /
Reason
This occurs because the disk drive has no more free “inodes” . In simple terms, inodes are the “metadata” for files written on the disk. Consider this like a database which stores all the “information” about a file on the disk which is used to refer the file on the disk.
A certain (small) percentage of the disk space is reserved to store this inode database for the filesystem written on the disk. As and when files are created on the disk, their corresponding inode information is also creates on the space reserved for the inode.
The command df -i shows the inode information for that file system. Eg, to see the free inodes for / ::
# df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 9961472 37264 9924208 1% /
On the system which is 100% full on the inodes, the output of df -i would look something like this ::
# df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 9961472 9961472 0 100% /
Solution
There is (unfortunately) no way to dynamically add more inode space on the filesystem after it is created. So to recover from the current solution, the only way to delete some files from the disk until it free’s up some inode space on the disk for you to be able to add more files to the disk.
Long term solution
While creating the filesystem, plan to allocate more inode space on the disk than the default allocated inodes on that file system. This can be done using the -i when creating the filesystem. The bigger the number, the less the number of inodes you will have.
Also, it is always a best practice to create and manage filesystems using LVM (Linux Volume Manager) . LVM allows dynamic expansion of filesystems but adding more disk space to it and add more space to the disk also add more inodes to the file system.
More info can be found here
Ref Links for more information
No space left on device
What is an inode
Specifying inode when creating the filesystem