Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- The df -h on the partition /data was showing full usage of 20GB but du -sh /data was showing a usage of only 581MB.
- ubuntu@ip-xxx-xx-x-xx:/data$ df -h
- Filesystem Size Used Avail Use% Mounted on
- ...
- /dev/xvdb 20G 20G 0G 100% /data
- ubuntu@ip-xxx-xx-x-xx:/data$ du -sh .
- 581M .
- This huge difference means that files are deleted from filesystem but are still referenced from some open applications.
- There were no results of deleted files after running the following command.
- ubuntu@ip-xxx-xx-x-xx:/data$ lsof -nP | grep '(deleted)'
- ubuntu@ip-xxx-xx-x-xx:/data$
- After few minutes of confusion I tried running the same command with sudo and there it was.
- ubuntu@ip-xxx-xx-x-xx:/data$ sudo lsof -nP | grep deleted
- sh 5267 root cwd DIR 202,16 0 393217 /data/xxxxxxxxxxxxx (deleted)
- sh 5267 root 2w REG 202,16 3742909615 397337 /data/xxxxxxxxxxxxx/nohup.out (deleted)
- tail 5270 root cwd DIR 202,16 0 393217 /data/xxxxxxxxxxxxx (deleted)
- tail 5270 root 2w REG 202,16 3742909615 397337 /data/xxxxxxxxxxxxx/nohup.out (deleted)
- tail 5270 root 3r REG 202,16 1196962556 397656 /data/xxxxxxxxxxxxx/log/production.log (deleted)
- tail 14934 root 2w REG 202,16 1424371 398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted)
- tail 14934 root 3r REG 202,16 9281997539 659099 /data/xxxxxxxxxxxxx/log/production.log.1 (deleted)
- tail 19409 root 2w REG 202,16 1424371 398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted)
- tail 19409 root 3r REG 202,16 4805248107 655668 /data/xxxxxxxxxxxxx/log/production.log.9 (deleted)
- Looking at the corresponding processes, I found that there were open processes that were holding references to the old files.
- ubuntu@ip-xxx-xx-x-xx:/data$ ps -ef | grep 5267
- ubuntu 2712 1616 0 13:13 pts/3 00:00:00 grep --color=auto 5267
- root 5267 1 0 Jan10 ? 00:00:00 sh -c tail -f -n 0 "/data/xxxxxxxxxxxxx/log/production.log"
- root 5270 5267 0 Jan10 ? 00:00:00 tail -f -n 0 /data/xxxxxxxxxxxxx/log/production.log
- After killing all of such process, df and du matched completely and 18GB of disk space got freed up.
- ubuntu@ip-xxx-xx-x-xx:/data$ df -h
- Filesystem Size Used Avail Use% Mounted on
- ...
- /dev/xvdb 20G 625M 18G 4% /data
- Hurray!
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement