Advertisement
Tritonio

Determining Specific File Responsible for High I/O

Mar 15th, 2021
112
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 6.15 KB | None | 0 0
  1. There are several aspects to this question which have been addressed partially through other tools, but there doesn't appear to be a single tool that provides all the features you're looking for.
  2. iotop
  3.  
  4. This tools shows which processes are consuming the most I/O. But it lacks options to show specific file names.
  5.  
  6. $ sudo iotop
  7. Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  8. TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
  9. 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
  10. 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
  11. 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
  12. 5 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u:0]
  13. 6 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
  14. 7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
  15.  
  16. By default it does what regular top does for processes vying for the CPU's time, except for disk I/O. You can coax it to give you a 30,000 foot view by using the -a switch so that it shows an accumulation by process, over time.
  17.  
  18. $ sudo iotop -a
  19. Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  20. TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
  21. 258 be/3 root 0.00 B 896.00 K 0.00 % 0.46 % [jbd2/dm-0-8]
  22. 22698 be/4 emma 0.00 B 72.00 K 0.00 % 0.00 % chrome
  23. 22712 be/4 emma 0.00 B 172.00 K 0.00 % 0.00 % chrome
  24. 1177 be/4 root 0.00 B 36.00 K 0.00 % 0.00 % cupsd -F
  25. 22711 be/4 emma 0.00 B 120.00 K 0.00 % 0.00 % chrome
  26. 22703 be/4 emma 0.00 B 32.00 K 0.00 % 0.00 % chrome
  27. 22722 be/4 emma 0.00 B 12.00 K 0.00 % 0.00 % chrome
  28.  
  29. i* tools (inotify, iwatch, etc.)
  30.  
  31. These tools provide access to the file access events, however they need to be specifically targeted to specific directories or files. So they aren't that helpful when trying to trace down a rogue file access by an unknown process, when debugging performance issues.
  32.  
  33. Also the inotify framework doesn't provide any particulars about the files being accessed. Only the type of access, so no information about the amount of data being moved back and forth is available, using these tools.
  34. iostat
  35.  
  36. Shows overall performance (reads & writes) based on access to a given device (hard drive) or partition. But doesn't provide any insight into which files are generating these accesses.
  37.  
  38. $ iostat -htx 1 1
  39. Linux 3.5.0-19-generic (manny) 08/18/2013 _x86_64_ (3 CPU)
  40.  
  41. 08/18/2013 10:15:38 PM
  42. avg-cpu: %user %nice %system %iowait %steal %idle
  43. 18.41 0.00 1.98 0.11 0.00 79.49
  44.  
  45. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  46. sda
  47. 0.01 0.67 0.09 0.87 1.45 16.27 37.06 0.01 10.92 11.86 10.82 5.02 0.48
  48. dm-0
  49. 0.00 0.00 0.09 1.42 1.42 16.21 23.41 0.01 9.95 12.22 9.81 3.19 0.48
  50. dm-1
  51. 0.00 0.00 0.00 0.02 0.01 0.06 8.00 0.00 175.77 24.68 204.11 1.43 0.00
  52.  
  53. blktrace
  54.  
  55. This option is too low level. It lacks visibility as to which files and/or inodes are being accessed, just raw block numbers.
  56.  
  57. $ sudo blktrace -d /dev/sda -o - | blkparse -i -
  58. 8,5 0 1 0.000000000 258 A WBS 0 + 0 <- (252,0) 0
  59. 8,0 0 2 0.000001644 258 Q WBS [(null)]
  60. 8,0 0 3 0.000007636 258 G WBS [(null)]
  61. 8,0 0 4 0.000011344 258 I WBS [(null)]
  62. 8,5 2 1 1266874889.709032673 258 A WS 852117920 + 8 <- (252,0) 852115872
  63. 8,0 2 2 1266874889.709033751 258 A WS 852619680 + 8 <- (8,5) 852117920
  64. 8,0 2 3 1266874889.709034966 258 Q WS 852619680 + 8 [jbd2/dm-0-8]
  65. 8,0 2 4 1266874889.709043188 258 G WS 852619680 + 8 [jbd2/dm-0-8]
  66. 8,0 2 5 1266874889.709045444 258 P N [jbd2/dm-0-8]
  67. 8,0 2 6 1266874889.709051409 258 I WS 852619680 + 8 [jbd2/dm-0-8]
  68. 8,0 2 7 1266874889.709053080 258 U N [jbd2/dm-0-8] 1
  69. 8,0 2 8 1266874889.709056385 258 D WS 852619680 + 8 [jbd2/dm-0-8]
  70. 8,5 2 9 1266874889.709111456 258 A WS 482763752 + 8 <- (252,0) 482761704
  71. ...
  72. ^C
  73. ...
  74. Total (8,0):
  75. Reads Queued: 0, 0KiB Writes Queued: 7, 24KiB
  76. Read Dispatches: 0, 0KiB Write Dispatches: 3, 24KiB
  77. Reads Requeued: 0 Writes Requeued: 0
  78. Reads Completed: 0, 0KiB Writes Completed: 5, 24KiB
  79. Read Merges: 0, 0KiB Write Merges: 3, 12KiB
  80. IO unplugs: 2 Timer unplugs: 0
  81.  
  82. Throughput (R/W): 0KiB/s / 510KiB/s
  83. Events (8,0): 43 entries
  84. Skips: 0 forward (0 - 0.0%)
  85.  
  86. fatrace
  87.  
  88. This is a new addition to the Linux Kernel and a welcomed one, so it's only in newer distros such as Ubuntu 12.10. My Fedora 14 system was lacking it 8-).
  89.  
  90. It provides the same access that you can get through inotify without having to target a particular directory and/or files.
  91.  
  92. $ sudo fatrace
  93. pickup(4910): O /var/spool/postfix/maildrop
  94. pickup(4910): C /var/spool/postfix/maildrop
  95. sshd(4927): CO /etc/group
  96. sshd(4927): CO /etc/passwd
  97. sshd(4927): RCO /var/log/lastlog
  98. sshd(4927): CWO /var/log/wtmp
  99. sshd(4927): CWO /var/log/lastlog
  100. sshd(6808): RO /bin/dash
  101. sshd(6808): RO /lib/x86_64-linux-gnu/ld-2.15.so
  102. sh(6808): R /lib/x86_64-linux-gnu/ld-2.15.so
  103. sh(6808): O /etc/ld.so.cache
  104. sh(6808): O /lib/x86_64-linux-gnu/libc-2.15.so
  105.  
  106. The above shows you the process ID that's doing the file accessing and which file it's accessing, but it doesn't give you any overall bandwidth usage, so each access is indistinguishable to any other access.
  107. So what to do?
  108.  
  109. The fatrace option shows the most promise for FINALLY providing a tool that can show you aggregate usage of disk I/O based on files being accessed, rather than the processes doing the accessing.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement