API tools faq
paste
cakemaker

iaStorAC.sys BSOD on wakeup from hibernation

cakemaker
Oct 7th, 2025 (edited)
299
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 216.59 KB | None | 0 0
raw download clone embed print report
  1. # Analysis of a system crash related to Intel iaStorAC.sys on wakeup from hibernation
  2. Crashdump analysis: serbian.intel.hibernate.wakeup.3rd_party.debugprivilege.dmp.
  3. Kernel dump file provided by DebugPrivilege: https://mega.nz/file/oo8kXSpA#U7LNLJSdXtAq4Vfq6TFv1KcJ9WzQm_NBDwbQHQrKbQo
  4. Analysis by sixtyvividtails.
  5.  
  6. Here I'll just provide windbg output with my comments.
  7. You can spot windbg commands and important stuff in the wall of text by scanning for chars in the higher Unicode planes;
  8. if you have human eyes you should see colored emojis.
  9.  
  10. Edit 2025-10-08:
  11. * slightly rephrased the first encounter with iaStorAC to indicate there's more evidence towards the analysis end
  12. * added link to the 3rd analysis
  13.  
  14.  
  15. ## Analysis Links
  16. There are now three analysis takes for the very same dump :)
  17.  
  18. 1. First analysis by DebugPrivilege. Has some daring takes, but further reviews show the root cause lies elsewhere.
  19. https://medium.com/@Debugger/hibernation-crash-traced-to-intel-gpu-driver-igdkmdn64-during-power-transition-5b05b30c94ce
  20. thread: https://twitter.com/DebugPrivilege/status/1970720013125361763
  21.  
  22. 2. Analysis by sixtyvividtails. In response to the first analysis. Identifies iaStorAC.sys as the root cause.
  23. 🚏 [You are here]: https://pastebin.com/psVhq6Pe
  24. 👉 thread: https://x.com/sixtyvividtails/status/1975606844363075714
  25.  
  26. 3. Second analysis by DebugPrivilege. Contains additional pointers and evidence, marking iaStorAC.sys as the culprit.
  27. https://medium.com/@Debugger/unexpected-store-exception-0x154-root-cause-storage-i-o-failure-in-iastorac-sys-2745d9de6202
  28. thread: https://x.com/DebugPrivilege/status/1975733380395495671
  29.  
  30.  
  31. 🤨 Wait, which analysis to read then?
  32. Since you're already here at (2), you should save traffic and read this one 😸. Or you can read the (3), your decision.
  33. Or, if you like, you can look at both (2) and (3) - while the conclusion is basically the same,
  34. the flow in (2) and (3) is quite different, and analysis steps don't even intersect much!
  35.  
  36.  
  37. ## Additional Links
  38. script to enum ntfs volumes: https://twitter.com/sixtyvividtails/status/1960578133616152712
  39. Whitepaper referenced: 2019.Extracting Compressed Pages from the Windows 10 Virtual Store.Sardar.[MemCompression].pdf
  40.  
  41.  
  42. ## Analysis
  43. 🌽 [opens crashdump file in the newest Windbg]
  44.  
  45. Microsoft (R) Windows Debugger Version 10.0.27920.1001 AMD64
  46. Copyright (c) Microsoft Corporation. All rights reserved.
  47.  
  48. Loading Dump File [C:\stuff\dumps\serbian.intel.hibernate.wakeup.3rd_party.debugprivilege.dmp]
  49. Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.
  50.  
  51. Windows 10 Kernel Version 19041 MP (8 procs) Free x64
  52. Product: WinNt, suite: TerminalServer SingleUserTS Personal
  53. Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
  54. Kernel base = 0xfffff801`14200000 PsLoadedModuleList = 0xfffff801`14e2a230
  55. Debug session time: Thu Jun 24 04:42:52.788 2021 (UTC + 0:00) 🌽 time of crash (analyst local time)
  56. System Uptime: 6 days 17:51:30.552 🌽 uptime at the moment of crash
  57. Loading Kernel Symbols...
  58. Loading User Symbols...
  59. PEB address is NULL ! 🌽 no usermode dll symbols: no PEB[.Ldr]; that's ok
  60. 🌽 ^^ Note "Debug session time" displays time of crash as local time of the analyst box (machine with Windbg running),
  61. it is NOT the local time on the target box (the crashed machine).
  62. The value in parens is simply analyst timezone. Since I live in Utc, my timezone is set to 0 (convenient).
  63.  
  64.  
  65. 🌽 Since we've started talking about timezones, let's check target box timezone too.
  66. ✨✨✨✨✨✨✨✨
  67. 2: kd> !kuser
  68. _KUSER_SHARED_DATA at fffff78000000000 🛰️ I've copypasted this address for command right below
  69. TickCount: fa00000 * 00000000023908a3 (6:17:51:30.546)
  70. TimeZone Id: 2 🌽 2: TIME_ZONE_ID_DAYLIGHT (daylight saving)
  71. ImageNumber Range: [8664 .. 8664]
  72. Crypto Exponent: 0
  73. SystemRoot: 'C:\WINDOWS'
  74. BootId: 10 🌽 system has booted only 0n10 times since install
  75.  
  76.  
  77. ✨✨✨
  78. 2: kd> ?? *(int64*)(0xfffff78000000000 + #FIELD_OFFSET(KUSER_SHARED_DATA, TimeZoneBias))/(10*1000*1000*60)
  79. int64 0n-120 🌽 target timezone is -120 minutes, i.e. UTC-2:00 (includes daylight bias)
  80. 🌽 ^^ If you're unsure what just happened:
  81. I've used C++ expression evaluation "??" (similar to masm evaluation "?"), to do a very simple thing:
  82. read KUSER_SHARED_DATA.TimeZoneBias, cast it to int64, and convert value from centums (100 ns units) into minutes.
  83.  
  84.  
  85. 🌽 Let's continue with time.
  86. ✨✨✨
  87. 2: kd> !time
  88. CURRENT TIME:
  89. System: 01d768b3`64a7f260 (2021 Jun 24 04:42:52.788) 🌽 target box time, *always* UTC here
  90. Interrupt: 0000054c`aea24ca6 (6 days, 17:51:30.552) 🌽 target "uptime"
  91. Unbiased Interrupt: 000001ba`87a64e35 (2 days, 04:47:45.136) 🌽 target uptime without sleeping
  92. 🌽 ^^ We are already starting to get some insights. We can see that system "uptime" is more than 6 days.
  93. However, that "uptime" is biased. That is, it includes sleep/hibernate time! (Also some of DRIPS time).
  94. For the real uptime, see Unbiased Interrupt Time.
  95. 💎 So the system has booted 6¾ days ago [before the crash], but actual total work time is about 53 hours (2 days+).
  96. I guess such use pattern is fairly normal for a laptop.
  97.  
  98.  
  99. 🌽 Okay, enough messing around. Let's run the magic command which shall solve everything.
  100. ✨✨✨✨✨✨✨✨
  101. 2: kd> !analyze -v
  102. 🌽 [outputs nothing]
  103. 🌽 [nothing happens]
  104. Okay. I've actually just updated DbgX (aka Windbg Preview). That's version from 2025-08-27. Today is 2025-09-28.
  105. I was pretty sure Microsoft ought to unfuckup any serious Windbg fuckups in a month time? Lol, wtf.
  106. 💀 Yep, the basic command "!analyze" doesn't work in this Windbg for kernel dumps. Looks like Microsoft code became
  107. so good, they just don't need to analyse kernel crashdumps anymore! 😸. Fine. Let's just switch to Windbg Classic.
  108.  
  109.  
  110. ✨✨✨✨✨✨✨✨
  111. 🌽 [reopens crashdump file in the Windbg classic]
  112.  
  113. Microsoft (R) Windows Debugger Version 10.0.26100.1 AMD64
  114. Copyright (c) Microsoft Corporation. All rights reserved.
  115.  
  116. Loading Dump File [C:\stuff\dumps\serbian.intel.hibernate.wakeup.3rd_party.debugprivilege.dmp]
  117. Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.
  118.  
  119. Windows 10 Kernel Version 19041 MP (8 procs) Free x64
  120. Product: WinNt, suite: TerminalServer SingleUserTS Personal
  121. Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
  122. Kernel base = 0xfffff801`14200000 PsLoadedModuleList = 0xfffff801`14e2a230
  123. Debug session time: Thu Jun 24 04:42:52.788 2021 (UTC + 0:00) 🌽 time of crash (analyst local time)
  124. System Uptime: 6 days 17:51:30.552 🌽 uptime at the moment of crash
  125. Loading Kernel Symbols...
  126. Loading User Symbols...
  127. PEB address is NULL ! 🌽 no usermode dll symbols: no PEB[.Ldr]; that's ok
  128.  
  129.  
  130. 🌽 We got same output as before so far. Let's run the magic command [round 2].
  131. ✨✨✨✨✨✨✨✨
  132. 2: kd> !analyze -v
  133. *******************************************************************************
  134. * *
  135. * Bugcheck Analysis *
  136. * *
  137. *******************************************************************************
  138.  
  139. UNEXPECTED_STORE_EXCEPTION (154) 🌽🟦 note the bugcheck code and parameters
  140. The store component caught an unexpected exception.
  141. Arguments:
  142. Arg1: ffffb005e91ba000, Pointer to the store context or data manager 🌽🌽🏪 address of SMKM_STORE (found out below)
  143. Arg2: ffffe509f8198620, Exception information 🌽🧨🎰 address of EXCEPTION_POINTERS
  144. Arg3: 0000000000000002, Reserved 🌽🌽❓ windbg doesn't know (found out below)
  145. Arg4: 0000000000000000, Reserved
  146.  
  147. Debugging Details:
  148. ------------------
  149. KEY_VALUES_STRING: 1
  150. 🌽 this badly presented block is for the analyst box; i.e. it's extension internal debug info, ignore it
  151. Key : Analysis.CPU.mSec
  152. Value: 5561
  153. Key : Analysis.DebugAnalysisManager
  154. Value: Create
  155. Key : Analysis.Elapsed.mSec
  156. Value: 5478
  157. Key : Analysis.Init.CPU.mSec
  158. Value: 2467
  159. Key : Analysis.Init.Elapsed.mSec
  160. Value: 8304
  161. Key : Analysis.Memory.CommitPeak.Mb
  162. Value: 94
  163.  
  164. 🌽 this block is for the target box (but we've already seen that info right after loading the dump file)
  165. Key : WER.OS.Branch
  166. Value: vb_release
  167. Key : WER.OS.Timestamp
  168. Value: 2019-12-06T14:06:00Z
  169. Key : WER.OS.Version
  170. Value: 10.0.19041.1
  171.  
  172. FILE_IN_CAB: serbian.intel.hibernate.wakeup.3rd_party.debugprivilege.dmp
  173. BUGCHECK_CODE: 154
  174. BUGCHECK_P1: ffffb005e91ba000
  175. BUGCHECK_P2: ffffe509f8198620
  176. BUGCHECK_P3: 2
  177. BUGCHECK_P4: 0
  178.  
  179. EXCEPTION_RECORD: ffffe509f81995a8 -- (.exr 0xffffe509f81995a8) 🌽🧨 note the .exr XXX command
  180. ExceptionAddress: fffff801144d10d0 (nt!RtlDecompressBufferXpressLz+0x0000000000000050)
  181. ExceptionCode: c0000006 (In-page I/O error)
  182. ExceptionFlags: 00000000
  183. NumberParameters: 3
  184. Parameter[0]: 0000000000000000
  185. Parameter[1]: 0000027e296ab9f0
  186. Parameter[2]: 00000000c000000e
  187. Inpage operation failed at 0000027e296ab9f0, due to I/O error 00000000c000000e 🌽🎱 almost everything we need
  188.  
  189. EXCEPTION_PARAMETER1: 0000000000000000
  190. EXCEPTION_PARAMETER2: 0000027e296ab9f0
  191.  
  192. CONTEXT: ffffe509f8198de0 -- (.cxr 0xffffe509f8198de0) 🌽🎰 note the .cxr XXX command
  193. rax=fffff801144d1080 rbx=ffffd3011992f000 rcx=ffffd3011992f000
  194. rdx=ffffd3011992f000 rsi=0000000000000002 rdi=0000027e296ab9f0
  195. rip=fffff801144d10d0 rsp=ffffe509f81997e8 rbp=0000027e296ab9af
  196. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0
  197. r11=0000027e296aba05 r12=ffffe509f8199a58 r13=ffffb005ee4ef000
  198. r14=ffffd30119930000 r15=0000000000000000
  199. iopl=0 nv up ei pl zr na po nc
  200. cs=0010 ss=0000 ds=002b es=002b fs=0053 gs=002b efl=00050246
  201. nt!RtlDecompressBufferXpressLz+0x50:
  202. fffff801`144d10d0 418b08 mov ecx,dword ptr [r8] ds:002b:0000027e`296ab9f0=????????
  203. Resetting default scope
  204.  
  205. BLACKBOXBSD: 1 (!blackboxbsd)
  206. BLACKBOXNTFS: 1 (!blackboxntfs)
  207. BLACKBOXPNP: 1 (!blackboxpnp)
  208. BLACKBOXWINLOGON: 1
  209.  
  210. PROCESS_NAME: MemCompression 🌽⚙️ note the process name
  211. ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at 0x%p referenced memory at 0x%p. The required data was not placed into memory because of an I/O error status of 0x%x.
  212. EXCEPTION_CODE_STR: c0000006
  213. EXCEPTION_PARAMETER3: 00000000c000000e
  214. EXCEPTION_STR: 0xc0000006
  215.  
  216. STACK_TEXT: 🌽🥞 note the callstack
  217. ffffe509`f81997e8 fffff801`144a6bf0 : ffffd301`1992f000 ffffd301`1992f000 00000000`00000002 0000027e`296ab9f0 : nt!RtlDecompressBufferXpressLz+0x50
  218. ffffe509`f8199800 fffff801`144a6938 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000002 : nt!RtlDecompressBufferEx+0x60
  219. ffffe509`f8199850 fffff801`144a67c5 : 00000000`00000004 fffff801`144a632e 00000000`00000000 00000000`00000001 : nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  220. ffffe509`f8199910 fffff801`144a5ffc : 00000000`00000001 00000000`0000b9f0 ffffb005`e91ba000 ffffb005`00001000 : nt!ST_STORE<SM_TRAITS>::StDmSinglePageTransfer+0xa5
  221. ffffe509`f8199960 fffff801`144a5e2c : 00000000`ffffffff ffffb005`ee4ef000 ffffe509`f8199a40 ffffb005`f7e24710 : nt!ST_STORE<SM_TRAITS>::StDmpSinglePageRetrieve+0x180
  222. ffffe509`f8199a00 fffff801`144a5c79 : ffffd301`19bda730 00000000`00000001 00000000`00000000 00000000`00000000 : nt!ST_STORE<SM_TRAITS>::StDmPageRetrieve+0xc8
  223. ffffe509`f8199ab0 fffff801`144a5b31 : ffffb005`e91ba000 ffffb005`f7e24710 ffffb005`ee4ef000 ffffb005`e91bb9c0 : nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue+0x85
  224. ffffe509`f8199b30 fffff801`14498bc8 : ffffb005`df2c8480 ffffb005`e91ba000 00000000`00000000 ffffb005`ff1175f0 : nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadCallout+0x21
  225. ffffe509`f8199b60 fffff801`144a368f : fffff801`144a5b10 ffffe509`f8199c00 00000000`00000003 00000000`00000000 : nt!KeExpandKernelStackAndCalloutInternal+0x78
  226. ffffe509`f8199bd0 fffff801`14559934 : ffffe509`f8199cd0 00000000`31526d73 00000000`000003ff fffff801`14f239c0 : nt!SMKM_STORE<SM_TRAITS>::SmStDirectRead+0xc7
  227. ffffe509`f8199ca0 fffff801`14559368 : 00000000`0000000c 00000000`000003ff ffffe509`f8199d50 fffff801`14f239c0 : nt!SMKM_STORE<SM_TRAITS>::SmStWorkItemQueue+0x1ac
  228. ffffe509`f8199cf0 fffff801`144a4117 : 00000000`0000000c 00000000`00000001 ffffb005`f7e24710 ffffb005`ff1175f0 : nt!SMKM_STORE_MGR<SM_TRAITS>::SmIoCtxQueueWork+0xc0
  229. ffffe509`f8199d80 fffff801`144fa96b : ffffb005`00000001 ffffb005`ff1176b0 00000000`00000000 ffffb005`e91ba000 : nt!SMKM_STORE_MGR<SM_TRAITS>::SmPageRead+0x167
  230. ffffe509`f8199df0 fffff801`1445e0a0 : ffffe509`00000100 00000000`00000000 ffffe509`f819a048 fffff801`1445bff0 : nt!SmPageRead+0x33
  231. ffffe509`f8199e40 fffff801`1445bb4d : 00000000`00000002 ffffe509`f8199ed0 ffffe509`f819a048 ffffb005`ff1175a0 : nt!MiIssueHardFaultIo+0x10c
  232. ffffe509`f8199e90 fffff801`14528278 : 00000000`c0033333 00000000`00000000 ffff9c89`4f503000 00000000`00000000 : nt!MiIssueHardFault+0x29d
  233. ffffe509`f8199f50 fffff801`1442a4d2 : ffff9c89`46e00100 00000000`00000000 ffff9c89`46e00340 0078d807`00002084 : nt!MmAccessFault+0x468
  234. ffffe509`f819a0f0 fffff801`14460836 : ffffb005`f0086090 ffffe509`f819a218 ffffb005`fd4b7111 ffffb005`fdb2fc50 : nt!MiDeleteSubsectionPages+0x172
  235. ffffe509`f819a1b0 fffff801`14808d56 : 00000000`000800a1 00000000`00000000 fffff801`14e50b00 00000000`000800a1 : nt!MiDeleteSegmentPages+0x56
  236. ffffe509`f819a210 fffff801`14584a89 : 00000000`00000001 00000000`00000000 ffffe509`f819a2e0 ffffb005`f0086018 : nt!MiSegmentDelete+0x92
  237. ffffe509`f819a260 fffff801`145ba179 : 00000000`00000000 fffff801`00000001 00000000`00000000 00000000`00000000 : nt!MiProcessDereferenceList+0xc1
  238. ffffe509`f819a320 fffff801`144f53b5 : ffffb005`df2c8480 ffffb005`df2c8480 00000000`00000080 fffff801`145ba050 : nt!MiDereferenceSegmentThread+0x129
  239. ffffe509`f819a550 fffff801`145fe278 : ffffd301`19bd2180 ffffb005`df2c8480 fffff801`144f5360 00000000`00000000 : nt!PspSystemThreadStartup+0x55
  240. ffffe509`f819a5a0 00000000`00000000 : ffffe509`f819b000 ffffe509`f8194000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28
  241.  
  242. SYMBOL_NAME: nt!RtlDecompressBufferXpressLz+50
  243. MODULE_NAME: nt
  244. IMAGE_NAME: ntkrnlmp.exe
  245. STACK_COMMAND: .cxr 0xffffe509f8198de0 ; kb
  246. BUCKET_ID_FUNC_OFFSET: 50
  247. FAILURE_BUCKET_ID: 0x154_c0000006_c000000e_nt!RtlDecompressBufferXpressLz
  248. OS_VERSION: 10.0.19041.1
  249. BUILDLAB_STR: vb_release
  250. OSPLATFORM_TYPE: x64
  251. OSNAME: Windows 10
  252. FAILURE_ID_HASH: {04bbc5ba-a28a-9efe-b2fd-8956bfd9dd67}
  253. Followup: MachineOwner
  254. ---------
  255.  
  256.  
  257. ✨✨✨✨✨✨✨✨
  258. 🌽 What do we have here? Command "!analyze -v" has extracted significant information from the dump for us.
  259.  
  260. 🌽🟦 Bugcheck code: 0x154, UNEXPECTED_STORE_EXCEPTION. Let's duplicate that piece of output for quick reference:
  261. """
  262. UNEXPECTED_STORE_EXCEPTION (154) 🌽🟦
  263. The store component caught an unexpected exception.
  264. Arguments:
  265. Arg1: ffffb005e91ba000, Pointer to the store context or data manager 🌽🌽🏪 address of SMKM_STORE (found out below)
  266. Arg2: ffffe509f8198620, Exception information 🌽🧨🎰 address of EXCEPTION_POINTERS
  267. Arg3: 0000000000000002, Reserved 🌽🌽❓ windbg doesn't know (found out below)
  268. Arg4: 0000000000000000, Reserved
  269. """
  270. 🌽 The bugcheck code itself doesn't immediatelly tell us much.
  271. We can lookup more info in Windbg HELP.CHM: press F1 and search, or simply type windbg command `.hh bug check 0x154`.
  272. However, that won't give us much either, as the help file doesn't have any extra info compared to the already displayed.
  273.  
  274. Function KeBugCheckEx has 4 [arbitrary] arguments beside the bugcheck code, and some Args have non-zero values here.
  275. Arg1. Pointer to SMKM_STORE object (we actually don't know it, but willl find out soon; well, doesn't matter anyway).
  276. Arg2. That's an address of EXCEPTION_POINTERS: simply a container for 🧨 EXCEPTION_RECORD and 🎰 CONTEXT pointers.
  277. Note "!analyze -v" has already extracted these pointers (you can run "dps ffffe509f8198620 L2" and they'll match).
  278. Arg3. We don't know what "2" means. We can disasm ntoskrnl and find out (look for KeBugCheckEx with code 0x154),
  279. but right now there's no point doing that, as we have much better options to explore.
  280.  
  281. The main takeaway right now is the description string "The store component caught an unexpected exception".
  282. Looking at the 🥞callstack, it makes sense - there is indeed "ST_STORE" and "SMKM_STORE" 🏪 object methods there,
  283. so let's move to explore the exception which the store component couldn't handle.
  284.  
  285.  
  286. ✨✨✨✨✨✨✨✨
  287. 🌽 Let's rerun 🧨 ".exr XXX" to avoid scrolling up to "!analyze -v" output.
  288.  
  289. 🌽🧨
  290. 2: kd> .exr 0xffffe509f81995a8 🌽🧨
  291. ExceptionAddress: fffff801144d10d0 (nt!RtlDecompressBufferXpressLz+0x0000000000000050)
  292. ExceptionCode: c0000006 (In-page I/O error) 🟥🔴🍎 nobody expects the STATUS_IN_PAGE_ERROR
  293. ExceptionFlags: 00000000
  294. NumberParameters: 3 🌽 rdx: KiExceptionDispatch extra args count (in r9, r10, r11)
  295. Parameter[0]: 0000000000000000 🌽 r9, xarg1: mem op, from CPU-pushed #PF code (mask 1: W; 8: X)
  296. Parameter[1]: 0000027e296ab9f0 🌽 r10, xarg2: address referenced which caused fault
  297. Parameter[2]: 00000000c000000e 🌽🍄 r11, xarg3: error code returned from MmAccessFault
  298. Inpage operation failed at 0000027e296ab9f0, due to I/O error 00000000c000000e 🌽🎱 almost everything we needed
  299.  
  300. 🌽 ^^ So something needed a page-in operation (previously paged out data needed to be read), but for some reason that
  301. operation failed. STATUS_IN_PAGE_ERROR (C0000006) is a rare exception. 🟥🔴🍎 In fact, its chief weapon is surprise!
  302. Some situations when you can see it:
  303. * Executable image was mapped over the network; app tried to access currently missing page, but network got unplugged.
  304. * The app run from a removable media; user removed the media, then app accessed some absent page. [Note both network
  305. and removable media failures can be mitigated by special PE header flags, forcing full load and pagefile use later].
  306. * App used Jonas Lyk rad :bbq self-deletion method; then OS dropped some image pages (to read 'em from the image later);
  307. app accesses absent page with no file to read it from. [You've used the method and didn't know that could happen? Ha!]
  308. * Disk read failed for some reason.
  309.  
  310. Now, what is the low-level sequence for in-page error? Let's see abridged flow for x64 (for ARM64 it's similar anyway).
  311. 1. CPU executes code which causes it to access a memory page with bit PTE.Valid == 0.
  312. 2. Page fault happens (#PF). Windows pagefault handler is nt!KiPageFault. That's very low-level function, written
  313. in assembly. It's also quite small, and delegates real fault resolution work to high-level func nt!MmAccessFault.
  314. 3. MmAccessFault attempts to resolves the fault. In most cases it's a soft pagefault (so its job is "PTE.Valid := 1"),
  315. but sometimes it's a hard pagefault, requiring data reading from disk/network. In any case, MmAccessFault returns
  316. NTSTATUS to KiPageFault. Succeeded? All good, continue execution. Failed? KiPageFault invokes KiExceptionDispatch.
  317. 4. KiExceptionDispatch is a low-level function, invoked from multiple fault handlers; arguments:
  318. ecx: exception code (for KiPageFault its derived from the code MmAccessFault returned); edx: number of extra args;
  319. r8: exception address (rip of the faulting instruction); r9, r10, r11: extra args.
  320.  
  321. Of course, all that info is not that important, but it helps to know the details when looking at 🧨".exr XXX" output.
  322. If you check KiPageFault disasm, you'll see it discerns only like 5 error codes from MmAccessFault. For codes it doesn't
  323. know/care about, it simply sets KiExceptionDispatch ecx arg (the error code) to C0000006 (STATUS_IN_PAGE_ERROR). The
  324. orig error code from MmAccessFault is still retained as the third extra argument for KiExceptionDispatch (in r11).
  325. 💎 What it means: the status we really need to examine is in the extra xarg3: 🍄 C000000E, STATUS_NO_SUCH_DEVICE.
  326.  
  327.  
  328. ✨✨✨✨✨✨✨✨
  329. 🌽 Let's check 🎰 context/disasm around the faulting instruction, along with the 🥞 callstack.
  330.  
  331. 2: kd> .cxr 0xffffe509f8198de0 🌽🎰
  332. rax=fffff801144d1080 rbx=ffffd3011992f000 rcx=ffffd3011992f000
  333. rdx=ffffd3011992f000 rsi=0000000000000002 rdi=0000027e296ab9f0
  334. rip=fffff801144d10d0 rsp=ffffe509f81997e8 rbp=0000027e296ab9af
  335. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0
  336. r11=0000027e296aba05 r12=ffffe509f8199a58 r13=ffffb005ee4ef000
  337. r14=ffffd30119930000 r15=0000000000000000
  338. iopl=0 nv up ei pl zr na po nc
  339. cs=0010 ss=0000 ds=002b es=002b fs=0053 gs=002b efl=00050246
  340. nt!RtlDecompressBufferXpressLz+0x50:
  341. fffff801`144d10d0 418b08 mov ecx,dword ptr [r8] ds:002b:0000027e`296ab9f0=????????
  342.  
  343. 🌽 ^^ That's the CONTEXT record OS helpfully saved at the time of exception.
  344. Note ".cxr XXX" command doesn't just display the context, it actually sets ("activates") it for the debugger. Thus
  345. subsequent commands (like "k" would use this context, and not the real CPU context).
  346.  
  347.  
  348. ✨✨✨
  349. 🌽 Continue by quickly checking the disasm around the faulted "mov ecx, [r8]" instruction.
  350. Huge 0x0E-byte nop right before: faulted instruction is at the start of a cycle. Nothing stands out so far.
  351.  
  352. 2: kd> ub . L7 🌽 Unassemble Backward from current point ("."), Length: 7 instructions
  353. nt!RtlDecompressBufferXpressLz+0x2b:
  354. fffff801`144d10ae 458bd9 mov r11d,r9d
  355. fffff801`144d10b1 4d03d8 add r11,r8
  356. fffff801`144d10b4 4533ff xor r15d,r15d
  357. fffff801`144d10b7 4d8d96a0feffff lea r10,[r14-160h]
  358. fffff801`144d10be 498d6baa lea rbp,[r11-56h]
  359. fffff801`144d10c2 0f1f4000 nop dword ptr [rax]
  360. fffff801`144d10c6 66660f1f840000000000 nop word ptr [rax+rax]
  361.  
  362. 2: kd> u . L3
  363. nt!RtlDecompressBufferXpressLz+0x50:
  364. fffff801`144d10d0 418b08 mov ecx,dword ptr [r8]
  365. fffff801`144d10d3 4983c004 add r8,4
  366. fffff801`144d10d7 493bd2 cmp rdx,r10
  367.  
  368.  
  369. ✨✨✨
  370. 🌽 Overview the 🥞 callstack at the time of fault.
  371.  
  372. 2: kd> k 🌽🥞
  373. *** Stack trace for last set context - .thread/.cxr resets it
  374. # Child-SP RetAddr Call Site
  375. 00 ffffe509`f81997e8 fffff801`144a6bf0 nt!RtlDecompressBufferXpressLz+0x50
  376. 01 ffffe509`f8199800 fffff801`144a6938 nt!RtlDecompressBufferEx+0x60
  377. 02 ffffe509`f8199850 fffff801`144a67c5 nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  378. 03 ffffe509`f8199910 fffff801`144a5ffc nt!ST_STORE<SM_TRAITS>::StDmSinglePageTransfer+0xa5
  379. 04 ffffe509`f8199960 fffff801`144a5e2c nt!ST_STORE<SM_TRAITS>::StDmpSinglePageRetrieve+0x180
  380. 05 ffffe509`f8199a00 fffff801`144a5c79 nt!ST_STORE<SM_TRAITS>::StDmPageRetrieve+0xc8
  381. 06 ffffe509`f8199ab0 fffff801`144a5b31 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue+0x85
  382. 07 ffffe509`f8199b30 fffff801`14498bc8 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadCallout+0x21
  383. 08 ffffe509`f8199b60 fffff801`144a368f nt!KeExpandKernelStackAndCalloutInternal+0x78
  384. 09 ffffe509`f8199bd0 fffff801`14559934 nt!SMKM_STORE<SM_TRAITS>::SmStDirectRead+0xc7
  385. 0a ffffe509`f8199ca0 fffff801`14559368 nt!SMKM_STORE<SM_TRAITS>::SmStWorkItemQueue+0x1ac
  386. 0b ffffe509`f8199cf0 fffff801`144a4117 nt!SMKM_STORE_MGR<SM_TRAITS>::SmIoCtxQueueWork+0xc0
  387. 0c ffffe509`f8199d80 fffff801`144fa96b nt!SMKM_STORE_MGR<SM_TRAITS>::SmPageRead+0x167
  388. 0d ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  389. 0e ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  390. 0f ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  391. 10 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468 🌽🐦 "fake" fault (described below)
  392. 11 ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  393. 12 ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56
  394. 13 ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92
  395. 14 ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1
  396. 15 ffffe509`f819a320 fffff801`144f53b5 nt!MiDereferenceSegmentThread+0x129
  397. 16 ffffe509`f819a550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  398. 17 ffffe509`f819a5a0 00000000`00000000 nt!KiStartSystemThread+0x28
  399.  
  400.  
  401. ✨✨✨
  402. 🌽 But let's reset the debugger context using argless ".cxr" and overview the 🥞 callstack again.
  403. All frame numbers we reference below would be for *this* callstack rather than the partial one above.
  404.  
  405. 2: kd> .cxr 🌽 reset the debugger context
  406. 2: kd> k 🌽🥞
  407. # Child-SP RetAddr Call Site
  408. 00 ffffe509`f8198568 fffff801`14794dee nt!KeBugCheckEx 🌽🟦
  409. 01 ffffe509`f8198570 fffff801`14610023 nt!SMKM_STORE<SM_TRAITS>::SmStUnhandledExceptionFilter+0x7e
  410. 02 ffffe509`f81985c0 fffff801`145cca4f nt!`SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue'::`1'::filt$0+0x22 🌽🏢
  411. 03 ffffe509`f81985f0 fffff801`145ffadf nt!_C_specific_handler+0x9f
  412. 04 ffffe509`f8198660 fffff801`14487547 nt!RtlpExecuteHandlerForException+0xf
  413. 05 ffffe509`f8198690 fffff801`14486136 nt!RtlDispatchException+0x297
  414. 06 ffffe509`f8198db0 fffff801`14608cac nt!KiDispatchException+0x186
  415. 07 ffffe509`f8199470 fffff801`14604e43 nt!KiExceptionDispatch+0x12c
  416. 08 ffffe509`f8199650 fffff801`144d10d0 nt!KiPageFault+0x443 🌽📝 called "real" MmAccessFault (it has returned)
  417. 09 ffffe509`f81997e8 fffff801`144a6bf0 nt!RtlDecompressBufferXpressLz+0x50 🌽💥
  418. 0a ffffe509`f8199800 fffff801`144a6938 nt!RtlDecompressBufferEx+0x60
  419. 0b ffffe509`f8199850 fffff801`144a67c5 nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  420. 0c ffffe509`f8199910 fffff801`144a5ffc nt!ST_STORE<SM_TRAITS>::StDmSinglePageTransfer+0xa5
  421. 0d ffffe509`f8199960 fffff801`144a5e2c nt!ST_STORE<SM_TRAITS>::StDmpSinglePageRetrieve+0x180
  422. 0e ffffe509`f8199a00 fffff801`144a5c79 nt!ST_STORE<SM_TRAITS>::StDmPageRetrieve+0xc8
  423. 0f ffffe509`f8199ab0 fffff801`144a5b31 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue+0x85 🌽🏢
  424. 10 ffffe509`f8199b30 fffff801`14498bc8 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadCallout+0x21
  425. 11 ffffe509`f8199b60 fffff801`144a368f nt!KeExpandKernelStackAndCalloutInternal+0x78
  426. 12 ffffe509`f8199bd0 fffff801`14559934 nt!SMKM_STORE<SM_TRAITS>::SmStDirectRead+0xc7
  427. 13 ffffe509`f8199ca0 fffff801`14559368 nt!SMKM_STORE<SM_TRAITS>::SmStWorkItemQueue+0x1ac
  428. 14 ffffe509`f8199cf0 fffff801`144a4117 nt!SMKM_STORE_MGR<SM_TRAITS>::SmIoCtxQueueWork+0xc0
  429. 15 ffffe509`f8199d80 fffff801`144fa96b nt!SMKM_STORE_MGR<SM_TRAITS>::SmPageRead+0x167
  430. 16 ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  431. 17 ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  432. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  433. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468 🌽🐦 "fake" fault (described below)
  434. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  435. 1b ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56
  436. 1c ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92
  437. 1d ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1 🌽🗑️
  438. 1e ffffe509`f819a320 fffff801`144f53b5 nt!MiDereferenceSegmentThread+0x129
  439. 1f ffffe509`f819a550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  440. 20 ffffe509`f819a5a0 00000000`00000000 nt!KiStartSystemThread+0x28
  441.  
  442. 🌽 ^^ So far so good. We see RtlDecompressBufferXpressLz at frame 9 tried to access page with PTE.Valid == 0, which
  443. caused #PF; CPU consulted its Interrupt Descriptor Table to find the #PF handler, which is KiPageFault. As previously
  444. noted, KiPageFault invoked MmAccessFault to handle the fault, but that failed with STATUS_NO_SUCH_DEVICE (C000000E).
  445. [Ofc MmAccessFault has already returned and thus absent in the callstack; the one you see at frame 0x19 is different].
  446. So KiPageFault invoked KiExceptionDispatch, which tried to dispatch the exception using SEH frames. It so happened
  447. 🏢 SmStDirectReadIssue at frame 0x0F had SEH frame (try-except) block, with quite simple exception filter (frame 2).
  448.  
  449. I wouldn't normally decompile bugchecked func in such simple cases, but let's do it for a better overview.
  450. I've used ntoskrnl.exe of similar build, but you can use the exact version - windbg should have already downloaded the
  451. exact bin from the MS symbol server and saved it to your symstore (defined by _NT_SYMBOL_PATH envar). Example location:
  452. C:\stuff\symbols\ms\ntkrnlmp.exe\05C0E3E01046000\ntkrnlmp.exe; `05C0E3E0` is PE.timestamp, [0]1046000 is PE.SizeOfImage.
  453. If for some reason binary wasn't downloaded yet, you can force download using `!chkimg` command, e.g. `!chkimg nt`.
  454.  
  455. ```
  456. // function with SEH, frame 0x0F 🌽🏢
  457. NTSTATUS SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue(...)
  458. {
  459. ...
  460. __try
  461. {
  462. ...
  463. // in this case 💥 RtlDecompressBufferXpressLz (frame 9) gets invoked eventually, and Page Fault happens
  464. ...
  465. }
  466. __except(this->SmStUnhandledExceptionFilter(GetExceptionPointers(), FaultingFunc=2`SmStDirectReadIssue))
  467. {
  468. NOTHING;
  469. }
  470. ...
  471. }
  472.  
  473.  
  474. // entire func which invoked KeBugCheckEx, frame 0x02 🌽🏢; used as filter from 4 different "FaultingFunc" methods
  475. int SMKM_STORE<SM_TRAITS>::SmStUnhandledExceptionFilter(this, EXCEPTION_POINTERS* ExceptionPointers, int FaultingFunc)
  476. {
  477. if (FaultingFunc == 1`SmStWorkerThread) // only process it if faulting func is SmStWorkerThread (not our case)
  478. {
  479. int exceptionCode = ExceptionPointers->ExceptionRecord->ExceptionCode;
  480. if (exceptionCode == 0xC0000005`STATUS_ACCESS_VIOLATION || exceptionCode == 0xC0000420`STATUS_ASSERTION_FAILURE)
  481. {
  482. __try
  483. {
  484. SmHpChunkHeapProtect(this + 0x110, 2);
  485. }
  486. __except(EXCEPTION_EXECUTE_HANDLER)
  487. {
  488. NOTHING;
  489. }
  490. }
  491. }
  492. if (ExceptionPointers) // that's always true
  493. KeBugCheckEx(0x154, this, ExceptionPointers, FaultingFunc, 0); 🌽🟦 bugcheck Arg3 🌽❓ mystery revealed
  494. return EXCEPTION_CONTINUE_SEARCH; // that's never executed
  495. }
  496. ```
  497. 🌽 ^^ That's actually really weird exception filter. Normally neither user nor kernel code should be written like that.
  498. IDA shows the filter is used in 4 different places, and in all cases it tries to catch/eat all exceptions, w/o giving
  499. higher-level components chance to handle them. Well, that's probably okay (it enriches the bugcheck with 'this' pointer,
  500. i.e. address of the SMKM_STORE object); but filter tries to process certain fails using SmHpChunkHeapProtect, yet it
  501. still invokes KeBugCheckEx basically unconditionally! And returns EXCEPTION_CONTINUE_SEARCH as if it handled nothing.
  502. 💎 The exception filter seems like a wreckorama, but it's not the BSOD root cause. Continue investigation.
  503.  
  504.  
  505. ✨✨✨✨✨✨✨✨
  506. 🌽 We need the faulting ⚙️ process name for a better reflection.
  507.  
  508. 2: kd> !process -1 0 🌽⚙️ using "-1" to display current process
  509. PROCESS ffffb005e91bc040
  510. SessionId: none Cid: 0c0c Peb: 00000000 ParentCid: 0004
  511. DirBase: 1a7f49000 ObjectTable: ffff9c894d4f4500 HandleCount: 0.
  512. Image: MemCompression ⚙️⚙️⚙️
  513.  
  514. 2: kd> ?? @$proc->Minimal 🌽 current EPROCESS.Minimal == 1
  515. unsigned long 1
  516.  
  517.  
  518. ✨✨✨✨✨✨✨✨
  519. 🌽 Okay, CPU was running kernelmode code in the context of the MemCompression process. What is that process?
  520. You guessed it: the MemCompression process is a process which compresses memory! 🤯
  521. It is a *Minimal* process: a real process with its own address space. Its usermode virtual address space part is
  522. separate from the System process (different KPROCESS.DirectoryTableBase, aka DirBase, aka CR3 register).
  523. Since it's a Minimal process, it doesn't have the usual usermode stuff, like mapped ntdll, KUSER_SHARED_DATA, or PEB.
  524.  
  525. The MS assumption is that it's faster to decompress a page than to read it from disk. However, it's not that simple and
  526. is not always an easy win. In case of disk read the CPU is free to do something else while disk device reads data, while
  527. decompression is intense on CPU usage/power/cache. There are also no transition pages parallel to compressed ones (as
  528. that'll just increase the RAM usage), so no fast/soft pagefaults for compressed pages, sometimes making them... slower?
  529. But it does save memory, and still is a win in many scenarios (just not in all of them). Also note the beauty:
  530. compressed data itself can be paged out to disk.
  531.  
  532. You can read a bit more about memory compression in Windows Internals, "Memory compression" section. But there's
  533. more: see "2019.Extracting Compressed Pages from the Windows 10 Virtual Store.Sardar.[MemCompression].pdf".
  534.  
  535. ✨✨✨
  536. 🌽 So, what was the kernel thread in the context of MemCompression doing anyway?
  537. Here's the duplicated callstack portion, to avoid wearing down the encoder of your mouse scrollwheel:
  538.  
  539. 00 ffffe509`f8198568 fffff801`14794dee nt!KeBugCheckEx
  540. ...
  541. 08 ffffe509`f8199650 fffff801`144d10d0 nt!KiPageFault+0x443
  542. 09 ffffe509`f81997e8 fffff801`144a6bf0 nt!RtlDecompressBufferXpressLz+0x50 🌽💥
  543. 0a ffffe509`f8199800 fffff801`144a6938 nt!RtlDecompressBufferEx+0x60
  544. 0b ffffe509`f8199850 fffff801`144a67c5 nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  545. ...
  546. 16 ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  547. 17 ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  548. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  549. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468 🌽🐦 "fake" fault (described below)
  550. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  551. 1b ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56
  552. 1c ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92
  553. 1d ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1 🌽🗑️
  554. 1e ffffe509`f819a320 fffff801`144f53b5 nt!MiDereferenceSegmentThread+0x129
  555. 1f ffffe509`f819a550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  556. 20 ffffe509`f819a5a0 00000000`00000000 nt!KiStartSystemThread+0x28
  557.  
  558. 🌽 From the looks of it, thread was... Deleting a segment? I.e. deleting a section object, I guess?
  559. But is that right? Why would MemCompression process would be deleting a segment? Ah, of course. Let's see:
  560. ✨✨✨
  561. 2: kd> !thread -1 🌽 using "-1" to display current thread
  562. THREAD ffffb005df2c8480 Cid 0004.01c0 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2
  563. Not impersonating
  564. Owning Process ffffb005c509f100 Image: System 🌽 thread reallly belongs to System, but
  565. Attached Process ffffb005e91bc040 Image: MemCompression 🌽 was KeStackAttach'd to MemCompression
  566. Wait Start TickCount 37292117 Ticks: 78 (0:00:00:01.218)
  567. Context Switch Count 8329 IdealProcessor: 3
  568. UserTime 00:00:00.000
  569. KernelTime 00:00:03.328
  570. Win32 Start Address nt!MiDereferenceSegmentThread (0xfffff801145ba050)
  571. Stack Init ffffe509f819a5d0 Current ffffe509f8198ef0
  572. Base ffffe509f819b000 Limit ffffe509f8194000 Call 0000000000000000
  573. Priority 19 BasePriority 8 IoPriority 2 PagePriority 5
  574.  
  575. ✨✨✨
  576. So yeah, it makes more sense now. Thread in the System process was deleting a section object, and simply attached the
  577. address space to the MemCompression process at certain point. If you were working at Microsoft at the time, you might
  578. have seen evil internal memos: memory compression was too good, so to force users to upgrade their hardware Microsoft
  579. slowed it down by separating it into own process (to incur more TLB flushes). Same with the Registry process, btw.
  580.  
  581. 🌽 Anyway, at certain point thread issued a fake/manual 🌽🐦 MmAccessFault (frame 0x1A to 0x19).
  582. That looks a bit sus, let's verify:
  583. ✨✨✨
  584. 2: kd> .cxr 🌽 reset the debugger context, just in case
  585. 2: kd> .frame /c /r 1A 🌽 perform virtual unwind to frame 0x1A
  586. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  587. rax=fffff801144d1080 rbx=0000000000000000 rcx=ffffd3011992f000 🌽 it is NOT a full context: we asked debugger to do
  588. rdx=ffffd3011992f000 rsi=ffff9c894f503000 rdi=ffffe509f819a218 🌽 a Virtual Unwind using ".frame /c /r #", so only
  589. rip=fffff8011442a4d2 rsp=ffffe509f819a0f0 rbp=0000000000000000 🌽 NON-volatile regs are valid; there's no way to
  590. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0 🌽 restore volatile regs like rax, so debugger has
  591. r11=0000027e296aba05 r12=0000000000000000 r13=ffffb005f0086010 🌽 set them to values from in the closest TrapFrame,
  592. r14=0000000000000000 r15=0000000000000000 🌽 address of which can be seen with "kv" command
  593. iopl=0 nv up ei pl zr na po nc
  594. cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00040246
  595. nt!MiDeleteSubsectionPages+0x172:
  596. fffff801`1442a4d2 eb90 jmp nt!MiDeleteSubsectionPages+0x104 (fffff801`1442a464)
  597.  
  598. ✨✨✨
  599. 2: kd> ub . LD 🌽 check disasm (Unassemble Backwards), 0xD opcodes
  600. nt!MiDeleteSubsectionPages+0x145:
  601. fffff801`1442a4a5 a802 test al,2
  602. fffff801`1442a4a7 752b jne nt!MiDeleteSubsectionPages+0x174 (fffff801`1442a4d4)
  603. fffff801`1442a4a9 488d9424d0000000 lea rdx,[rsp+0D0h]
  604. fffff801`1442a4b1 488bce mov rcx,rsi 🌽 rcx: address of proto _MMPTE
  605. fffff801`1442a4b4 e827faffff call nt!MiLockProtoPoolPage (fffff801`14429ee0) 🌽🔒 this failed ->...
  606. fffff801`1442a4b9 488be8 mov rbp,rax 🌽 rax: address of locked _MMPFN
  607. fffff801`1442a4bc 4885c0 test rax,rax 🌽 (but this func failed: rax==0)
  608. fffff801`1442a4bf 7531 jne nt!MiDeleteSubsectionPages+0x192 (fffff801`1442a4f2)
  609. fffff801`1442a4c1 4533c9 xor r9d,r9d
  610. fffff801`1442a4c4 8d4802 lea ecx,[rax+2] 🌽 ecx: 2, MiFaultWrite
  611. fffff801`1442a4c7 4533c0 xor r8d,r8d
  612. fffff801`1442a4ca 488bd6 mov rdx,rsi 🌽 rdx: address of proto _MMPTE
  613. fffff801`1442a4cd e83ed90f00 call nt!MmAccessFault (fffff801`14527e10) 🌽🐦
  614.  
  615. 🌽 ^^ What do you know? In this case we can easily reconstruct all four args to the "fake" 🌽🐦 MmAccessFault, just
  616. from this tiny disasm snippet and unwound non-volatile regs alone. [Well, strictly speaking we might need more than this
  617. disasm, but if you issue "uf nt!MiDeleteSubsectionPages" (Unassemble Function), you'll see basic blocks are in order].
  618. 💎🌽🐦 MmAccessFault(FaultStatus=MiFaultWrite`2, VirtualAddress=ffff9c894f503000, PreviousMode=Kernel, TrapInfo=null).
  619.  
  620. 🌽 As noted in the snippet comments above, the MiLockProtoPoolPage intakes address of the prototype PTE (_MMPTE), and
  621. returns address of the corresponding _MMPFN, now locked. But in this case MiLockProtoPoolPage failed, returning null.
  622. That's a normal situation, e.g. it can happen if *address* of the proto PTE itself is not valid (or proto PTE is in
  623. copy-on-write memory). To resolve that, oskernel invokes 🌽🐦 MmAccessFault() to basically page-in the prototype PTE.
  624. cyra> Do you want me to show what section was being deleted?
  625.  
  626. 🌽 As you'll see shortly, the section being deleted is not that relevat, but just for kicks let's check section info.
  627. ✨✨✨
  628. 2: kd> .frame /c /r 1d 🌽🗑️ do virtual unwind to frame 0x1D
  629. 1d ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1
  630. rax=fffff801144d1080 rbx=0000000000000000 rcx=ffffd3011992f000 🌽 got valid non-volatile regs
  631. rdx=ffffd3011992f000 rsi=ffffb005f0086018 rdi=fffff80114e50b00
  632. rip=fffff80114584a89 rsp=ffffe509f819a260 rbp=ffffe509f819a2e0
  633. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0
  634. r11=0000027e296aba05 r12=000000000000031f r13=fffff80114e51040
  635. r14=0000000000000000 r15=fffff80114e51110
  636. iopl=0 nv up ei pl zr na po nc
  637. cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00040246
  638. nt!MiProcessDereferenceList+0xc1:
  639. fffff801`14584a89 498bcd mov rcx,r13
  640.  
  641. 2: kd> ub . L2 🌽 disasm around the call
  642. nt!MiProcessDereferenceList+0xb8:
  643. fffff801`14584a80 488d4ef8 lea rcx,[rsi-8] 🌽🏞️ rcx: address of _CONTROL_AREA
  644. fffff801`14584a84 e83b422800 call nt!MiSegmentDelete
  645.  
  646. 🌽 ^^ Again, it was very easy to find out the argument [for the MiSegmentDelete]. Sometimes it's not that easy, and you
  647. gotta check multiple frames, or do some very hard add/sub math to find stack slot location for the spilled argument.
  648.  
  649. 🌽 Argument info for MiSgmentDelete was in my training set, so I know the first argument is pointer to _CONTROL_AREA.
  650. Let's assign it to $t0 pseudo-register for later use; go with C++ syntax (with type info) via "r?" instead of plain "r".
  651. ✨✨✨
  652. 2: kd> r?$t0=(nt!_control_area*)(@rsi-8) 🌽🏞️ use non-vol rsi, not rcx
  653. 2: kd> ?? @$t0 🌽 and dump the CONTROL_AREA immediately
  654. struct _CONTROL_AREA * 0xffffb005`f0086010
  655. +0x000 Segment : 0xffff9c89`6570e840 _SEGMENT
  656. +0x008 ListHead : _LIST_ENTRY [ 0xffffb005`f0086018 - 0xffffb005`f0086018 ]
  657. +0x008 AweContext : 0xffffb005`f0086018 Void
  658. +0x018 NumberOfSectionReferences : 0
  659. +0x020 NumberOfPfnReferences : 0
  660. +0x028 NumberOfMappedViews : 0
  661. +0x030 NumberOfUserReferences : 0
  662. +0x038 u : <anonymous-tag> 🌽🖱️ click this link
  663. +0x03c u1 : <anonymous-tag>
  664. +0x040 FilePointer : _EX_FAST_REF 🌽🗄️ +40, CONTROL_AREA.FilePointer
  665. +0x048 ControlAreaLock : 0n0
  666. +0x04c ModifiedWriteCount : 0
  667. +0x050 WaitList : (null)
  668. +0x058 u2 : <anonymous-tag>
  669. +0x068 FileObjectLock : _EX_PUSH_LOCK
  670. +0x070 LockedPages : 1
  671. +0x078 u3 : <anonymous-tag>
  672.  
  673. 🌽 ^^ You don't see it here, but a lot of members above are actually clickable links in Windbg. So don't freak out
  674. by the couple of commands below, I've just clicked the links to "enter" them.
  675.  
  676. ✨✨✨
  677. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_CONTROL_AREA *)0xffffb005f0086010)).u 🌽🖱️ clicked "u" above
  678. (*((ntkrnlmp!_CONTROL_AREA *)0xffffb005f0086010)).u [Type: <anonymous-tag>]
  679. [+0x000] LongFlags : 0x800a1 [Type: unsigned long]
  680. [+0x000] Flags [Type: _MMSECTION_FLAGS] 🌽🖱️ clicked this link
  681.  
  682. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMSECTION_FLAGS *)0xffffb005f0086048)) 🌽 got _MMSECTION_FLAGS
  683. (*((ntkrnlmp!_MMSECTION_FLAGS *)0xffffb005f0086048)) [Type: _MMSECTION_FLAGS]
  684. [+0x000 ( 0: 0)] BeingDeleted : 0x1 [Type: unsigned int] 🌽 correct, the section is being deleted!
  685. [+0x000 ( 1: 1)] BeingCreated : 0x0 [Type: unsigned int]
  686. [+0x000 ( 2: 2)] BeingPurged : 0x0 [Type: unsigned int]
  687. [+0x000 ( 3: 3)] NoModifiedWriting : 0x0 [Type: unsigned int]
  688. [+0x000 ( 4: 4)] FailAllIo : 0x0 [Type: unsigned int]
  689. [+0x000 ( 5: 5)] Image : 0x1 [Type: unsigned int] 🌽 it's an image section (exe/dll)
  690. [+0x000 ( 6: 6)] Based : 0x0 [Type: unsigned int]
  691. [+0x000 ( 7: 7)] File : 0x1 [Type: unsigned int]
  692. [+0x000 ( 8: 8)] AttemptingDelete : 0x0 [Type: unsigned int]
  693. [+0x000 ( 9: 9)] PrefetchCreated : 0x0 [Type: unsigned int]
  694. [+0x000 (10:10)] PhysicalMemory : 0x0 [Type: unsigned int]
  695. [+0x000 (11:11)] ImageControlAreaOnRemovableMedia : 0x0 [Type: unsigned int]
  696. [+0x000 (12:12)] Reserve : 0x0 [Type: unsigned int]
  697. [+0x000 (13:13)] Commit : 0x0 [Type: unsigned int]
  698. [+0x000 (14:14)] NoChange : 0x0 [Type: unsigned int]
  699. [+0x000 (15:15)] WasPurged : 0x0 [Type: unsigned int]
  700. [+0x000 (16:16)] UserReference : 0x0 [Type: unsigned int]
  701. [+0x000 (17:17)] GlobalMemory : 0x0 [Type: unsigned int]
  702. [+0x000 (18:18)] DeleteOnClose : 0x0 [Type: unsigned int]
  703. [+0x000 (19:19)] FilePointerNull : 0x1 [Type: unsigned int]
  704. [+0x000 (25:20)] PreferredNode : 0x0 [Type: unsigned long]
  705. [+0x000 (26:26)] GlobalOnlyPerSession : 0x0 [Type: unsigned int]
  706. [+0x000 (27:27)] UserWritable : 0x0 [Type: unsigned int]
  707. [+0x000 (28:28)] SystemVaAllocated : 0x0 [Type: unsigned int]
  708. [+0x000 (29:29)] PreferredFsCompressionBoundary : 0x0 [Type: unsigned int]
  709. [+0x000 (30:30)] UsingFileExtents : 0x0 [Type: unsigned int]
  710. [+0x000 (31:31)] PageSize64K : 0x0 [Type: unsigned int]
  711.  
  712. 2: kd> dq @@(&@$t0->FilePointer) L1 🌽🗄️ +40, CONTROL_AREA.FilePointer
  713. ffffb005`f0086050 ffffb005`f611bcd6
  714. 🌽 CONTROL_AREA.FilePointer is actually EX_FAST_REF (pointer-with-embedded-refcount), so align it down by 0x10
  715. 2: kd> r?$t1=(nt!_file_object*)(@$p&-0x10) 🌽 use @$p to access the last db/dw/dd/dq value
  716.  
  717. 2: kd> !fileobj @$t1 🌽 or we could've used "?? @$t1" for entire struct
  718. \Program Files\WindowsApps\Microsoft.OneConnect_5.2011.3081.0_x64__8wekyb3d8bbwe\OneConnect.dll 💎
  719. Device Object: 0xffffb005cc3f1c00 \Driver\volmgr 🌽 ^^^ got filepath
  720. Vpb: 0xffffb005cc3d5cd0
  721. Access: Read SharedRead SharedDelete 🌽 ...got share access
  722. Flags: 0x44042 🌽 ...and got decoded flags
  723. Synchronous IO
  724. Cache Supported
  725. Cleanup Complete
  726. Handle Created
  727. FsContext: 0xffff9c8959df69f0 FsContext2: 0xffff9c8967d8c640
  728. CurrentByteOffset: 0
  729. Cache Data:
  730. Section Object Pointers: ffffb005f90731d8
  731. Shared Cache Map: 00000000
  732.  
  733. 🌽 Now we know the filepath for the section which was being deleted! That's very nice.
  734. Just think how useful it is! Well, for this analysis it's absolutely useless. Hahaha.
  735.  
  736. 🌽 Since we've already sidetracked a bit, let's also check something else related to section objects.
  737. Right beyond the _CONTROL_AREA for section image there's a subsection (not a coincidence, it's designed that way).
  738. Or rather, there are multiple subsections: one for the file header (MZ header), and one for each image section.
  739. Let's display subsection for the file header (with MZ signature):
  740. ✨✨✨
  741. 2: kd> dt _subsection @@(@$t0+1) 🌽 C++ expression: right after the CONTROL_AREA
  742. nt!_SUBSECTION
  743. +0x000 ControlArea : 0xffffb005`f0086010 _CONTROL_AREA 🌽 points back to the CONTROL_AREA in $t0
  744. +0x008 SubsectionBase : 0xffff9c89`4f503000 _MMPTE 🌽🎈💎 ptr to an array of 🍃1 (one) prototype PTEs
  745. +0x010 NextSubsection : 0xffffb005`f00860c8 _SUBSECTION 🌽🖱️ click this link
  746. +0x018 GlobalPerSessionHead : _RTL_AVL_TREE
  747. +0x018 CreationWaitList : (null)
  748. +0x018 SessionDriverProtos : (null)
  749. +0x020 u : <anonymous-tag>
  750. +0x024 StartingSector : 0
  751. +0x028 NumberOfFullSectors : 3 🌽 3 disk sectors for file header
  752. +0x02c PtesInSubsection : 1 🌽🍃 1 PTE (1 page) for the file header
  753. +0x030 u1 : <anonymous-tag>
  754. +0x034 UnusedPtes : 0y000000000000000000000000000000 (0)
  755. +0x034 ExtentQueryNeeded : 0y0
  756. +0x034 DirtyPages : 0y0
  757.  
  758. 🌽🖱️ Clicked link right above. It's actually just an array of _SUBSECTION structs. This one is for the first PE section.
  759. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 ((ntkrnlmp!_SUBSECTION *)0xffffb005f00860c8)
  760. ((ntkrnlmp!_SUBSECTION *)0xffffb005f00860c8) : 0xffffb005f00860c8 [Type: _SUBSECTION*]
  761. [+0x000] ControlArea : 0xffffb005f0086010 [Type: _CONTROL_AREA*]
  762. [+0x008] SubsectionBase : 0xffff9c894f503008 [Type: _MMPTE*] 🌽 ptr to an array of 🍂 0x4C3 prototype PTEs
  763. [+0x010] NextSubsection : 0xffffb005f0086100 [Type: _SUBSECTION*] 🌽 the last subsection will have null there
  764. [+0x018] GlobalPerSessionHead [Type: _RTL_AVL_TREE]
  765. [+0x018] CreationWaitList : 0x0 [Type: _MI_CONTROL_AREA_WAIT_BLOCK *]
  766. [+0x018] SessionDriverProtos : 0x0 [Type: _MI_PER_SESSION_PROTOS *]
  767. [+0x020] u [Type: <anonymous-tag>]
  768. [+0x024] StartingSector : 0x3 [Type: unsigned long]
  769. [+0x028] NumberOfFullSectors : 0x2612 [Type: unsigned long]
  770. [+0x02c] PtesInSubsection : 0x4c3 [Type: unsigned long] 🌽🍂 0x4C3 PTEs for the first PE section
  771. [+0x030] u1 [Type: <anonymous-tag>]
  772. [+0x034 (29: 0)] UnusedPtes : 0x0 [Type: unsigned long]
  773. [+0x034 (30:30)] ExtentQueryNeeded : 0x0 [Type: unsigned long]
  774. [+0x034 (31:31)] DirtyPages : 0x0 [Type: unsigned long]
  775.  
  776. ✨✨✨
  777. Here's the relevant portion of the callstack:
  778.  
  779. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  780. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468 🌽🐦 "fake" fault
  781. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172 🌽 deref backing PFNs/pagefiles for subsection
  782. 1b ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56 🌽 deref backing PFNs/pagefiles
  783. 1c ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92 🌽 deleting CONTROL_AREA 0xffffb005`f0086010
  784.  
  785. Okay, let's reiterate once more. The segment (section object) was being deleted (frame 0x1C).
  786. As part of cleanup MiDeleteSegmentPages was invoked, which proceeded to call MiDeleteSubsectionPages for each
  787. SUBSECTION struct in the segment.
  788. We've just dumped two SUBSECTION structs: the one for the file MZ header, and the next one - for the first PE section.
  789. Job of MiDeleteSubsectionPages appears to dereference/cleanup backing PFNs/pagefiles of prototype PTEs.
  790. And it appears in certain situation the function may need to lock pages which contain the prototype PTEs; it does it
  791. via MiLockProtoPoolPage. In this case that function failed, forcing OS to issue MmAccessFault in order to make the PTE
  792. describing the prototype PTE valid.
  793. Here it is (line copypasted from above): vvv 🌽🎈💎 vvv
  794. 💎🌽🐦 MmAccessFault(FaultStatus=MiFaultWrite`2, VirtualAddress=ffff9c894f503000, PreviousMode=Kernel, TrapInfo=null).
  795. As you can see, the address which ought to be accessed is the address of the prototype PTE for the file header.
  796.  
  797. 🌽 Was MmAccessFault call really needed?
  798. ✨✨✨
  799. 2: kd> !pte ffff9c894f503000 🌽🎈
  800. VA ffff9c894f503000
  801. PXE at FFFFFCFE7F3F99C8 PPE at FFFFFCFE7F339128 PDE at FFFFFCFE672253D0 PTE at FFFFFCCE44A7A818 🌽🐛
  802. contains 0A0000047C068863 contains 0A0000047C069863 contains 0A000003CE6B8863 contains 00000003DBF30880
  803. pfn 47c068 ---DA--KWEV pfn 47c069 ---DA--KWEV pfn 3ce6b8 ---DA--KWEV not valid
  804. Transition: 3dbf30 🌽🦗
  805. Protect: 4 - ReadWrite
  806. 🌽 ^^ Apparently yes - address 🎈 ffff9c894f503000 (which contains Prototype PTE) is not valid, because PTE which
  807. describes it has bit 0 cleared (the PTE 'Valid' bit). Windbg conveniently displayed the PFN - 0x3dbf30 🦗.
  808. But have you noticed there's tons of bugs today in 🪳🦟Windbg🐜?! We gotta verify the PTE/PFN manually, just in case:
  809.  
  810. 2: kd> dt _mmpte . FFFFFCCE44A7A818 🌽🐛
  811. nt!_MMPTE
  812. +0x000 u :
  813. +0x000 Long : 0x00000003`dbf30880
  814. +0x000 VolatileLong : 0x00000003`dbf30880
  815. +0x000 Hard : _MMPTE_HARDWARE
  816. +0x000 Proto : _MMPTE_PROTOTYPE 🌽🖱️ click this link
  817. +0x000 Soft : _MMPTE_SOFTWARE
  818. +0x000 TimeStamp : _MMPTE_TIMESTAMP
  819. +0x000 Trans : _MMPTE_TRANSITION
  820. +0x000 Subsect : _MMPTE_SUBSECTION
  821. +0x000 List : _MMPTE_LIST
  822.  
  823. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMPTE_TRANSITION *)0xfffffcce44a7a818))
  824. (*((ntkrnlmp!_MMPTE_TRANSITION *)0xfffffcce44a7a818)) [Type: _MMPTE_TRANSITION]
  825. [+0x000 ( 0: 0)] Valid : 0x0 [Type: unsigned __int64]
  826. [+0x000 ( 1: 1)] Write : 0x0 [Type: unsigned __int64]
  827. [+0x000 ( 2: 2)] Spare : 0x0 [Type: unsigned __int64]
  828. [+0x000 ( 3: 3)] IoTracker : 0x0 [Type: unsigned __int64]
  829. [+0x000 ( 4: 4)] SwizzleBit : 0x0 [Type: unsigned __int64]
  830. [+0x000 ( 9: 5)] Protection : 0x4 [Type: unsigned __int64]
  831. [+0x000 (10:10)] Prototype : 0x0 [Type: unsigned __int64]
  832. [+0x000 (11:11)] Transition : 0x1 [Type: unsigned __int64] 🌽 yes, marked as transition
  833. [+0x000 (47:12)] PageFrameNumber : 0x3dbf30 [Type: unsigned __int64] 🌽🦗 yes, PFN was valid
  834. [+0x000 (63:48)] Unused : 0x0 [Type: unsigned __int64]
  835.  
  836. 2: kd> !pfn 0x3dbf30 🌽🦗
  837. PFN 003DBF30 at address FFFFEA000B93D900 🌽🦗 got address of _mmpfn in PFN db
  838. flink 5FF1175C0 blink / share count 00000000 pteaddress FFFFFCCE44A7A818
  839. reference count 0001 used entry count 0000 Cached color 0 Priority 5
  840. restore pte 78D80700002084 containing page 3CE6B8 Standby R
  841. ReadInProgress
  842.  
  843. 2: kd> dt _mmpfn FFFFEA000B93D900 🌽🦗
  844. nt!_MMPFN
  845. +0x000 ListEntry : _LIST_ENTRY [ 0xffffb005`ff1175c0 - 0xfffffcce`44a7a818 ]
  846. +0x000 TreeNode : _RTL_BALANCED_NODE
  847. +0x000 u1 : <anonymous-tag>
  848. +0x008 PteAddress : 0xfffffcce`44a7a818 _MMPTE
  849. +0x008 PteLong : 0xfffffcce`44a7a818
  850. +0x010 OriginalPte : _MMPTE 🌽🖱️ click this link
  851. +0x018 u2 : _MIPFNBLINK
  852. +0x020 u3 : <anonymous-tag>
  853. +0x024 NodeBlinkLow : 0xffff
  854. +0x026 Unused : 0y0000
  855. +0x026 Unused2 : 0y0000
  856. +0x027 ViewCount : 0x31 '1'
  857. +0x027 NodeFlinkLow : 0x31 '1'
  858. +0x027 ModifiedListBucketIndex : 0y0001
  859. +0x027 AnchorLargePageSize : 0y01
  860. +0x028 u4 : <anonymous-tag>
  861.  
  862. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMPTE *)0xffffea000b93d910))
  863. (*((ntkrnlmp!_MMPTE *)0xffffea000b93d910)) [Type: _MMPTE]
  864. [+0x000] u [Type: <anonymous-tag>] 🌽🖱️ and click this
  865. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMPTE *)0xffffea000b93d910)).u
  866. (*((ntkrnlmp!_MMPTE *)0xffffea000b93d910)).u [Type: <anonymous-tag>]
  867. [+0x000] Long : 0x78d80700002084 [Type: unsigned __int64]
  868. [+0x000] VolatileLong : 0x78d80700002084 [Type: unsigned __int64]
  869. [+0x000] Hard [Type: _MMPTE_HARDWARE]
  870. [+0x000] Proto [Type: _MMPTE_PROTOTYPE]
  871. [+0x000] Soft [Type: _MMPTE_SOFTWARE] 🌽🖱️ this one we really want
  872. [+0x000] TimeStamp [Type: _MMPTE_TIMESTAMP]
  873. [+0x000] Trans [Type: _MMPTE_TRANSITION]
  874. [+0x000] Subsect [Type: _MMPTE_SUBSECTION]
  875. [+0x000] List [Type: _MMPTE_LIST]
  876.  
  877. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMPTE_SOFTWARE *)0xffffea000b93d910))
  878. (*((ntkrnlmp!_MMPTE_SOFTWARE *)0xffffea000b93d910)) [Type: _MMPTE_SOFTWARE]
  879. [+0x000 ( 0: 0)] Valid : 0x0 [Type: unsigned __int64]
  880. [+0x000 ( 1: 1)] PageFileReserved : 0x0 [Type: unsigned __int64]
  881. [+0x000 ( 2: 2)] PageFileAllocated : 0x1 [Type: unsigned __int64] 🌽
  882. [+0x000 ( 3: 3)] ColdPage : 0x0 [Type: unsigned __int64]
  883. [+0x000 ( 4: 4)] SwizzleBit : 0x0 [Type: unsigned __int64]
  884. [+0x000 ( 9: 5)] Protection : 0x4 [Type: unsigned __int64]
  885. [+0x000 (10:10)] Prototype : 0x0 [Type: unsigned __int64]
  886. [+0x000 (11:11)] Transition : 0x0 [Type: unsigned __int64]
  887. [+0x000 (15:12)] PageFileLow : 0x2 [Type: unsigned __int64] 🌽💎 aha, got pagefile number!
  888. [+0x000 (25:16)] UsedPageTableEntries : 0x0 [Type: unsigned __int64]
  889. [+0x000 (26:26)] ShadowStack : 0x0 [Type: unsigned __int64]
  890. [+0x000 (31:27)] Unused : 0x0 [Type: unsigned __int64]
  891. [+0x000 (63:32)] PageFileHigh : 0x78d807 [Type: unsigned __int64]
  892.  
  893.  
  894. ✨✨✨
  895. 🌽 Here's the relevant callstack, as we may need to check a few more things.
  896. 16 ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  897. 17 ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  898. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  899. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468
  900.  
  901. ✨✨✨
  902. 2: kd> .cxr
  903. Resetting default scope
  904.  
  905. 2: kd> .frame /c /r 18
  906. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  907. rax=fffff801144d1080 rbx=0000000000000002 rcx=ffffd3011992f000
  908. rdx=ffffd3011992f000 rsi=ffffe509f819a048 rdi=ffffb005ff1175a0
  909. rip=fffff8011445bb4d rsp=ffffe509f8199e90 rbp=ffffe509f8199ed0
  910. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0
  911. r11=0000027e296aba05 r12=ffffb005df2c8480 r13=ffffe509f819a010
  912. r14=0000000000000000 r15=0000000000000000
  913. iopl=0 nv up ei pl zr na po nc
  914. cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00040246
  915. nt!MiIssueHardFault+0x29d:
  916. fffff801`1445bb4d eb09 jmp nt!MiIssueHardFault+0x2a8 (fffff801`1445bb58)
  917.  
  918. 2: kd> ub .
  919. nt!MiIssueHardFault+0x282:
  920. fffff801`1445bb32 33d2 xor edx,edx
  921. fffff801`1445bb34 e8c73effff call nt!KiStackAttachProcess (fffff801`1444fa00)
  922. fffff801`1445bb39 4d8b4510 mov r8,qword ptr [r13+10h]
  923. fffff801`1445bb3d 83cb02 or ebx,2
  924. fffff801`1445bb40 8bd3 mov edx,ebx
  925. fffff801`1445bb42 488bcf mov rcx,rdi
  926. fffff801`1445bb45 83e201 and edx,1
  927. fffff801`1445bb48 e847240000 call nt!MiIssueHardFaultIo (fffff801`1445df94)
  928.  
  929. ✨✨✨
  930. 2: kd> dt @rdi nt!_mminpage_support 🌽 dump arg1 (rcx/rdi) for MiIssueHardFaultIo
  931. +0x000 ListEntry : _LIST_ENTRY [ 0x00000000`00000000 - 0x00000000`00000000 ]
  932. +0x000 SListEntry : _SLIST_ENTRY
  933. +0x010 ListHead : _LIST_ENTRY [ 0xffffb005`ff1175b0 - 0xffffb005`ff1175b0 ]
  934. +0x020 Event : _KEVENT
  935. +0x038 CollidedEvent : _KEVENT
  936. +0x050 IoStatus : _IO_STATUS_BLOCK
  937. +0x060 ReadOffset : _LARGE_INTEGER 0x20000000`0078d807
  938. +0x068 u2 : <anonymous-tag>
  939. +0x098 Thread : 0xffffb005`df2c8480 _ETHREAD
  940. +0x0a0 LockedProtoPfn : (null)
  941. +0x0a8 PteContents : _MMPTE
  942. +0x0b0 WaitCount : 0n1
  943. +0x0b4 InjectRetry : 0n0
  944. +0x0b8 ByteCount : 0x1000
  945. +0x0bc u3 : <anonymous-tag>
  946. +0x0c0 u1 : <anonymous-tag> 🌽🖱️ click
  947. +0x0c8 FilePointer : (null)
  948. +0x0c8 PagingFile : (null)
  949. +0x0d0 ControlArea : (null)
  950. +0x0d0 Subsection : (null)
  951. +0x0d8 Autoboost : 0xffffb005`df2c8ad0 Void
  952. +0x0e0 FaultingAddress : 0xffff9c89`4f503000 Void
  953. +0x0e8 PointerPte : 0xfffffcce`44a7a818 _MMPTE
  954. +0x0f0 BasePte : 0xfffffcce`44a7a818 _MMPTE
  955. +0x0f8 Pfn : 0xffffea00`0b93d900 _MMPFN
  956. +0x100 PrefetchMdl : (null)
  957. +0x108 ProbeCount : 0n0
  958. +0x110 Mdl : _MDL
  959. +0x140 Page : [16] 0x3dbf30 🌽🦗 our PFN we've seen before
  960. +0x140 FlowThrough : _MMINPAGE_SUPPORT_FLOW_THROUGH
  961.  
  962. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMINPAGE_SUPPORT *)0xffffb005ff1175a0)).u1
  963. (*((ntkrnlmp!_MMINPAGE_SUPPORT *)0xffffb005ff1175a0)).u1 [Type: <anonymous-tag>]
  964. [+0x000] e1 [Type: _MMINPAGE_FLAGS] 🌽🖱️ click
  965. [+0x000] LongFlags : 0x284b00 [Type: unsigned long]
  966. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 (*((ntkrnlmp!_MMINPAGE_FLAGS *)0xffffb005ff117660))
  967. (*((ntkrnlmp!_MMINPAGE_FLAGS *)0xffffb005ff117660)) [Type: _MMINPAGE_FLAGS]
  968. [+0x000 ( 0: 0)] GetExtents : 0x0 [Type: unsigned long]
  969. [+0x000 ( 2: 1)] PrefetchSystemVmType : 0x0 [Type: unsigned long]
  970. [+0x000 ( 3: 3)] VaPrefetchReadBlock : 0x0 [Type: unsigned long]
  971. [+0x000 ( 4: 4)] CollidedFlowThrough : 0x0 [Type: unsigned long]
  972. [+0x000 ( 5: 5)] ForceCollisions : 0x0 [Type: unsigned long]
  973. [+0x000 ( 6: 6)] InPageExpanded : 0x0 [Type: unsigned long]
  974. [+0x000 ( 7: 7)] IssuedAtLowPriority : 0x0 [Type: unsigned long]
  975. [+0x000 ( 8: 8)] FaultFromStore : 0x1 [Type: unsigned long] 🌽 yes, from store
  976. [+0x000 (11: 9)] PagePriority : 0x5 [Type: unsigned long]
  977. [+0x000 (14:12)] ClusteredPagePriority : 0x4 [Type: unsigned long]
  978. [+0x000 (15:15)] MakeClusterValid : 0x0 [Type: unsigned long]
  979. [+0x000 (16:16)] PerformRelocations : 0x0 [Type: unsigned long]
  980. [+0x000 (17:17)] ZeroLastPage : 0x0 [Type: unsigned long]
  981. [+0x000 (18:18)] UserFault : 0x0 [Type: unsigned long]
  982. [+0x000 (19:19)] StandbyProtectionNeeded : 0x1 [Type: unsigned long]
  983. [+0x000 (20:20)] PteChanged : 0x0 [Type: unsigned long]
  984. [+0x000 (21:21)] PageFileFault : 0x1 [Type: unsigned long] 🌽 kinda
  985. [+0x000 (22:22)] PageFilePageHashActive : 0x0 [Type: unsigned long]
  986. [+0x000 (23:23)] CoalescedIo : 0x0 [Type: unsigned long]
  987. [+0x000 (24:24)] VmLockNotNeeded : 0x0 [Type: unsigned long]
  988. [+0x000 (25:25)] Spare0 : 0x0 [Type: unsigned long]
  989. [+0x000 (31:26)] Spare1 : 0x0 [Type: unsigned long]
  990.  
  991.  
  992. ✨✨✨✨✨✨✨✨
  993. Okay, checkpoint. Let's see what we have. To delete a segment, system needed to lock the page containing prototype PTE
  994. for the file header, but that page was not valid. So OS called MmAccessFault to make it valid. The "invalid" PTE is
  995. marked as transition, and contains PFN. Via PFN database we found out contents of the original PTE (MMPFN.OriginalPte).
  996. It contained "offset" and pagefile number == 2 💎.
  997. If you read "2019.Extracting Compressed Pages from the Windows 10 Virtual Store.Sardar.[MemCompression].pdf", you'll
  998. know that pagefile number 2 refers to fake pagefile - the store manager; i.e. the page was compressed. OS needs to
  999. decompress the page, so it invokes SmPageRead (frame 0x16).
  1000. That means funcs MiIssueHardFault/MiIssueHardFaultIo do not imply disk read request at all! Such claims are incorrect.
  1001.  
  1002. 08 ffffe509`f8199650 fffff801`144d10d0 nt!KiPageFault+0x443 🌽📝 called the "real" MmAccessFault
  1003. 09 ffffe509`f81997e8 fffff801`144a6bf0 nt!RtlDecompressBufferXpressLz+0x50 🌽💥
  1004. 0a ffffe509`f8199800 fffff801`144a6938 nt!RtlDecompressBufferEx+0x60
  1005. 0b ffffe509`f8199850 fffff801`144a67c5 nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  1006. ...
  1007. 15 ffffe509`f8199d80 fffff801`144fa96b nt!SMKM_STORE_MGR<SM_TRAITS>::SmPageRead+0x167
  1008. 16 ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  1009. 17 ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  1010. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  1011. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468 🌽🐦 "fake" fault (described below)
  1012. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  1013. 1b ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56
  1014. 1c ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92
  1015. 1d ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1 🌽🗑️
  1016.  
  1017.  
  1018. 🌽 We've confirmed that everything was in order, and we're back at pagefault at frames 09/08. To decompress the
  1019. packed page, RtlDecompressBufferXpressLz was invoked. And here's the twist: the compressed data itself was paged out!
  1020. So the real pagefault happened (frame 8). Is it normal to page out the compressed data? Actually yes, it can happen.
  1021.  
  1022. 2: kd> .cxr 0xffffe509f8198de0 🌽🎰 restore context at the moment of #PF
  1023. rax=fffff801144d1080 rbx=ffffd3011992f000 rcx=ffffd3011992f000 🌽 (it's .cxr, so context is fully valid)
  1024. rdx=ffffd3011992f000 rsi=0000000000000002 rdi=0000027e296ab9f0
  1025. rip=fffff801144d10d0 rsp=ffffe509f81997e8 rbp=0000027e296ab9af
  1026. r8=0000027e296ab9f0 r9=0000000000000015 r10=ffffd3011992fea0
  1027. r11=0000027e296aba05 r12=ffffe509f8199a58 r13=ffffb005ee4ef000
  1028. r14=ffffd30119930000 r15=0000000000000000
  1029. iopl=0 nv up ei pl zr na po nc
  1030. cs=0010 ss=0000 ds=002b es=002b fs=0053 gs=002b efl=00050246
  1031. nt!RtlDecompressBufferXpressLz+0x50:
  1032. fffff801`144d10d0 418b08 mov ecx,dword ptr [r8] ds:002b:0000027e`296ab9f0=???????? 🌽🥁
  1033.  
  1034. 🌽 You can see the faulted address is in usermode range. That's normal, that's just MemCompression process stores
  1035. the packed data. It also explains the reason why the System-owned thread had to attach to MemCompression process.
  1036. We can use `!pte` command to help us find out the pagefile which contains the compressed data.
  1037.  
  1038. 2: kd> !pte 27e`296ab9f0 🌽🥁
  1039. VA 0000027e296ab9f0
  1040. PXE at FFFFFCFE7F3F9020 PPE at FFFFFCFE7F204FC0 PDE at FFFFFCFE409F8A58 PTE at FFFFFC813F14B558
  1041. Page 131a53 not present in the dump file. Type ".hh dbgerr004" for details
  1042. Unable to get PXE FFFFFCFE7F204FC0
  1043.  
  1044. 🌽 Oops! No, we can't use `!pte` here. The address is in usermode range, and the dump is kernel-only dump;
  1045. apparently it means that PTEs for usermode ranges are not included in the dump either.
  1046. But we need the relevant dump file location. Because we know that system tried to read the compressed page from the
  1047. pagefile, and that read failed with 🍄 C000000E (STATUS_NO_SUCH_DEVICE). Yet there can be multiple pagefiles, on
  1048. different disks: we *really* want to know which disk read failed.
  1049.  
  1050. 🌽 But whatever; OS allows less than 0x10 pagefiles, let's simply list all of them.
  1051. ✨✨✨✨✨✨✨✨
  1052. 2: kd> !vm
  1053. Page File: \??\C:\pagefile.sys 🌽 the only real pagefile, on C:
  1054. Current: 12582912 Kb Free Space: 11681548 Kb
  1055. Minimum: 12582912 Kb Maximum: 31023400 Kb
  1056. Page File: \??\C:\swapfile.sys 🌽 for UWP, not our case
  1057. Current: 16384 Kb Free Space: 16376 Kb
  1058. Minimum: 16384 Kb Maximum: 24818720 Kb
  1059. No Name for Paging File 🌽 fake pagefile; it's the Store Manager (compressed data)
  1060. Current: 47601456 Kb Free Space: 41454724 Kb
  1061. Minimum: 47601456 Kb Maximum: 47601456 Kb
  1062.  
  1063. Physical Memory: 4144514 ( 16578056 Kb)
  1064. Available Pages: 719871 ( 2879484 Kb)
  1065. ResAvail Pages: 3437549 ( 13750196 Kb)
  1066. Locked IO Pages: 0 ( 0 Kb)
  1067. Free System PTEs: 4294975607 (17179902428 Kb)
  1068.  
  1069. ******* 69376 kernel stack PTE allocations have failed ******
  1070. ******* 350546176 kernel stack growth attempts have failed ******
  1071. 🌽 ^^ `!vm` command shows obviously bogus alloc failure info; no way system sustained that many allocation failures.
  1072. Reason for this bogus info is Microsoft's fail to provide correct _MI_SYSTEM_INFORMATION struct symbols (at nt!MiState).
  1073. After many years MS fixed symbols btw, but only for some win11 builds; MS generally neglects OS kernel debugging.
  1074.  
  1075. Modified Pages: 8438 ( 33752 Kb)
  1076. Modified PF Pages: 8436 ( 33744 Kb)
  1077. Modified No Write Pages: 24 ( 96 Kb)
  1078. NonPagedPool Usage: 241 ( 964 Kb)
  1079. NonPagedPoolNx Usage: 233372 ( 933488 Kb)
  1080. NonPagedPool Max: 4294967296 (17179869184 Kb)
  1081. PagedPool Usage: 130023 ( 520092 Kb)
  1082. PagedPool Maximum: 4294967296 (17179869184 Kb)
  1083. Processor Commit: 1304 ( 5216 Kb)
  1084. Unable to read nt!_LIST_ENTRY.Flink at 0000000000000000
  1085. Session Commit: 0 ( 0 Kb)
  1086. Shared Commit: 766838 ( 3067352 Kb)
  1087. Special Pool: 0 ( 0 Kb)
  1088. Kernel Stacks: 41507 ( 166028 Kb)
  1089. Pages For MDLs: 52320 ( 209280 Kb)
  1090. ContigMem Pages: 0 ( 0 Kb)
  1091. Partition Pages: 0 ( 0 Kb)
  1092. Pages For AWE: 0 ( 0 Kb)
  1093. NonPagedPool Commit: 238303 ( 953212 Kb)
  1094. PagedPool Commit: 130023 ( 520092 Kb)
  1095. Driver Commit: 44256 ( 177024 Kb)
  1096. Boot Commit: 4794 ( 19176 Kb)
  1097. PFN Array Commit: 49119 ( 196476 Kb)
  1098. SmallNonPagedPtesCommit: 875 ( 3500 Kb)
  1099. SlabAllocatorPages: 4608 ( 18432 Kb)
  1100. SkPagesInUnchargedSlabs: 0 ( 0 Kb)
  1101. System PageTables: 5695 ( 22780 Kb)
  1102. ProcessLockedFilePages: 123 ( 492 Kb)
  1103. Pagefile Hash Pages: 462 ( 1848 Kb)
  1104. Sum System Commit: 1340227 ( 5360908 Kb)
  1105. Total Private: 4620955 ( 18483820 Kb)
  1106. Misc/Transient Commit: 59369 ( 237476 Kb)
  1107. Committed pages: 6020551 ( 24082204 Kb)
  1108. Commit limit: 7290242 ( 29160968 Kb)
  1109.  
  1110. Pid ImageName Commit SharedCommit Debt
  1111. bb0 chrome.exe 1365620 Kb 864360 Kb 0 Kb
  1112. ... <tons of processes here, removed>
  1113. ...
  1114.  
  1115.  
  1116. ✨✨✨✨✨✨✨✨
  1117. 🌽 So we learned that there's only one real pagefile - C:\pagefile.sys. Thus the device stack which failed the read
  1118. request with 🍄 C000000E (STATUS_NO_SUCH_DEVICE) is related to the C: volume.
  1119.  
  1120. ✨✨✨
  1121. 2: kd> !driveinfo C:
  1122. Drive C:, DriveObject ffff9c894949acd0 🌽 not really a drive, it is just the "C:" symlink object;
  1123. Directory Object: ffff9c8947629280 Name: C: 🌽 you get same output via `!object \\GLOBAL??\\C:`
  1124. Flags: 00000000 ( Local )
  1125. Target String is '\Device\HarddiskVolume4' 🌽🍥
  1126. Drive Letter Index is 3 (C:)
  1127. Volume DevObj: ffffb005cc3f1c00 🌽⚽ device object for \Driver\volmgr =VPB.RealDevice
  1128. Vpb: ffffb005cc3d5cd0 DeviceObject: ffffb005df2d5030 🏈🏀 device object for \FileSystem\Ntfs =VPB.DeviceObject
  1129. FileSystem: \FileSystem\Ntfs
  1130. Cannot get ntfs!VOLUME_DEVICE_OBJECT.Vcb @ ffffb005df2d5030
  1131.  
  1132. 🌽 ^^ Wow, what a horrible performance! It took 6 bloody minutes to complete this simple command, I'm kidding you not.
  1133. When the whole dump analysis takes may be 15-30 minutes, that's prohibitively expensive.
  1134. And the info this command gave us, while useful, is quite incomplete: it doesn't tell us anything about the disk device.
  1135.  
  1136. ✨✨✨
  1137. 🌽 A while ago I've made a simple windbg script to list ntfs volumes and free space (because Microsoft's `!diskspace`
  1138. command was broken [naturally 🙄]). Let's adjust it a little bit and run it here:
  1139.  
  1140. 2: kd> r$t0=0; r$t1=0;.foreach(v {#"lock cmpxchg" ntfs!NtfsAllocateBitmapRun L1000}) {.if (@$t1<2) {r$t1=@$t1+1} .elsif (@$t1<9) { r$t1=( v >>18&FF)+( v >>8&FF00) }};
  1141. 2: kd> .if(@$t0){r?$t0=(nt!_FILE_OBJECT*)@$t0;r?$t2=@$t0->Vpb->DeviceObject->DeviceExtension;r$t3=@$t2+8;r$t2=poi(@$t3)}.else{r$t2=ntfs!NtfsData+0x18;r$t3=poi(@$t2)};.while(@$t3!=@$t2){r$t4=@$t3-8;.if(wo(@$t4)!=701){.printf "dt ntfs!_VCB %p : bad VCB\n",@$t4}.else{r?$t5=((nt!_DEVICE_OBJECT*)(@$t4)-1);.while(@@(@$t5->DeviceExtension != @$t4)){r?$t5=(nt!_DEVICE_OBJECT*)((int64)@$t5-0x10)};.printf /D "dt <link cmd=\"dt nt!_DEVICE_OBJECT %p\">nt!_DEVICE_OBJECT</link> %p\ndt ntfs!_VCB %p\n TotalClusters : %p\n FreeClusters : %p\n LowestFreeClusters : %p\n HighestFreeClusters : %p\n ClusterSize : %p\n",@$t5,@$t5,@$t4,qwo(@$t4+128),qwo(@$t4+130),qwo(@$t4+@$t1),qwo(@$t4+@$t1+8),dwo(@$t4+160);r?$t5=@$t5->DeviceObjectExtension->Vpb;??@$t5;!devobj @@(@$t5->RealDevice); .printf "\n"};r$t3=poi(@$t3);}
  1142.  
  1143. ✨✨✨
  1144. 🌽 ^^ Isnt't this 2-line script just beautiful? Like a tiny pearl! Or a writhing ball of worms. Ew. Lol. Anyway, script
  1145. output is below, and since it works instantly, the next time we can save a few minutes.
  1146.  
  1147. dt nt!_DEVICE_OBJECT ffffb005df2d5030
  1148. dt ntfs!_VCB ffffb005df2d5180
  1149. TotalClusters : 0000000003b2c251
  1150. FreeClusters : 000000000284f806
  1151. LowestFreeClusters : 000000000284f7f6
  1152. HighestFreeClusters : 0000000002aaf799
  1153. ClusterSize : 0000000000001000
  1154. struct _VPB * 0xffffb005`cc3d5cd0 🌽 that's Volume Parameters Block
  1155. +0x000 Type : 0n10
  1156. +0x002 Size : 0n96
  1157. +0x004 Flags : 1
  1158. +0x006 VolumeLabelLength : 4
  1159. +0x008 DeviceObject : 0xffffb005`df2d5030 _DEVICE_OBJECT 🏈🏀
  1160. +0x010 RealDevice : 0xffffb005`cc3f1c00 _DEVICE_OBJECT 🌽⚽
  1161. +0x018 SerialNumber : 0x36008800
  1162. +0x01c ReferenceCount : 0x5530
  1163. +0x020 VolumeLabel : [32] "OS" 🌽 apparently volume had label
  1164. Device object (ffffb005cc3f1c00) is for:
  1165. HarddiskVolume4 \Driver\volmgr DriverObject ffffb005c5106e20 🌽🍥
  1166. Current Irp 00000000 RefCount 21808 Type 00000007 Flags 00001150
  1167. Vpb 0xffffb005cc3d5cd0 SecurityDescriptor ffff9c8949423920 DevExt ffffb005cc3f1d50 DevObjExt ffffb005cc3f1f18 Dope ffffb005df287490 DevNode ffffb005df28eba0
  1168. ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
  1169. Characteristics (0x00020000) FILE_DEVICE_ALLOW_APPCONTAINER_TRAVERSAL
  1170. AttachedDevice (Upper) ffffb005df2be030 \Driver\fvevol
  1171. Device queue is not busy.
  1172.  
  1173. dt nt!_DEVICE_OBJECT ffffb005e4cd6030
  1174. dt ntfs!_VCB ffffb005e4cd6180
  1175. TotalClusters : 000000000e8e0bff
  1176. FreeClusters : 000000000e8d521c
  1177. LowestFreeClusters : 000000000e8d521c
  1178. HighestFreeClusters : 000000000e8d521c
  1179. ClusterSize : 0000000000001000
  1180. struct _VPB * 0xffffb005`e4b7bd20
  1181. +0x000 Type : 0n10
  1182. +0x002 Size : 0n96
  1183. +0x004 Flags : 1
  1184. +0x006 VolumeLabelLength : 8
  1185. +0x008 DeviceObject : 0xffffb005`e4cd6030 _DEVICE_OBJECT
  1186. +0x010 RealDevice : 0xffffb005`df25cc40 _DEVICE_OBJECT
  1187. +0x018 SerialNumber : 0x7463ceff
  1188. +0x01c ReferenceCount : 0x24
  1189. +0x020 VolumeLabel : [32] "DATA"
  1190. Device object (ffffb005df25cc40) is for:
  1191. HarddiskVolume1 \Driver\volmgr DriverObject ffffb005c5106e20
  1192. Current Irp 00000000 RefCount 36 Type 00000007 Flags 00003050
  1193. Vpb 0xffffb005e4b7bd20 SecurityDescriptor ffff9c8949423920 DevExt ffffb005df25cd90 DevObjExt ffffb005df25cf58 Dope ffffb005df286d20 DevNode ffffb005df26abb0
  1194. ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
  1195. Characteristics (0x00020000) FILE_DEVICE_ALLOW_APPCONTAINER_TRAVERSAL
  1196. AttachedDevice (Upper) ffffb005df2ab030 \Driver\fvevol
  1197. Device queue is not busy.
  1198.  
  1199. <..>
  1200.  
  1201. dt nt!_DEVICE_OBJECT ffffb005f7cbb030
  1202. dt ntfs!_VCB ffffb005f7cbb180
  1203. TotalClusters : 0000000003b2c251
  1204. FreeClusters : 0000000002ab3146
  1205. LowestFreeClusters : 0000000002ab3146
  1206. HighestFreeClusters : 0000000002ab3146
  1207. ClusterSize : 0000000000001000
  1208. struct _VPB * 0xffffb005`edd65a30
  1209. +0x000 Type : 0n10
  1210. +0x002 Size : 0n96
  1211. +0x004 Flags : 1
  1212. +0x006 VolumeLabelLength : 4
  1213. +0x008 DeviceObject : 0xffffb005`f7cbb030 _DEVICE_OBJECT
  1214. +0x010 RealDevice : 0xffffb005`f7b7f2a0 _DEVICE_OBJECT
  1215. +0x018 SerialNumber : 0x36008800
  1216. +0x01c ReferenceCount : 0x17b
  1217. +0x020 VolumeLabel : [32] "OS"
  1218. Device object (ffffb005f7b7f2a0) is for:
  1219. HarddiskVolumeShadowCopy4 \Driver\volsnap DriverObject ffffb005cc364d30 🌽 shadow copy volume, ignore it
  1220. Current Irp 00000000 RefCount 379 Type 00000024 Flags 00001050
  1221. Vpb 0xffffb005edd65a30 SecurityDescriptor ffff9c8949423920 DevExt ffffb005f7b7f3f0 DevObjExt ffffb005f7b7f988 Dope ffffb005edd65b80 DevNode ffffb005efe04cc0
  1222. ExtensionFlags (0000000000)
  1223. Characteristics (0000000000)
  1224. Device queue is not busy.
  1225.  
  1226.  
  1227. 🌽 Let's continue with information we've just retrieved.
  1228. ✨✨✨
  1229. 2: kd> dt _device_object ffffb005cc3f1c00 🌽⚽ checkout volume "RealDevice"; drv: \Driver\volmgr
  1230. nt!_DEVICE_OBJECT 🌽 it's pointed to by FILE_OBJECT.DeviceObject
  1231. +0x000 Type : 0n3
  1232. +0x002 Size : 0x318
  1233. +0x004 ReferenceCount : 0n21808
  1234. +0x008 DriverObject : 0xffffb005`c5106e20 _DRIVER_OBJECT
  1235. +0x010 NextDevice : 0xffffb005`df264c40 _DEVICE_OBJECT
  1236. +0x018 AttachedDevice : 0xffffb005`df2be030 _DEVICE_OBJECT
  1237. +0x020 CurrentIrp : (null)
  1238. +0x028 Timer : (null)
  1239. +0x030 Flags : 0x1150
  1240. +0x034 Characteristics : 0x20000
  1241. +0x038 Vpb : 0xffffb005`cc3d5cd0 _VPB 🌽🏈 VPB pointer we've seen above
  1242. +0x040 DeviceExtension : 0xffffb005`cc3f1d50 Void 🌽 not the device extension we want (see below)
  1243. +0x048 DeviceType : 7
  1244. +0x04c StackSize : 6 ''
  1245. +0x050 Queue : <anonymous-tag>
  1246. +0x098 AlignmentRequirement : 3
  1247. +0x0a0 DeviceQueue : _KDEVICE_QUEUE
  1248. +0x0c8 Dpc : _KDPC
  1249. +0x108 ActiveThreadCount : 0
  1250. +0x110 SecurityDescriptor : 0xffff9c89`49423920 Void
  1251. +0x118 DeviceLock : _KEVENT
  1252. +0x130 SectorSize : 0x200
  1253. +0x132 Spare1 : 1
  1254. +0x138 DeviceObjectExtension : 0xffffb005`cc3f1f18 _DEVOBJ_EXTENSION
  1255. +0x140 Reserved : (null)
  1256.  
  1257. 2: kd> !devobj ffffb005cc3f1c00 🌽⚽ same "RealDevice", just different command
  1258. Device object (ffffb005cc3f1c00) is for:
  1259. HarddiskVolume4 \Driver\volmgr DriverObject ffffb005c5106e20
  1260. Current Irp 00000000 RefCount 21808 Type 00000007 Flags 00001150
  1261. Vpb 0xffffb005cc3d5cd0 SecurityDescriptor ffff9c8949423920 DevExt ffffb005cc3f1d50 DevObjExt ffffb005cc3f1f18 Dope ffffb005df287490 DevNode ffffb005df28eba0
  1262. ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
  1263. Characteristics (0x00020000) FILE_DEVICE_ALLOW_APPCONTAINER_TRAVERSAL
  1264. AttachedDevice (Upper) ffffb005df2be030 \Driver\fvevol
  1265. Device queue is not busy.
  1266.  
  1267. 2: kd> !devstack ffffb005cc3f1c00 🌽⚽ check where's the device in the device stack
  1268. !DevObj !DrvObj !DevExt ObjectName
  1269. ffffb005df2b9080 \Driver\volsnap ffffb005df2b91d0
  1270. ffffb005cc3f2e10 \Driver\volume ffffb005cc3f2f60
  1271. ffffb005df2bf7d0 \Driver\iorate ffffb005df2bf920
  1272. ffffb005df2be030 \Driver\fvevol ffffb005df2be180
  1273. > ffffb005cc3f1c00 \Driver\volmgr ffffb005cc3f1d50 HarddiskVolume4 🌽⚽ ...here it is
  1274. !DevNode ffffb005df28eba0 :
  1275. DeviceInst is "STORAGE\Volume\{b3d462db-0a60-11eb-9bca-806e6f6e6963}#0000000011500000"
  1276. ServiceName is "volume"
  1277.  
  1278. 🌽 ^^ Note devices/drivers in device stack. Everything is standard.
  1279. If you have doubts about driver, download it to your symstore and check if vendor is MS, e.g. via `!chkimg iorate`.
  1280.  
  1281. ✨✨✨
  1282. 2: kd> dt _device_object ffffb005df2d5030 🌽🏀 checkout "DeviceObject"; drv: \FileSystem\Ntfs
  1283. nt!_DEVICE_OBJECT
  1284. +0x000 Type : 0n3
  1285. +0x002 Size : 0x2b20
  1286. +0x004 ReferenceCount : 0n0
  1287. +0x008 DriverObject : 0xffffb005`c82eeb50 _DRIVER_OBJECT
  1288. +0x010 NextDevice : 0xffffb005`c8232a70 _DEVICE_OBJECT
  1289. +0x018 AttachedDevice : 0xffffb005`df267c20 _DEVICE_OBJECT
  1290. +0x020 CurrentIrp : (null)
  1291. +0x028 Timer : (null)
  1292. +0x030 Flags : 0x8060000
  1293. +0x034 Characteristics : 0
  1294. +0x038 Vpb : (null)
  1295. +0x040 DeviceExtension : 0xffffb005`df2d5180 Void 🌽 pointer to ntfs!_VCB (Volume Control Block)
  1296. +0x048 DeviceType : 8
  1297. +0x04c StackSize : 11 ''
  1298. +0x050 Queue : <anonymous-tag>
  1299. +0x098 AlignmentRequirement : 3
  1300. +0x0a0 DeviceQueue : _KDEVICE_QUEUE
  1301. +0x0c8 Dpc : _KDPC
  1302. +0x108 ActiveThreadCount : 0
  1303. +0x110 SecurityDescriptor : 0xffff9c89`49423920 Void
  1304. +0x118 DeviceLock : _KEVENT
  1305. +0x130 SectorSize : 0x200
  1306. +0x132 Spare1 : 1
  1307. +0x138 DeviceObjectExtension : 0xffffb005`df2d7b50 _DEVOBJ_EXTENSION
  1308. +0x140 Reserved : (null)
  1309.  
  1310. 2: kd> !devobj 🌽🏀 same "DeviceObject"
  1311. Device object not found
  1312. 2: kd> !devobj ffffb005df2d5030
  1313. Device object (ffffb005df2d5030) is for:
  1314. \FileSystem\Ntfs DriverObject ffffb005c82eeb50
  1315. Current Irp 00000000 RefCount 0 Type 00000008 Flags 08060000
  1316. SecurityDescriptor ffff9c8949423920 DevExt ffffb005df2d5180 DevObjExt ffffb005df2d7b50
  1317. ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
  1318. Characteristics (0000000000)
  1319. AttachedDevice (Upper) ffffb005df267c20 \FileSystem\FltMgr
  1320. Device queue is not busy.
  1321.  
  1322. 2: kd> !devstack ffffb005df2d5030 🌽🏀 check where's the device in the device stack
  1323. !DevObj !DrvObj !DevExt ObjectName
  1324. ffffb005df267c20 \FileSystem\FltMgr ffffb005df267d70 🌽 there can be minifilters there
  1325. > ffffb005df2d5030 \FileSystem\Ntfs ffffb005df2d5180
  1326.  
  1327.  
  1328. ✨✨✨✨✨✨✨✨
  1329. 🌽 Well, the device stack here is fairly standard. What I'm not sure is filesystem filters (or rather minifilters).
  1330. I don't know if they really can affect paging path (probably they shouldn't, but at the same time they probably can).
  1331. So let's dump filters.
  1332.  
  1333. 2: kd> !fltkd.filters 🌽 note the "S" in filterS
  1334. Filter List: ffffb005c82600c0 "Frame 0"
  1335. FLT_FILTER: ffffb005e9097010 "bindflt" "409800" 🌽 for mounting
  1336. FLT_INSTANCE: ffffb005ee495050 "bindflt Instance" "409800"
  1337. FLT_FILTER: ffffb005c82ef7e0 "WdFilter" "328010" 🌽 Windows Defender
  1338. FLT_INSTANCE: ffffb005df26eab0 "WdFilter Instance" "328010"
  1339. FLT_INSTANCE: ffffb005df320b40 "WdFilter Instance" "328010"
  1340. FLT_INSTANCE: ffffb005dfb1d010 "WdFilter Instance" "328010"
  1341. FLT_INSTANCE: ffffb005dfb19b30 "WdFilter Instance" "328010"
  1342. FLT_INSTANCE: ffffb005e4d51b40 "WdFilter Instance" "328010"
  1343. FLT_INSTANCE: ffffb005f5124820 "WdFilter Instance" "328010"
  1344. FLT_INSTANCE: ffffb005f57ea260 "WdFilter Instance" "328010"
  1345. FLT_INSTANCE: ffffb005f6370700 "WdFilter Instance" "328010"
  1346. FLT_FILTER: ffffb005cc224210 "mfehidk" "321300.00" 🌽💩💎 crap, that's McAfee
  1347. FLT_INSTANCE: ffffb005df26f010 "mfehidk" "321300.00"
  1348. FLT_INSTANCE: ffffb005df5e94e0 "mfehidk" "321300.00"
  1349. FLT_INSTANCE: ffffb005e41d97e0 "mfehidk" "321300.00"
  1350. FLT_INSTANCE: ffffb005e48f4060 "mfehidk" "321300.00"
  1351. FLT_INSTANCE: ffffb005e4d3b010 "mfehidk" "321300.00"
  1352. FLT_INSTANCE: ffffb005e4d515f0 "mfehidk" "321300.00"
  1353. FLT_INSTANCE: ffffb005ef37b010 "mfehidk" "321300.00"
  1354. FLT_INSTANCE: ffffb005f49ed010 "mfehidk" "321300.00"
  1355. FLT_INSTANCE: ffffb005f2ced490 "mfehidk" "321300.00"
  1356. FLT_FILTER: ffffb005e4d43be0 "storqosflt" "244000" 🌽 for throttling
  1357. FLT_FILTER: ffffb005e66c9a20 "wcifs" "189900" 🌽 Windows Container Isolation
  1358. FLT_FILTER: ffffb005e6370520 "CldFlt" "180451" 🌽 OneDrive and the like
  1359. FLT_INSTANCE: ffffb005e90e2a20 "CldFlt" "180451"
  1360. FLT_INSTANCE: ffffb005f4ae7010 "CldFlt" "180451"
  1361. FLT_INSTANCE: ffffb005f56f5a20 "CldFlt" "180451"
  1362. FLT_INSTANCE: ffffb005f4cea010 "CldFlt" "180451"
  1363. FLT_FILTER: ffffb005df75a5e0 "FileCrypt" "141100" 🌽 EFS
  1364. FLT_FILTER: ffffb005e4eed010 "luafv" "135000" 🌽 redir to virtual store from protected dirs
  1365. FLT_INSTANCE: ffffb005e90b5010 "luafv" "135000"
  1366. FLT_FILTER: ffffb005e48f8ca0 "npsvctrig" "46000" 🌽 triggers services start on pipes access
  1367. FLT_INSTANCE: ffffb005e3f11b10 "npsvctrig" "46000"
  1368. FLT_FILTER: ffffb005c81d97e0 "Wof" "40700" 🌽 Windows Overlay Filesystem
  1369. FLT_INSTANCE: ffffb005df5e5800 "Wof Instance" "40700"
  1370. FLT_INSTANCE: ffffb005e4d34520 "Wof Instance" "40700"
  1371. FLT_INSTANCE: ffffb005e4d34050 "Wof Instance" "40700"
  1372. FLT_INSTANCE: ffffb005ebf67010 "Wof Instance" "40700"
  1373. FLT_INSTANCE: ffffb005ed352520 "Wof Instance" "40700"
  1374. FLT_INSTANCE: ffffb005ebbd6b20 "Wof Instance" "40700"
  1375. FLT_FILTER: ffffb005c83099b0 "FileInfo" "40500" 🌽 Prefetch-related
  1376. FLT_INSTANCE: ffffb005df26e010 "FileInfo" "40500"
  1377. FLT_INSTANCE: ffffb005df5ef9f0 "FileInfo" "40500"
  1378. FLT_INSTANCE: ffffb005e3f34340 "FileInfo" "40500"
  1379. FLT_INSTANCE: ffffb005e4817690 "FileInfo" "40500"
  1380. FLT_INSTANCE: ffffb005e41857c0 "FileInfo" "40500"
  1381. FLT_INSTANCE: ffffb005f28d2010 "FileInfo" "40500"
  1382. FLT_INSTANCE: ffffb005f4bf0a20 "FileInfo" "40500"
  1383. FLT_INSTANCE: ffffb005f5005a20 "FileInfo" "40500"
  1384.  
  1385. 🌽 Let's check McAfee. Command below outputs bunch of errors, I've cleaned it up for clarity.
  1386. ✨✨✨
  1387. 2: kd> !fltkd.filter ffffb005cc224210 🌽💩 check the McAfee (note singular "filter")
  1388. FLT_FILTER: ffffb005cc224210 "mfehidk" "321300.00"
  1389. FLT_OBJECT: ffffb005cc224210 [02000000] Filter
  1390. RundownRef : 0x00000000000093a8 (18900)
  1391. PointerCount : 0x00000001
  1392. PrimaryLink : [ffffb005e4d43bf0-ffffb005c82ef7f0]
  1393. Frame : ffffb005c8260010 "Frame 0"
  1394. Flags : [0000001a] FilteringInitiated SupportsPipesMailslots BackedByPagefile
  1395. DriverObject : ffffb005c82f0830 🌽🚋
  1396. FilterLink : [ffffb005e4d43bf0-ffffb005c82ef7f0]
  1397. PreVolumeMount : 0000000000000000 (null)
  1398. PostVolumeMount : 0000000000000000 (null) Unable to load image \SystemRoot\system32\drivers\mfehidk.sys, Win32 error 0n2
  1399.  
  1400. FilterUnload : fffff8011802ee00 mfehidk+0x7ee00
  1401. InstanceSetup : fffff8011802e3b0 mfehidk+0x7e3b0
  1402. InstanceQueryTeardown : 0000000000000000 (null)
  1403. InstanceTeardownStart : fffff8011802ec40 mfehidk+0x7ec40
  1404. InstanceTeardownComplete : fffff8011802ecf0 mfehidk+0x7ecf0
  1405. ActiveOpens : (ffffb005cc2243c8) mCount=0
  1406. Communication Port List : (ffffb005cc224418) mCount=0
  1407. Client Port List : (ffffb005cc224468) mCount=0
  1408. VerifierExtension : 0000000000000000
  1409. Operations : ffffb005cc2244c0
  1410. OldDriverUnload : 0000000000000000 (null)
  1411. SupportedContexts : (ffffb005cc224340)
  1412.  
  1413. VolumeContexts : (ffffb005cc224340)
  1414. InstanceContexts : (ffffb005cc224340)
  1415. FileContexts : (ffffb005cc224340)
  1416. StreamContexts : (ffffb005cc224340)
  1417. StreamHandleContexts : (ffffb005cc224340)
  1418. TransactionContext : (ffffb005cc224340)
  1419. (null) : (ffffb005cc224340)
  1420. InstanceList : (ffffb005cc224278)
  1421. FLT_INSTANCE: ffffb005df26f010 "mfehidk" "321300.00"
  1422. FLT_INSTANCE: ffffb005df5e94e0 "mfehidk" "321300.00"
  1423. FLT_INSTANCE: ffffb005e41d97e0 "mfehidk" "321300.00"
  1424. FLT_INSTANCE: ffffb005e48f4060 "mfehidk" "321300.00"
  1425. FLT_INSTANCE: ffffb005e4d3b010 "mfehidk" "321300.00"
  1426. FLT_INSTANCE: ffffb005e4d515f0 "mfehidk" "321300.00"
  1427. FLT_INSTANCE: ffffb005ef37b010 "mfehidk" "321300.00"
  1428. FLT_INSTANCE: ffffb005f49ed010 "mfehidk" "321300.00"
  1429. FLT_INSTANCE: ffffb005f2ced490 "mfehidk" "321300.00"
  1430.  
  1431. ✨✨✨
  1432. 2: kd> !drvobj ffffb005c82f0830 🌽🚋
  1433. Driver object (ffffb005c82f0830) is for:
  1434. \Driver\mfehidk 🌽🚋
  1435. Driver Extension List: (id , addr)
  1436. Device Object list:
  1437. ffffb005c82f3ab0 ffffb005c82f3830
  1438.  
  1439. 2: kd> lmvm mfehidk 🌽🚋 "LoadedModules, Verbose, by-pattern-Match"
  1440. Browse full module list
  1441. start end module name
  1442. fffff801`17fb0000 fffff801`180b5000 mfehidk (export symbols) mfehidk.sys
  1443. Loaded symbol image file: mfehidk.sys
  1444. Image path: \SystemRoot\system32\drivers\mfehidk.sys
  1445. Image name: mfehidk.sys
  1446. Browse all global symbols functions data
  1447. Timestamp: Tue Dec 8 21:10:49 2020 (5FCFEBD9) 🌽💎 McAfee driver is ~7 months old
  1448. CheckSum: 000FF041
  1449. ImageSize: 00105000
  1450. Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
  1451. Information from resource tables:
  1452.  
  1453.  
  1454. ✨✨✨
  1455. Okay, all that info is very pretty, but it doesn't tell us anything substantial right now.
  1456. I'm not even sure this filter affects the paging path. McAfee driver is ~7 months old - not fresh, but not old either.
  1457. We'll keep it in mind as a potential culprit, but most likely McAfee simply was in a wrong place at a wrong time.
  1458.  
  1459.  
  1460. ✨✨✨✨✨✨✨✨
  1461. 🌽 Alright, what else do we have? The device stack is standard. And I highly doubt that Microsoft `volmgr` driver or
  1462. `Ntfs` driver were directly involved in returning 🍄 STATUS_NO_SUCH_DEVICE out of the blue.
  1463.  
  1464. What we need to examine now is not the volume, but the disk device object, and its device stack. How to do it?
  1465. Run Sysinternals WinObj. You'll see directories `\Device\Harddisk0`, `\Device\Harddisk1`, etc. These directories contain
  1466. single device object, like `DR0` or `DR1` (physical drive 0, physical drive 1), and a bunch of symlinks for that drive
  1467. partitions. Symlinks would point to partition devices, such as `\Device\HarddiskVolume4`.
  1468.  
  1469. ✨✨✨
  1470. 2: kd> !object \Device\Harddisk0 🌽 explore Physical Drive 0
  1471. Object: ffff9c89490cc5a0 Type: (ffffb005c50cf640) Directory
  1472. ObjectHeader: ffff9c89490cc570 (new version)
  1473. HandleCount: 1 PointerCount: 5
  1474. Directory Object: ffff9c8947603780 Name: Harddisk0
  1475.  
  1476. Hash Address Type Name
  1477. ---- ------- ---- ----
  1478. 21 ffffb005df271170 Device DR0
  1479. 25 ffff9c8949498460 SymbolicLink Partition0 🌽 Parition0 is not actually a parition
  1480. 26 ffff9c8949499f50 SymbolicLink Partition1 🌽 the only partition on drive0
  1481.  
  1482. 2: kd> !object ffff9c8949498460 🌽 check symlink for Partition0...
  1483. Object: ffff9c8949498460 Type: (ffffb005c50cfd20) SymbolicLink
  1484. ObjectHeader: ffff9c8949498430 (new version)
  1485. HandleCount: 0 PointerCount: 1
  1486. Directory Object: ffff9c89490cc5a0 Name: Partition0
  1487. Flags: 00000000 ( Local )
  1488. Target String is '\Device\Harddisk0\DR0' 🌽 ...it's kind of a loopback, back to the drive
  1489.  
  1490. 2: kd> !object ffff9c8949499f50
  1491. Object: ffff9c8949499f50 Type: (ffffb005c50cfd20) SymbolicLink 🌽 check symlink for Partition1...
  1492. ObjectHeader: ffff9c8949499f20 (new version)
  1493. HandleCount: 0 PointerCount: 1
  1494. Directory Object: ffff9c89490cc5a0 Name: Partition1
  1495. Flags: 00000000 ( Local )
  1496. Target String is '\Device\HarddiskVolume1' 🌽 ...it's not our HarddiskVolume4; move on
  1497.  
  1498. ✨✨✨
  1499. 2: kd> !object \Device\Harddisk1 🌽 explore Physical Drive 1
  1500. Object: ffff9c89490cbce0 Type: (ffffb005c50cf640) Directory
  1501. ObjectHeader: ffff9c89490cbcb0 (new version)
  1502. HandleCount: 1 PointerCount: 8
  1503. Directory Object: ffff9c8947603780 Name: Harddisk1
  1504.  
  1505. Hash Address Type Name
  1506. ---- ------- ---- ----
  1507. 22 ffffb005df273060 Device DR1 🌽💾
  1508. 25 ffff9c89494994a0 SymbolicLink Partition0
  1509. 26 ffff9c89494991d0 SymbolicLink Partition1 🌽 if you click this, that'll be HarddiskVolume2
  1510. 27 ffff9c8949499920 SymbolicLink Partition2
  1511. 28 ffff9c894949a580 SymbolicLink Partition3 🌽🖱️ I'm feeling lucky, let's just click this
  1512. 29 ffff9c894949abb0 SymbolicLink Partition4
  1513.  
  1514. 2: kd> !object ffff9c894949a580 🌽 check symlink for Partition 3...
  1515. Object: ffff9c894949a580 Type: (ffffb005c50cfd20) SymbolicLink
  1516. ObjectHeader: ffff9c894949a550 (new version)
  1517. HandleCount: 0 PointerCount: 1
  1518. Directory Object: ffff9c89490cbce0 Name: Partition3
  1519. Flags: 00000000 ( Local )
  1520. Target String is '\Device\HarddiskVolume4' 🌽 ...yes, it's the C: volume
  1521.  
  1522. ✨✨✨✨✨✨✨✨
  1523. 🌽 Excellent, we've identified that drive device with pagefile is \Device\Harddisk1\DR1, 💾 ffffb005df273060.
  1524.  
  1525. 2: kd> dt _device_object ffffb005df273060 🌽💾 eye the disk device object
  1526. nt!_DEVICE_OBJECT
  1527. +0x000 Type : 0n3
  1528. +0x002 Size : 0xa00
  1529. +0x004 ReferenceCount : 0n11
  1530. +0x008 DriverObject : 0xffffb005`df26fdc0 _DRIVER_OBJECT
  1531. +0x010 NextDevice : 0xffffb005`df271170 _DEVICE_OBJECT
  1532. +0x018 AttachedDevice : 0xffffb005`df274040 _DEVICE_OBJECT
  1533. +0x020 CurrentIrp : (null)
  1534. +0x028 Timer : (null)
  1535. +0x030 Flags : 0x1000050
  1536. +0x034 Characteristics : 0x100
  1537. +0x038 Vpb : 0xffffb005`df2867e0 _VPB
  1538. +0x040 DeviceExtension : 0xffffb005`df2731b0 Void
  1539. +0x048 DeviceType : 7
  1540. +0x04c StackSize : 3 ''
  1541. +0x050 Queue : <anonymous-tag>
  1542. +0x098 AlignmentRequirement : 3
  1543. +0x0a0 DeviceQueue : _KDEVICE_QUEUE
  1544. +0x0c8 Dpc : _KDPC
  1545. +0x108 ActiveThreadCount : 0
  1546. +0x110 SecurityDescriptor : 0xffff9c89`49279860 Void
  1547. +0x118 DeviceLock : _KEVENT
  1548. +0x130 SectorSize : 0
  1549. +0x132 Spare1 : 1
  1550. +0x138 DeviceObjectExtension : 0xffffb005`df273a60 _DEVOBJ_EXTENSION
  1551. +0x140 Reserved : (null)
  1552.  
  1553. 2: kd> !devobj ffffb005df273060 🌽💾 eye the disk device object
  1554. Device object (ffffb005df273060) is for:
  1555. DR1 \Driver\disk DriverObject ffffb005df26fdc0
  1556. Current Irp 00000000 RefCount 11 Type 00000007 Flags 01000050
  1557. Vpb 0xffffb005df2867e0 SecurityDescriptor ffff9c8949279860 DevExt ffffb005df2731b0 DevObjExt ffffb005df273a60 Dope ffffb005df286770
  1558. ExtensionFlags (0000000000)
  1559. Characteristics (0x00000100) FILE_DEVICE_SECURE_OPEN
  1560. AttachedDevice (Upper) ffffb005df274040 \Driver\partmgr
  1561. AttachedTo (Lower) ffffb005c5106b00 \Driver\ACPI
  1562. Device queue is not busy.
  1563.  
  1564. ✨✨✨
  1565. 2: kd> !devstack ffffb005df273060 🌽💾 check where's the device in the device stack
  1566. !DevObj !DrvObj !DevExt ObjectName
  1567. ffffb005df274040 \Driver\partmgr ffffb005df274190
  1568. > ffffb005df273060 \Driver\disk ffffb005df2731b0 DR1
  1569. ffffb005c5106b00 \Driver\ACPI ffffb005c80d5be0
  1570. ffffb005cc218050 Unable to load image \SystemRoot\System32\drivers\iaStorAC.sys, Win32 error 0n2 🌽 wtf: no symbols
  1571. \Driver\iaStorAC ffffb005cc2181a0 0000003e
  1572. !DevNode ffffb005c50db010 :
  1573. DeviceInst is "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000"
  1574. ServiceName is "disk"
  1575.  
  1576. 2: kd> !devstack ffffb005df273060 🌽💾 repeat the command to get rid of the error
  1577. !DevObj !DrvObj !DevExt ObjectName
  1578. ffffb005df274040 \Driver\partmgr ffffb005df274190
  1579. > ffffb005df273060 \Driver\disk ffffb005df2731b0 DR1
  1580. ffffb005c5106b00 \Driver\ACPI ffffb005c80d5be0
  1581. ffffb005cc218050 \Driver\iaStorAC ffffb005cc2181a0 0000003e 🌽☣️💎 Intel-bloody-Rapid-Storage in device stack
  1582. !DevNode ffffb005c50db010 :
  1583. DeviceInst is "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000" 🌽💿💎 hdd device name
  1584. ServiceName is "disk"
  1585.  
  1586. ✨✨✨
  1587. 2: kd> !DevNode ffffb005c50db010
  1588. DevNode 0xffffb005c50db010 for PDO 0xffffb005cc218050
  1589. Parent 0xffffb005c81eeca0 Sibling 0000000000 Child 0000000000
  1590. InstancePath is "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000"
  1591. ServiceName is "disk"
  1592. State = DeviceNodeStarted (0x308)
  1593. Previous State = DeviceNodeEnumerateCompletion (0x30d)
  1594. StateHistory[13] = DeviceNodeEnumerateCompletion (0x30d)
  1595. StateHistory[12] = DeviceNodeEnumeratePending (0x30c)
  1596. StateHistory[11] = DeviceNodeStarted (0x308) 🌽 unsure if these "starts" normal, may be they are
  1597. StateHistory[10] = DeviceNodeEnumerateCompletion (0x30d)
  1598. StateHistory[09] = DeviceNodeEnumeratePending (0x30c)
  1599. StateHistory[08] = DeviceNodeStarted (0x308)
  1600. StateHistory[07] = DeviceNodeEnumerateCompletion (0x30d)
  1601. StateHistory[06] = DeviceNodeEnumeratePending (0x30c)
  1602. StateHistory[05] = DeviceNodeStarted (0x308)
  1603. StateHistory[04] = DeviceNodeEnumerateCompletion (0x30d)
  1604. StateHistory[03] = DeviceNodeEnumeratePending (0x30c)
  1605. StateHistory[02] = DeviceNodeStarted (0x308)
  1606. StateHistory[01] = DeviceNodeEnumerateCompletion (0x30d)
  1607. StateHistory[00] = DeviceNodeEnumeratePending (0x30c)
  1608. StateHistory[19] = DeviceNodeStarted (0x308)
  1609. StateHistory[18] = DeviceNodeEnumerateCompletion (0x30d)
  1610. StateHistory[17] = DeviceNodeEnumeratePending (0x30c)
  1611. StateHistory[16] = DeviceNodeStarted (0x308)
  1612. StateHistory[15] = DeviceNodeEnumerateCompletion (0x30d)
  1613. StateHistory[14] = DeviceNodeEnumeratePending (0x30c)
  1614. Flags (0x24000130) DNF_ENUMERATED, DNF_IDS_QUERIED,
  1615. DNF_NO_RESOURCE_REQUIRED, DNF_NO_LOWER_DEVICE_FILTERS,
  1616. DNF_NO_UPPER_DEVICE_FILTERS
  1617. UserFlags (0x00000008) DNUF_NOT_DISABLEABLE
  1618. CapabilityFlags (0x00400380) SilentInstall, RawDeviceOK,
  1619. SurpriseRemovalOK
  1620. Unknown flags 0x00400000
  1621. DisableableDepends = 1 (including self)
  1622.  
  1623. ✨✨✨
  1624. 2: kd> lmvm iaStorAC 🌽☣️ "LoadedModules, Verbose, by-pattern-Match"
  1625. Browse full module list
  1626. start end module name
  1627. fffff801`171e0000 fffff801`17dc4000 iaStorAC (deferred)
  1628. Image path: \SystemRoot\System32\drivers\iaStorAC.sys
  1629. Image name: iaStorAC.sys
  1630. Browse all global symbols functions data
  1631. Timestamp: Wed Mar 18 08:02:55 2020 (5E71D5AF)
  1632. CheckSum: 00157209
  1633. ImageSize: 00BE4000
  1634. Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
  1635. Information from resource tables:
  1636.  
  1637. ✨✨✨✨✨✨✨✨
  1638. 🌽 Now that's a hit. "iaStorAC.sys" is related to Intel Rapid Storage (Intel RST) technology.
  1639. It is a well-known offender! And it's right in our storage device stack, so it very well may have returned error.
  1640. I don't know if the hardware controller is crap, or if the driver is crap, or if it's the combination of both, but in
  1641. my own experience the technology is a mess with unknown benefit. If you google "iaStorAC", the first link would be
  1642. about BSOD (on Microsoft.com), the second link would be about BSOD (on Intel.com, in its own knowledge base 😹), and
  1643. in fact all other links of the page would be about iaStorAC BSODs.
  1644.  
  1645. ✨✨✨
  1646. It might not sound fair to blame the driver just because it sits in the failing devstack and is known to have caused
  1647. a bunch of BSODs in the past. But if we won't find more evidence against this driver, we'd still recommend disable the
  1648. Intel Rapid Storage Technology in the BIOS.
  1649. SPOILER: we will find more evidence towards the end of the article, using storagekd extension.
  1650.  
  1651.  
  1652. ✨✨✨✨✨✨✨✨
  1653. 🌽 But let's continue investigation in search for more clues. We may still learn something new about the device or the
  1654. system. We know the hdd model name, so let's display PoFx-registered devices and find our 🌽💿 Prod_WDC_PC_SN530_SDB.
  1655.  
  1656. ✨✨✨
  1657. 2: kd> !fxdevice 🌽 display info about all devices registered in Power Management Framework (PoFx)
  1658. ********************************************************************************
  1659. Dumping non-idle PoFx devices
  1660. ********************************************************************************
  1661. !fxdevice 0xffffb005c826bae0
  1662. Device Object: 0xffffb005c5056e10
  1663. DevNode: 0xffffb005c82068a0
  1664. UniqueId: "\_SB.PCI0"
  1665. InstancePath: "ACPI\PNP0A08\0"
  1666. Device Power State: PowerDeviceD0
  1667. Component Count: 1
  1668. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 18)
  1669.  
  1670. !fxdevice 0xffffb005c8272010
  1671. Device Object: 0xffffb005c8294360
  1672. DevNode: 0xffffb005c8211ca0
  1673. UniqueId: "\_SB.PCI0.RP05"
  1674. InstancePath: "PCI\VEN_8086&DEV_34BC&SUBSYS_10521043&REV_30\3&11583659&1&E4"
  1675. Device Power State: PowerDeviceD0
  1676. Component Count: 1
  1677. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1678.  
  1679. !fxdevice 0xffffb005c82a35e0
  1680. Device Object: 0xffffb005c8290360
  1681. DevNode: 0xffffb005c818eca0
  1682. UniqueId: "\_SB.PCI0.RP01"
  1683. InstancePath: "PCI\VEN_8086&DEV_34B8&SUBSYS_10521043&REV_30\3&11583659&1&E0"
  1684. Device Power State: PowerDeviceD0
  1685. Component Count: 1
  1686. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 1)
  1687.  
  1688. !fxdevice 0xffffb005cc3668a0
  1689. Device Object: 0xffffb005c81ec360
  1690. DevNode: 0xffffb005c820dca0
  1691. UniqueId: "\_SB.PCI0.XHC"
  1692. InstancePath: "PCI\VEN_8086&DEV_34ED&SUBSYS_201F1043&REV_30\3&11583659&1&A0"
  1693. Device Power State: PowerDeviceD0
  1694. Component Count: 1
  1695. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1696.  
  1697. !fxdevice 0xffffb005df26c010
  1698. Device Object: 0xffffb005cc33cdc0
  1699. DevNode: 0xffffb005cc33fcb0
  1700. UniqueId: "\_SB.PCI0.XHC.RHUB"
  1701. InstancePath: "USB\ROOT_HUB30\4&36f9711c&0&0"
  1702. Device Power State: PowerDeviceD0
  1703. Component Count: 1
  1704. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1705.  
  1706. !fxdevice 0xffffb005df28a050
  1707. Device Object: 0xffffb005cc216050
  1708. DevNode: 0xffffb005c81dcc30
  1709. UniqueId: "\_SB.PCI0.SAT0.PRT0"
  1710. InstancePath: "SCSI\Disk&Ven_WDC&Prod_WD10SPZX-80Z10T2\4&e0fc7e4&0&000000"
  1711. Device Power State: PowerDeviceD0
  1712. Component Count: 1
  1713. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1714.  
  1715. !fxdevice 0xffffb005df2a0050 🌽🌕 we'll enter this command below
  1716. Device Object: 0xffffb005cc218050
  1717. DevNode: 0xffffb005c50db010
  1718. UniqueId: "\_SB.PCI0.SAT0.NVM3"
  1719. InstancePath: "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000" 🌽💿 found the hdd!
  1720. Device Power State: PowerDeviceD0
  1721. Component Count: 1
  1722. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 63)
  1723.  
  1724. !fxdevice 0xffffb005df5db010
  1725. Device Object: 0xffffb005df556830
  1726. DevNode: 0xffffb005df54f8a0
  1727. UniqueId: "USB\VID_046D&PID_C52B&MI_01\6&1458a8f4&0&0001"
  1728. InstancePath: "USB\VID_046D&PID_C52B&MI_01\6&1458a8f4&0&0001"
  1729. Device Power State: PowerDeviceD0
  1730. Component Count: 1
  1731. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 3)
  1732.  
  1733. !fxdevice 0xffffb005df5a7050
  1734. Device Object: 0xffffb005df82f060
  1735. DevNode: 0xffffb005df54fbd0
  1736. UniqueId: "USB\VID_046D&PID_C52B&MI_02\6&1458a8f4&0&0002"
  1737. InstancePath: "USB\VID_046D&PID_C52B&MI_02\6&1458a8f4&0&0002"
  1738. Device Power State: PowerDeviceD0
  1739. Component Count: 1
  1740. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1741.  
  1742. !fxdevice 0xffffb005dfaad8e0
  1743. Device Object: 0xffffb005df996d60
  1744. DevNode: 0xffffb005c8206bd0
  1745. UniqueId: "\_SB.PR00"
  1746. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_1"
  1747. Device Power State: PowerDeviceD0
  1748. Component Count: 1
  1749. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1750.  
  1751. !fxdevice 0xffffb005dfacbac0
  1752. Device Object: 0xffffb005dfa3a790
  1753. DevNode: 0xffffb005c81e08a0
  1754. UniqueId: "\_SB.PR01"
  1755. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_2"
  1756. Device Power State: PowerDeviceD0
  1757. Component Count: 1
  1758. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1759.  
  1760. !fxdevice 0xffffb005dfab48a0
  1761. Device Object: 0xffffb005dfa3acc0
  1762. DevNode: 0xffffb005c81e0bd0
  1763. UniqueId: "\_SB.PR02"
  1764. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_3"
  1765. Device Power State: PowerDeviceD0
  1766. Component Count: 1
  1767. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1768.  
  1769. !fxdevice 0xffffb005dfabb8a0
  1770. Device Object: 0xffffb005dfa3db60
  1771. DevNode: 0xffffb005c81e18a0
  1772. UniqueId: "\_SB.PR03"
  1773. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_4"
  1774. Device Power State: PowerDeviceD0
  1775. Component Count: 1
  1776. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1777.  
  1778. !fxdevice 0xffffb005dfac88a0
  1779. Device Object: 0xffffb005dfa3dda0
  1780. DevNode: 0xffffb005c81e1bd0
  1781. UniqueId: "\_SB.PR04"
  1782. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_5"
  1783. Device Power State: PowerDeviceD0
  1784. Component Count: 1
  1785. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1786.  
  1787. !fxdevice 0xffffb005dfaae8a0
  1788. Device Object: 0xffffb005dfa48040
  1789. DevNode: 0xffffb005c81e28a0
  1790. UniqueId: "\_SB.PR05"
  1791. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_6"
  1792. Device Power State: PowerDeviceD0
  1793. Component Count: 1
  1794. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1795.  
  1796. !fxdevice 0xffffb005dfa988a0
  1797. Device Object: 0xffffb005dfa48d80
  1798. DevNode: 0xffffb005c81e2bd0
  1799. UniqueId: "\_SB.PR06"
  1800. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_7"
  1801. Device Power State: PowerDeviceD0
  1802. Component Count: 1
  1803. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1804.  
  1805. !fxdevice 0xffffb005dfaba8a0
  1806. Device Object: 0xffffb005dfa4b410
  1807. DevNode: 0xffffb005c81e38a0
  1808. UniqueId: "\_SB.PR07"
  1809. InstancePath: "ACPI\GenuineIntel_-_Intel64_Family_6_Model_126_-_Intel(R)_Core(TM)_i7-1065G7_CPU_@_1.30GHz\_8"
  1810. Device Power State: PowerDeviceD0
  1811. Component Count: 1
  1812. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1813.  
  1814. !fxdevice 0xffffb005dfaaf8a0
  1815. Device Object: 0xffffb005c5107c60
  1816. DevNode: 0xffffb005c81e5bd0
  1817. UniqueId: "\_SB.GPI0"
  1818. InstancePath: "ACPI\INT3455\2&daba3ff&1"
  1819. Device Power State: PowerDeviceD0
  1820. Component Count: 11
  1821. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1822. Component 1: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1823. Component 2: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1824. Component 3: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1825. Component 4: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1826. Component 5: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1827. Component 6: Current:F0/Deepest:F0 - ACTIVE (RefCount = 4)
  1828. Component 7: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1829. Component 8: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1830. Component 9: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1831. Component 10: Current:F0/Deepest:F0 - ACTIVE (RefCount = 2)
  1832.  
  1833. !fxdevice 0xffffb005dfae0010
  1834. Device Object: 0xffffb005c828c360
  1835. DevNode: 0xffffb005c826fca0
  1836. UniqueId: "\_SB.PCI0.HECI"
  1837. InstancePath: "PCI\VEN_8086&DEV_34E0&SUBSYS_10521043&REV_30\3&11583659&1&B0"
  1838. Device Power State: PowerDeviceD0
  1839. Component Count: 1
  1840. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 1)
  1841.  
  1842. !fxdevice 0xffffb005dfadf520
  1843. Device Object: 0xffffb005c8205060
  1844. DevNode: 0xffffb005c8296aa0
  1845. UniqueId: "\_SB.PCI0.HDAS"
  1846. InstancePath: "PCI\VEN_8086&DEV_34C8&SUBSYS_10521043&REV_30\3&11583659&1&FB"
  1847. Device Power State: PowerDeviceD0
  1848. Component Count: 1
  1849. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 1)
  1850.  
  1851. !fxdevice 0xffffb005e422f820
  1852. Device Object: 0xffffb005c820b360
  1853. DevNode: 0xffffb005c828eca0
  1854. UniqueId: "\_SB.PCI0.PSDC"
  1855. InstancePath: "PCI\VEN_8086&DEV_34F8&SUBSYS_72708086&REV_30\3&11583659&1&A5"
  1856. Device Power State: PowerDeviceD3
  1857. Component Count: 1
  1858. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1859.  
  1860. !fxdevice 0xffffb005e40a6820
  1861. Device Object: 0xffffb005dfb22b50
  1862. DevNode: 0xffffb005e4058b00
  1863. UniqueId: "\_SB.PCI0.I2C1.ETPD"
  1864. InstancePath: "ACPI\ELAN1300\1"
  1865. Device Power State: PowerDeviceD0
  1866. Component Count: 1
  1867. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1868.  
  1869. !fxdevice 0xffffb005df948010
  1870. Device Object: 0xffffb005dfb09c00
  1871. DevNode: 0xffffb005dfc2f160
  1872. UniqueId: "INTELAUDIO\DSP_CTLR_DEV_34C8&VEN_8086&DEV_0222&SUBSYS_10521043&REV_0030\4&24c22537&0&0800"
  1873. InstancePath: "INTELAUDIO\DSP_CTLR_DEV_34C8&VEN_8086&DEV_0222&SUBSYS_10521043&REV_0030\4&24c22537&0&0800"
  1874. Device Power State: PowerDeviceD0
  1875. Component Count: 1
  1876. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 1)
  1877.  
  1878. !fxdevice 0xffffb005e4d3b560
  1879. Device Object: 0xffffb005df5cc030
  1880. DevNode: 0xffffb005c818aca0
  1881. UniqueId: "\_SB.PCI0.GFX0"
  1882. InstancePath: "PCI\VEN_8086&DEV_8A52&SUBSYS_1B2F1043&REV_07\3&11583659&1&10"
  1883. Device Power State: PowerDeviceD0
  1884. Component Count: 17
  1885. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1886. Component 1: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1887. Component 2: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1888. Component 3: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1889. Component 4: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1890. Component 5: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1891. Component 6: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1892. Component 7: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1893. Component 8: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1894. Component 9: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1895. Component 10: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1896. Component 11: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1897. Component 12: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1898. Component 13: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1899. Component 14: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1900. Component 15: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1901. Component 16: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1902.  
  1903. !fxdevice 0xffffb005e7ff1a30
  1904. Device Object: 0xffffb005df94d030
  1905. DevNode: 0xffffb005c8271520
  1906. UniqueId: "\_SB.PCI0.RP05.PEGP"
  1907. InstancePath: "PCI\VEN_10DE&DEV_1D16&SUBSYS_1B2F1043&REV_A1\4&38bf707b&0&00E4"
  1908. Device Power State: PowerDeviceD0
  1909. Component Count: 16
  1910. Component 0: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1911. Component 1: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1912. Component 2: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1913. Component 3: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1914. Component 4: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1915. Component 5: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1916. Component 6: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1917. Component 7: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1918. Component 8: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1919. Component 9: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1920. Component 10: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1921. Component 11: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1922. Component 12: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1923. Component 13: Current:F0/Deepest:F1 - ACTIVE (RefCount = 14)
  1924. Component 14: Current:F0/Deepest:F0 - ACTIVE (RefCount = 3)
  1925. Component 15: Current:F0/Deepest:F1 - ACTIVE (RefCount = 2)
  1926.  
  1927. !fxdevice 0xffffb005e4d76010
  1928. Device Object: 0xffffb005dfb22d80
  1929. DevNode: 0xffffb005dfc2f490
  1930. UniqueId: "\_SB.PCI0.SPI0.FPRT"
  1931. InstancePath: "ACPI\ELAN7001\4&1240dbbf&0"
  1932. Device Power State: PowerDeviceD3
  1933. Component Count: 1
  1934. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 3)
  1935.  
  1936. !fxdevice 0xffffb005eb71f010
  1937. Device Object: 0xffffb005df558880
  1938. DevNode: 0xffffb005df549b70
  1939. UniqueId: "USB\VID_046D&PID_C52B&MI_00\6&1458a8f4&0&0000"
  1940. InstancePath: "USB\VID_046D&PID_C52B&MI_00\6&1458a8f4&0&0000"
  1941. Device Power State: PowerDeviceD0
  1942. Component Count: 1
  1943. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 4)
  1944.  
  1945. ********************************************************************************
  1946. Dumping idle D0 PoFx devices
  1947. ********************************************************************************
  1948. !fxdevice 0xffffb005dfae0a30
  1949. Device Object: 0xffffb005e47bf050
  1950. DevNode: 0xffffb005e40584a0
  1951. UniqueId: "INTELAUDIO\FUNC_01&VEN_10EC&DEV_0256&SUBSYS_10431052&REV_1000\5&55d82b7&0&0001"
  1952. InstancePath: "INTELAUDIO\FUNC_01&VEN_10EC&DEV_0256&SUBSYS_10431052&REV_1000\5&55d82b7&0&0001"
  1953. Device Power State: PowerDeviceD0
  1954. Component Count: 1
  1955. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  1956.  
  1957. !fxdevice 0xffffb005df9b2010
  1958. Device Object: 0xffffb005dfb24530
  1959. DevNode: 0xffffb005dfc1d980
  1960. UniqueId: "INTELAUDIO\FUNC_01&VEN_8086&DEV_280F&SUBSYS_80860101&REV_1000\5&55d82b7&0&0201"
  1961. InstancePath: "INTELAUDIO\FUNC_01&VEN_8086&DEV_280F&SUBSYS_80860101&REV_1000\5&55d82b7&0&0201"
  1962. Device Power State: PowerDeviceD0
  1963. Component Count: 1
  1964. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  1965.  
  1966. ********************************************************************************
  1967. Dumping idle non-D0 PoFx devices
  1968. ********************************************************************************
  1969. !fxdevice 0xffffb005dfadea30
  1970. Device Object: 0xffffb005c820f360
  1971. DevNode: 0xffffb005c8294ca0
  1972. UniqueId: "\_SB.PCI0.I2C1"
  1973. InstancePath: "PCI\VEN_8086&DEV_34E9&SUBSYS_10521043&REV_30\3&11583659&1&A9"
  1974. Device Power State: PowerDeviceD3
  1975. Component Count: 1
  1976. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  1977.  
  1978. !fxdevice 0xffffb005dfade010
  1979. Device Object: 0xffffb005c820d360
  1980. DevNode: 0xffffb005c8290ca0
  1981. UniqueId: "\_SB.PCI0.I2C0"
  1982. InstancePath: "PCI\VEN_8086&DEV_34E8&SUBSYS_10521043&REV_30\3&11583659&1&A8"
  1983. Device Power State: PowerDeviceD3
  1984. Component Count: 1
  1985. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  1986.  
  1987. !fxdevice 0xffffb005e3d9c820
  1988. Device Object: 0xffffb005c81ee360
  1989. DevNode: 0xffffb005c81e8aa0
  1990. UniqueId: "\_SB.PCI0.UA00"
  1991. InstancePath: "PCI\VEN_8086&DEV_34A8&SUBSYS_10521043&REV_30\3&11583659&1&F0"
  1992. Device Power State: PowerDeviceD3
  1993. Component Count: 1
  1994. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  1995.  
  1996. !fxdevice 0xffffb005e4285820
  1997. Device Object: 0xffffb005c818e360
  1998. DevNode: 0xffffb005c818caa0
  1999. UniqueId: "\_SB.PCI0.SPI0"
  2000. InstancePath: "PCI\VEN_8086&DEV_34AA&SUBSYS_10521043&REV_30\3&11583659&1&F2"
  2001. Device Power State: PowerDeviceD3
  2002. Component Count: 1
  2003. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  2004.  
  2005. !fxdevice 0xffffb005e4dbd010
  2006. Device Object: 0xffffb005dfad8c80
  2007. DevNode: 0xffffb005e40587d0
  2008. UniqueId: "{DD8E82AE-334B-49A2-AEAE-AEB0FD5C40DD}\DetectionVerification\5&55d82b7&0&0"
  2009. InstancePath: "{DD8E82AE-334B-49A2-AEAE-AEB0FD5C40DD}\DetectionVerification\5&55d82b7&0&0"
  2010. Device Power State: PowerDeviceD3
  2011. Component Count: 1
  2012. Component 0: Current:F0/Deepest:F0 - IDLE (RefCount = 0)
  2013.  
  2014.  
  2015. 🌽 ^^ That was a lot of output, but we've found our hdd. Let's check it out.
  2016. ✨✨✨
  2017. 2: kd> !fxdevice 0xffffb005df2a0050 🌽🌕 enter suggested command from above
  2018. !fxdevice 0xffffb005df2a0050
  2019. Device Object: 0xffffb005cc218050 🌽 the device
  2020. DevNode: 0xffffb005c50db010
  2021. UniqueId: "\_SB.PCI0.SAT0.NVM3"
  2022. InstancePath: "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000" 🌽💿 hdd model name
  2023. Device Power State: PowerDeviceD0 🌽 power state: D0, fully powered
  2024. PEP Owner: Default PEP
  2025. Acpi Plugin: 0
  2026. Acpi Handle: 0
  2027. Device Status Flags: DevicePowerRequired
  2028. Device Idle Timeout: 0x2cb417800
  2029. Device Power On: No Activity
  2030. Device Power Off: No Activity
  2031. Device Unregister: No Activity
  2032. Component Count: 1
  2033. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 63)
  2034. Component Unsatisfied DRIPS Constraint Time: 05:32:41.907
  2035. Pep Component: 0xffffb005df262bf0
  2036. Active: 0 Latency: 0 Residency: 0 Wake: 0 Dx IRP: 0 WW IRP: 0
  2037. Component Idle State Change: No Activity
  2038. Component Activation: No Activity
  2039. Component Active: No Activity
  2040.  
  2041. 2: kd> !devobj 0xffffb005cc218050 🌽 the device
  2042. Device object (ffffb005cc218050) is for:
  2043. 0000003e \Driver\iaStorAC DriverObject ffffb005c81d0c60 🌽☣️ yeah, iaStorAC
  2044. Current Irp 00000000 RefCount 0 Type 00000007 Flags 00001050
  2045. SecurityDescriptor ffff9c8949279860 DevExt ffffb005cc2181a0 DevObjExt ffffb005cc219120 DevNode ffffb005c50db010
  2046. ExtensionFlags (0000000000)
  2047. Characteristics (0x00000180) FILE_AUTOGENERATED_DEVICE_NAME, FILE_DEVICE_SECURE_OPEN
  2048. AttachedDevice (Upper) ffffb005c5106b00 \Driver\ACPI
  2049. Device queue is not busy.
  2050.  
  2051. 🌽 Let's see a magic trick, the power of Ⓥ.
  2052. ✨✨✨
  2053. 2: kd> !fxdevice -v 0xffffb005df2a0050 🌽🌕 enter suggested command from above, with "-v"
  2054. !fxdevice 0xffffb005df2a0050
  2055. Device Object: 0xffffb005cc218050
  2056. DevNode: 0xffffb005c50db010
  2057. UniqueId: "\_SB.PCI0.SAT0.NVM3"
  2058. InstancePath: "SCSI\Disk&Ven_NVMe&Prod_WDC_PC_SN530_SDB\4&e0fc7e4&0&030000"
  2059. Device Power State: PowerDeviceD0
  2060. PEP Owner: Default PEP
  2061. Acpi Plugin: 0
  2062. Acpi Handle: 0
  2063. Device Status Flags: DevicePowerRequired
  2064. Device Idle Timeout: 0x2cb417800
  2065. Device Power On: No Activity
  2066. Device Power Off: No Activity
  2067. Device Unregister: No Activity
  2068. Component Count: 1
  2069. Component 0: Current:F0/Deepest:F0 - ACTIVE (RefCount = 63)
  2070. Component Unsatisfied DRIPS Constraint Time: 05:32:41.907
  2071. Pep Component: 0xffffb005df262bf0
  2072. Active: 0 Latency: 0 Residency: 0 Wake: 0 Dx IRP: 0 WW IRP: 0
  2073. Component Idle State Change: No Activity
  2074. Component Activation: No Activity
  2075. Component Active: No Activity
  2076. Log has 445938 entries starting at 50: 🌽 got PoFx log for 64 last events for the device
  2077. # IntTime CPU Cid Tid
  2078. --- ---------------- ---- ---- ----
  2079. 🌽🠋 this column is InterruptTime (that is, uptime in centums), aka kuser.InterruptTime
  2080. 50 00000504cd2465fb 4 0 0 Power not required from default PEP
  2081. 51 00000504cd2a29e8 3 4 7064 Power required from default PEP
  2082. 52 00000504cd2a29e8 3 4 7064 Device powered
  2083. 53 00000504cdc4c080 6 4 18 Power not required from default PEP
  2084. 54 00000504cdec698b 6 4 7064 Power required from default PEP
  2085. 55 00000504cdec698b 6 4 7064 Device powered
  2086. 56 00000504ce8748cb 0 4 18 Power not required from default PEP
  2087. 57 00000504cec8b90a 6 4 7598 Power required from default PEP
  2088. 58 00000504cec8b90a 6 4 7598 Device powered
  2089. 59 00000504cf633629 3 4 18 Power not required from default PEP
  2090. 60 00000504cfe8666e 6 4 12a0 Power required from default PEP
  2091. 61 00000504cfe8666e 6 4 12a0 Device powered
  2092. 62 00000504cfe9e730 6 0 0 Power not required from default PEP
  2093. 63 00000504d0ea2750 1 4 5558 Power required from default PEP
  2094. 0 00000504d0ea2750 1 4 5558 Device powered
  2095. 1 00000504d0ebac70 1 0 0 Power not required from default PEP
  2096. 2 00000504d0ebac70 1 4 5558 Power required from default PEP
  2097. 3 00000504d0ebac70 1 4 5558 Device powered
  2098. 4 00000504d1848e50 1 4 18 Power not required from default PEP
  2099. 5 00000504d1fe4bf0 6 7c 35c Power required from default PEP
  2100. 6 00000504d1fe4bf0 6 7c 35c Device powered
  2101. 7 00000504d1ffd66f 6 0 0 Power not required from default PEP
  2102. 8 00000504d1ffdabe 2 4 1d0 Power required from default PEP
  2103. 9 00000504d1ffdabe 2 4 1d0 Device powered
  2104. 10 00000504d298f046 7 4 18 Power not required from default PEP
  2105. 11 00000504d3e86e68 4 4 7064 Power required from default PEP
  2106. 12 00000504d3e86e68 4 4 7064 Device powered
  2107. 13 00000504d3e9f857 4 0 0 Power not required from default PEP
  2108. 14 00000504d3e9fcfd 3 4 7064 Power required from default PEP
  2109. 15 00000504d3e9fcfd 3 4 7064 Device powered
  2110. 16 00000504d484bbc7 4 4 18 Power not required from default PEP
  2111. 17 00000504d6d49cb1 6 4 7064 Power required from default PEP
  2112. 18 00000504d6d49cb1 6 4 7064 Device powered
  2113. 19 00000504d6da123e 6 0 0 Power not required from default PEP
  2114. 20 00000504d6da1605 2 4 7064 Power required from default PEP
  2115. 21 00000504d6da1605 2 4 7064 Device powered
  2116. 22 00000504d7735f88 0 4 18 Power not required from default PEP
  2117. 23 00000504d7856111 6 4 7064 Power required from default PEP
  2118. 24 00000504d7856111 6 4 7064 Device powered
  2119. 25 00000504d8209a3c 1 4 18 Power not required from default PEP
  2120. 26 00000504d8ed6265 2 4 e4 Power required from default PEP
  2121. 27 00000504d8ed6265 2 4 e4 Device powered
  2122. 28 00000504d8eec3ab 2 0 0 Power not required from default PEP
  2123. 29 00000504d8eec3ab 2 4 e4 Power required from default PEP
  2124. 30 00000504d8eec3ab 2 4 e4 Device powered
  2125. 31 00000504d937ef65 7 4 4f98 Power IRP requested with status 0
  2126. 32 00000504d937ef65 7 4 4f98 Power IRP type D3 dispatched to device stack
  2127. 33 00000504d93950de 4 4 10c Power IRP completed with status 0
  2128. 34 00000504d93950de 4 4 10c Device power state changed to D3 🌽 D3: lowest power state
  2129. 35 0000051e33474576 6 4 7e78 Power IRP requested with status 0 🌽 note InterruptTime gap
  2130. 36 0000051e33474576 6 4 7e78 Power IRP type D0 dispatched to device stack
  2131. 37 0000051e3440aeac 6 0 0 Device power state changed to D0 🌽 D0: device working state
  2132. 38 0000051e34689268 6 0 0 Power IRP completed with status 0
  2133. 39 0000051e3468e0fe 3 12a4 1710 Power not required from default PEP
  2134. 40 0000051e3468e0fe 2 4 745c Power required from default PEP
  2135. 41 0000051e3468e0fe 2 4 745c Device powered
  2136. 42 0000051e35304f56 2 4 7494 Power IRP requested with status 0
  2137. 43 0000051e35304f56 2 4 7494 Power IRP type D3 dispatched to device stack
  2138. 44 0000051e35311477 4 4 10c Power IRP completed with status 0
  2139. 45 0000051e35311477 4 4 10c Device power state changed to D3 🌽 D3: lowest power state
  2140. 46 0000054cae43feb3 1 4 2760 Power IRP requested with status 0 🌽 note InterruptTime gap again
  2141. 47 0000054cae43feb3 1 4 2760 Power IRP type D0 dispatched to device stack
  2142. 48 0000054cae7aee1f 1 4 72cc Device power state changed to D0 🌽 D0: device working state
  2143. 49 0000054caea24ca6 1 0 0 Power IRP completed with status 0 🌽⏰ note IRP completion InterruptTime
  2144.  
  2145. ✨✨✨
  2146. 🌽 We got ourselves nice PoFx log for the disk device, with exact time of each event.
  2147. The time is in centums (100 ns units) since system boot. SystemTime changes do not affect the InterruptTime while
  2148. system is working. But InterruptTime includes Sleep/Hibernate time (you can sorta conclude that from D3->D0 time gaps).
  2149. If you see this message, you may want to visit the mysterious area is.gd/laboratory_rule38 - just for fun.
  2150. Now, there are generally two ways to get InterruptTime: read its last snapshot from KUSER_SHARED_DATA, or get the
  2151. "precise" time by adding snapshot and the fraction of TSC ticks (rdtsc) elapsed since the last snapshot. For more info,
  2152. see https://ntdoc.m417z.com/kuser_shared_data.
  2153. I don't know if PoFx uses snapshot or the precise InterruptTime for its logs, so let's find out possible error by
  2154. retrieving current clock timer period (how often the Clock Owner Processor updates InterruptTime/SystemTime/TickCount).
  2155.  
  2156. ✨✨✨
  2157. 2: kd> dd nt!KeTimeIncrement L1
  2158. fffff801`14efc468 0000270f
  2159. 2: kd> ? 0000270f
  2160. Evaluate expression: 9999 = 00000000`0000270f
  2161. 🌽 ^^ that's 9999 centums == 999.9 us == 0.9999 ms. This box uses APIC timer with frequency 0.3 MHz, so we may as well
  2162. round up current clock timer period to 1 ms.
  2163.  
  2164. ✨✨✨
  2165. 2: kd> !time
  2166. CURRENT TIME:
  2167. System: 01d768b3`64a7f260 (2021 Jun 24 04:42:52.788)
  2168. Interrupt: 0000054c`aea24ca6 (6 days, 17:51:30.552) 🌽⏰💎 omg, same as the last event above
  2169. Unbiased Interrupt: 000001ba`87a64e35 (2 days, 04:47:45.136)
  2170.  
  2171. 2: kd> ? 0000054caea24ca6-0000054cae7aee1f 🌽 time elapsed since event 48...
  2172. Evaluate expression: 2580103 = 00000000`00275e87
  2173. 2: kd> ? (0000054caea24ca6-0000054cae7aee1f)/(0n10*0n1000)
  2174. Evaluate expression: 258 = 00000000`00000102 🌽 ...is 258 ms
  2175. 2: kd> ? (0000054caea24ca6-0000054cae43feb3)/(0n10*0n1000)
  2176. Evaluate expression: 618 = 00000000`0000026a 🌽 time since event 46 is 618 ms
  2177.  
  2178.  
  2179. ✨✨✨
  2180. 🌽 So we just learned that current kuser.InterruptTime is ⏰ 0000054caea24ca6, which is the same time as when the
  2181. Power IRP completed for the disk device. I think that's a bit too close. Sure, event 48 says that device power state
  2182. changed to D0 258 ms prior, but still the Power IRP has completed only about right now. 💎 May be there are some edge
  2183. cases in power handling in the Intel iaStorAC.sys driver, which could have prevented disk read request during page-in.
  2184.  
  2185. 🌽 Let's take one step back and explore what system was doing - how come the disk device received "power on to D0"
  2186. request only 618 ms ago?
  2187. ✨✨✨✨✨✨✨✨
  2188. 2: kd> !powertriage 🌽 very nice command, but lot's of errors in output which I had to clean up here
  2189.  
  2190. System Capabilities :
  2191. Machine is not AOAC capable.
  2192.  
  2193. Power Capabilities:
  2194. PopCapabilities @ 0xfffff80114e23e60
  2195. Misc Supported Features: PwrButton SlpButton Lid S3 S4 S5 HiberFile VideoDim
  2196. Processor Features: Thermal
  2197. Disk Features:
  2198. Battery Features: BatteriesPresent
  2199. Battery 0 - Capacity: 0 Granularity: 0
  2200. Battery 1 - Capacity: 0 Granularity: 0
  2201. Battery 2 - Capacity: 0 Granularity: 0
  2202. Wake Caps
  2203. Ac OnLine Wake: Sx
  2204. Soft Lid Wake: Sx
  2205. RTC Wake: S4
  2206. Min Device Wake: Sx
  2207. Default Wake: Sx
  2208.  
  2209. Power Action:
  2210.  
  2211. PopAction :fffff80114e23240 🌽🔋 address of _POP_POWER_ACTION
  2212. Current System State..: Hibernate 🌽 that's weird, hibernate to hibernate?
  2213. Target System State...: Hibernate 🌽 let's assume one of these states was not updated yet
  2214. State.................: Set System State in progress(3)
  2215.  
  2216. Devices with allocated Power IRPs:
  2217. + PCI\VEN_8086&DEV_8A52&SUBSYS_1B2F1043&REV_07\3&11583659&1&10
  2218. 0xffffb005c826f360 pci D0 !podev FxDevice: !fxdevice !rcdrlogdump
  2219. Upper DO 0xffffb005c54d2720 ACPI !podev
  2220. Upper DO 0xffffb005df5cc030 igfxn SET_POWER_IRP !irp Related Threads SET_POWER_IRP !irp Related Threads !podev
  2221.  
  2222. + USB\VID_046D&PID_C52B\5&139e9f7f&0&3
  2223. 0xffffb005df82ed80 USBHUB3 D0 !podev SET_POWER_IRP !irp Related Threads !rcdrlogdump !wdfdriverinfo !wdflogdump
  2224. Upper DO 0xffffb005c54ce720 ACPI !podev
  2225. Upper DO 0xffffb005df556060 usbccgp !podev !rcdrlogdump
  2226.  
  2227. + USB\VID_046D&PID_C52B&MI_00\6&1458a8f4&0&0000
  2228. 0xffffb005df558880 usbccgp D0 !podev FxDevice: !fxdevice SET_POWER_IRP !irp Related Threads !rcdrlogdump
  2229. Upper DO 0xffffb005df759120 HidUsb !podev
  2230.  
  2231. + HID\VID_046D&PID_C52B&MI_00\7&3da2ba5&0&0000
  2232. 0xffffb005eb60e060 HidUsb D0 !podev WAIT_WAKE_IRP !irp Related Threads
  2233. Upper DO 0xffffb005eb614040 kbdhid !podev
  2234. Upper DO 0xffffb005e4db1060 kbdclass !podev
  2235.  
  2236. + USB\VID_046D&PID_C52B&MI_01\6&1458a8f4&0&0001
  2237. 0xffffb005df556830 usbccgp D0 !podev FxDevice: !fxdevice SET_POWER_IRP !irp Related Threads !rcdrlogdump
  2238. Upper DO 0xffffb005df74d120 HidUsb !podev
  2239.  
  2240. + HID\VID_046D&PID_C52B&MI_01&Col01\7&27b16967&0&0000
  2241. 0xffffb005df7044b0 HidUsb D0 !podev WAIT_WAKE_IRP !irp Related Threads
  2242. Upper DO 0xffffb005df738550 mouhid !podev
  2243. Upper DO 0xffffb005df73b8e0 mouclass !podev
  2244.  
  2245. + USB\VID_046D&PID_C52B&MI_02\6&1458a8f4&0&0002
  2246. 0xffffb005df82f060 usbccgp D0 !podev FxDevice: !fxdevice SET_POWER_IRP !irp Related Threads !rcdrlogdump
  2247. Upper DO 0xffffb005df753120 HidUsb !podev
  2248.  
  2249. + USB\VID_8087&PID_0AAA\5&139e9f7f&0&10
  2250. 0xffffb005df2a2ac0 USBHUB3 D0 !podev SET_POWER_IRP !irp Related Threads !rcdrlogdump !wdfdriverinfo !wdflogdump
  2251. Upper DO 0xffffb005df716290 ACPI !podev
  2252. Upper DO 0xffffb005df573b10 ibtusb !podev
  2253. Upper DO 0xffffb005df739b90 BTHUSB !podev
  2254.  
  2255. + PCI\VEN_10DE&DEV_1D16&SUBSYS_1B2F1043&REV_A1\4&38bf707b&0&00E4
  2256. 0xffffb005c829f120 pci D0 !podev FxDevice: !fxdevice !rcdrlogdump
  2257. Upper DO 0xffffb005c81e8e00 ACPI !podev
  2258. Upper DO 0xffffb005df94d030 nvlddmkm SET_POWER_IRP !irp Related Threads SET_POWER_IRP !irp Related Threads !podev
  2259.  
  2260. + {DD8E82AE-334B-49A2-AEAE-AEB0FD5C40DD}\DetectionVerification\5&55d82b7&0&0
  2261. 0xffffb005dfad8c80 IntcOED D3 !podev FxDevice: !fxdevice WAIT_WAKE_IRP !irp Related Threads
  2262. Upper DO 0xffffb005e81e24c0 WUDFRd !podev !rcdrlogdump !wdfdriverinfo !wdflogdump
  2263.  
  2264. + ACPI\INT3455\2&daba3ff&1
  2265. 0xffffb005c5107c60 ACPI D0 !podev FxDevice: !fxdevice SET_POWER_IRP !irp Related Threads
  2266. Upper DO 0xffffb005dfa56860 iaLPSS2_GPIO2_ICL SET_POWER_IRP !irp Related Threads !podev
  2267.  
  2268. Device Tree Info:
  2269. !devpowerstate
  2270. !devpowerstate Complete
  2271.  
  2272. Links:
  2273. !poaction 🌽🖱️ we'll click this
  2274. !cstriage
  2275. !pdctriage
  2276. !pdcclients
  2277. !fxdevice
  2278. !pnptriage
  2279.  
  2280. ✨✨✨
  2281. 2: kd> !poaction
  2282. PopAction: fffff80114e23240 🌽🔋 address of _POP_POWER_ACTION (same as before)
  2283. State..........: 3 - Set System State 🌽 okay, that one believable
  2284. Updates........: 0
  2285. Action.........: Sleep 🌽 confusing; see below for better info
  2286. Lightest State.: Hibernate
  2287. Flags..........: 80000024 OverrideApps|Critical
  2288. Irp minor......: SetPower
  2289. System State...: Hibernate
  2290. Hiber Context..: ffffb005e46dfd70
  2291.  
  2292. Allocated power irps (PopIrpList - fffff80114e239b0)
  2293. IRP: ffffb005efff4010 (set/S0), PDO: ffffb005c5107c60, NOTIFY: ffffb005c81e5c70
  2294. IRP: ffffb005df5f2520 (set/D0,), PDO: ffffb005c5107c60, CURRENT: ffffb005dfa56860
  2295. IRP: ffffb005ed63ca20 (set/S0), PDO: ffffb005c826f360, CURRENT: ffffb005df5cc030, NOTIFY: ffffb005c818ad40
  2296. IRP: ffffb005e41b2360 (set/D0,), PDO: ffffb005c826f360, CURRENT: ffffb005df5cc030
  2297. IRP: ffffb005f3ad5a20 (set/S0), PDO: ffffb005c829f120, CURRENT: ffffb005df94d030, NOTIFY: ffffb005c82715c0
  2298. IRP: ffffb005e41afa20 (set/D0,), PDO: ffffb005c829f120, CURRENT: ffffb005df94d030
  2299. IRP: ffffb005f28c7a20 (set/D0,), PDO: ffffb005df82ed80, CURRENT: ffffb005df82ed80
  2300. IRP: ffffb005f50819e0 (set/D0,), PDO: ffffb005df2a2ac0, CURRENT: ffffb005df2a2ac0
  2301. IRP: ffffb005eb720010 (set/D0,), PDO: ffffb005df558880, CURRENT: ffffb005df558880
  2302. IRP: ffffb005df717930 (set/D0,), PDO: ffffb005df82f060, CURRENT: ffffb005df82f060
  2303. IRP: ffffb005fc1db8a0 (wait-wake/S4), PDO: ffffb005eb60e060
  2304. IRP: ffffb005df705010 (set/D0,), PDO: ffffb005df556830, CURRENT: ffffb005df556830
  2305. IRP: ffffb005f68888a0 (wait-wake/S4), PDO: ffffb005df7044b0
  2306. IRP: ffffb005f02c0250 (wait-wake/S0), PDO: ffffb005dfad8c80
  2307.  
  2308. Irp worker threads (PopIrpThreadList - fffff80114e20960)
  2309. THREAD: ffffb005c510f040 (static)
  2310. THREAD: ffffb005c510e040 (static)
  2311. THREAD: ffffb005f3ae4500 (dynamic)
  2312. THREAD: ffffb005e41cf080 (dynamic), IRP: ffffb005e41afa20, DEVICE: ffffb005df94d030
  2313. THREAD: ffffb005f5478040 (dynamic)
  2314. THREAD: ffffb005f1f13040 (dynamic)
  2315. THREAD: ffffb005f0b74040 (dynamic)
  2316. THREAD: ffffb005eb30a040 (dynamic)
  2317. THREAD: ffffb005f6dd8040 (dynamic)
  2318. THREAD: ffffb005f3cbe040 (dynamic), IRP: ffffb005e41b2360, DEVICE: ffffb005df5cc030
  2319. THREAD: ffffb005f5013080 (dynamic)
  2320. THREAD: ffffb005e9e044c0 (dynamic)
  2321. THREAD: ffffb005f53a8040 (dynamic)
  2322. THREAD: ffffb005cc20b040 (dynamic)
  2323. THREAD: ffffb005eed0e040 (dynamic)
  2324.  
  2325. Broadcast in progress: TRUE
  2326. Is Directed DRIPS Transition: FALSE
  2327.  
  2328. Device State ffffb005c83ec0e0
  2329. Irp minor......: SetPower
  2330. System State...: Working
  2331. Worker thread..: ffffb005ebd76040
  2332. Status.........: 0
  2333. Waking.........: TRUE 🌽 the system is waking from Hibernation?
  2334. Cancelled......: FALSE
  2335. Ignore errors..: TRUE
  2336. Ignore not imp.: FALSE
  2337.  
  2338. Order: 🌽 lot's of info in this block, we'll ignore it for now
  2339. Level 4 (ffffb005c83ec240) 0/21 Paged, Root-Enum
  2340. ReadyS0:
  2341. ffffb005c51dcac0: 00000000 \Driver\BasicDisplay
  2342. ffffb005c51ddac0: 00000000 \Driver\Vid
  2343. ffffb005c51deac0: 00000000 \Driver\CompositeBus
  2344. ffffb005c51e0ac0: 00000000 \Driver\vdrvroot
  2345. ffffb005c5111ac0: 00000000 \Driver\umbus
  2346. ffffb005c50b8730: 00000000 \Driver\BasicRender
  2347. ffffb005c50b8a60: 00000000 \Driver\NdisVirtualBus
  2348. ffffb005c5141ac0: 00000000 \Driver\CAD
  2349. ffffb005c50a39f0: 00000000 \Driver\swenum
  2350. ffffb005e4f1a0b0: 00000000 \Driver\RasSstp \Device\NDMP4
  2351. ffffb005e4f1c710: 00000000 \Driver\RasAgileVpn \Device\NDMP5
  2352. ffffb005e4f1b3e0: 00000000 \Driver\Rasl2tp \Device\NDMP6
  2353. ffffb005ea0beb80: 00000000 \Driver\PptpMiniport \Device\NDMP7
  2354. ffffb005e4f1a3e0: 00000000 \Driver\RasPppoe \Device\NDMP8
  2355. ffffb005e4f1f3e0: 00000000 \Driver\NdisWan \Device\NDMP9
  2356. ffffb005ea0c7ac0: 00000000 \Driver\NdisWan \Device\NDMP10
  2357. ffffb005ea0bfac0: 00000000 \Driver\NdisWan \Device\NDMP11
  2358. ffffb005df272c50: 00000000 \Driver\volsnap
  2359. ffffb005df274c50: 00000000 \Driver\volsnap
  2360. ffffb005df28fc40: 00000000 \Driver\volsnap
  2361. ffffb005df26ac50: 00000000 \Driver\volsnap
  2362. Level 3 (ffffb005c83ec1f8) 76/84 Paged, PnP
  2363. WaitSleep:
  2364. ffffb005c81e0940: 00000000 \Driver\intelppm
  2365. ffffb005c81e0c70: 00000000 \Driver\intelppm
  2366. ffffb005c8206c70: 00000000 \Driver\intelppm
  2367. ffffb005c81e1940: 00000000 \Driver\intelppm
  2368. ffffb005c81e2940: 00000000 \Driver\intelppm
  2369. ffffb005c81e1c70: 00000000 \Driver\intelppm
  2370. ffffb005c81e2c70: 00000000 \Driver\intelppm
  2371. ffffb005c81e3940: 00000000 \Driver\intelppm
  2372. ffffb005c81e3c70: 00000000 \Driver\acpipagr
  2373. ffffb005c81e4940: 00000000 \Driver\acpitime
  2374. ffffb005c8271c70: 00000000 \Driver\TPM
  2375. ffffb005c8273c70: 00000000 \Driver\dptf_acpi
  2376. ffffb005c8274940: 00000000 \Driver\dptf_acpi
  2377. ffffb005c820dd40: 00000000 \Driver\USBXHCI \Device\USBFDO-0
  2378. ffffb005c8290d40: 00000000 \Driver\iaLPSS2_I2C_ICL \Device\0000004f
  2379. ffffb005c8294d40: 00000000 \Driver\iaLPSS2_I2C_ICL \Device\00000050
  2380. ffffb005c826fd40: 00000000 \Driver\MEIx64
  2381. ffffb005c828cd40: 00000000 \Driver\vwifibus
  2382. ffffb005c81e8b40: 00000000 \Driver\iaLPSS2_UART2_ICL \Device\00000051
  2383. ffffb005c818cb40: 00000000 \Driver\iaLPSS2_SPI_ICL \Device\00000052
  2384. ffffb005c8296b40: 00000000 \Driver\IntcAudioBus
  2385. ffffb005c81dccd0: 00000000 \Driver\partmgr
  2386. ffffb005c50be0b0: 00000000 \Driver\kbdclass \Device\KeyboardClass1
  2387. ffffb005c82a1170: 00000000 \Driver\dptf_acpi
  2388. ffffb005c82a14a0: 00000000 \Driver\dptf_acpi
  2389. ffffb005c82a17d0: 00000000 \Driver\dptf_acpi
  2390. ffffb005c82a1b00: 00000000 \Driver\dptf_acpi
  2391. ffffb005c820bd40: 00000000 \Driver\WudfRd
  2392. ffffb005cc33fd50: 00000000 \Driver\USBHUB3
  2393. ffffb005dfc24a80: 00000000 \Driver\SoftwareDevice \Device\00000059
  2394. ffffb005e4058ba0: 00000000 \Driver\mshidkmdf \Device\_HID00000003
  2395. ffffb005dfc24750: 00000000 \Driver\SoftwareDevice \Device\0000005a
  2396. ffffb005dfc24420: 00000000 \Driver\SoftwareDevice \Device\0000005b
  2397. ffffb005dfc240f0: 00000000 \Driver\SoftwareDevice \Device\0000005c
  2398. ffffb005dfc2fb90: 00000000 \Driver\SoftwareDevice \Device\0000005d
  2399. ffffb005dfc2f860: 00000000 \Driver\SoftwareDevice \Device\0000005e
  2400. ffffb005e3dee0f0: 00000000 \Driver\SoftwareDevice \Device\00000061
  2401. ffffb005ee1f8a80: 00000000 \Driver\vwifimp \Device\NDMP12
  2402. ffffb005e4f1f0b0: 00000000 \Driver\vwifimp \Device\NDMP3
  2403. ffffb005dfc2f200: 00000000 \Driver\IntcOED \Device\OE_DEVICE
  2404. ffffb005df826d20: 00000000 \Driver\usbccgp \Device\00000041
  2405. ffffb005df5afce0: 00000000 \Driver\BTHUSB
  2406. ffffb005df5b2d10: 00000000 \Driver\usbccgp \Device\0000004d
  2407. ffffb005dfc1dd50: 00000000 \Driver\mshidkmdf \Device\00000066
  2408. ffffb005e3dee420: 00000000 \Driver\mouclass \Device\PointerClass1
  2409. ffffb005dfc1d6f0: 00000000 \Driver\AsusPTPDrv
  2410. ffffb005dfc21cc0: 00000000 \Driver\MTConfig
  2411. ffffb005dfc1da20: 00000000 \Driver\ksthunk \Device\0000006d
  2412. ffffb005e4058540: 00000000 \Driver\ksthunk \Device\0000006b
  2413. ffffb005df54fc70: 00000000 \Driver\HidUsb \Device\_HID00000002
  2414. ffffb005df549c10: 00000000 \Driver\HidUsb \Device\_HID00000004
  2415. ffffb005e4260750: 00000000 \Driver\BthEnum \Device\BthEnum0
  2416. ffffb005e4260a80: 00000000 \Driver\BthEnum \Device\BthEnum1
  2417. ffffb005df54f940: 00000000 \Driver\HidUsb \Device\_HID00000001
  2418. ffffb005e42870f0: 00000000 \Driver\BthPan \Device\NDMP2
  2419. ffffb005e420e0f0: 00000000 \Driver\BthLEEnum
  2420. ffffb005e4235c20: 00000000 \Driver\SoftwareDevice \Device\0000006e
  2421. ffffb005e425c0f0: 00000000 \Driver\SoftwareDevice \Device\0000006f
  2422. ffffb005e425c420: 00000000 \Driver\SoftwareDevice \Device\00000070
  2423. ffffb005e425c750: 00000000 \Driver\SoftwareDevice \Device\00000071
  2424. ffffb005e425ca80: 00000000 \Driver\SoftwareDevice \Device\00000072
  2425. ffffb005e425e0f0: 00000000 \Driver\SoftwareDevice \Device\00000073
  2426. ffffb005e4f24750: 00000000 \Driver\SoftwareDevice \Device\0000007b
  2427. ffffb005e4f1a710: 00000000 \Driver\SoftwareDevice \Device\0000007c
  2428. ffffb005df5b3c20: 00000000 \Driver\HidUsb \Device\0000004b
  2429. ffffb005df5b35c0: 00000000 \Driver\HidUsb \Device\0000004a
  2430. ffffb005df5b60b0: 00000000 \Driver\HidUsb \Device\0000004c
  2431. ffffb005e4f1b710: 00000000 \Driver\kbdclass \Device\KeyboardClass2
  2432. ffffb005df70bc40: 00000000 \Driver\HidUsb \Device\00000047
  2433. ffffb005df72dc80: 00000000 \Driver\mouclass \Device\PointerClass0
  2434. ffffb005df714c40: 00000000 \Driver\HidUsb \Device\00000048
  2435. ffffb005df7130b0: 00000000 \Driver\HidUsb \Device\00000049
  2436. ffffb005dfa71990: 00000000 \Driver\ksthunk \Device\0000009b
  2437. ffffb005e4058870: 00000000 \Driver\WudfRd
  2438. ffffb005c81e02e0: 00000000 \Driver\CmBatt
  2439. ffffb005c81853e0: 00000000 \Driver\CmBatt
  2440. WaitS0:
  2441. ffffb005df99c0f0: 00000000 \Driver\SoftwareDevice \Device\00000054
  2442. ffffb005df9b3300: 00000000 \Driver\SoftwareDevice \Device\00000055
  2443. ffffb005e4f1ea40: 00000000 \Driver\monitor
  2444. ffffb005dfc2f530: 00000000 \Driver\WudfRd
  2445. ffffb005c828ed40: 00000000 \Driver\sdbus \Device\SdBus-HC0
  2446. Level 1 (ffffb005c83ec168) 21/21 Non-Paged, Root-Enum
  2447. WaitSleep:
  2448. ffffb005c51e1ac0: 00000000 \Driver\spaceport \Device\Spaceport
  2449. ffffb005c52e6ac0: 00000000 \Driver\kdnic
  2450. ffffb005e4f11710: 00000000 \Driver\SoftwareDevice \Device\00000079
  2451. ffffb005c50a36c0: 00000000 \Driver\mssmbios
  2452. ffffb005c50d60f0: 00000000 \Driver\rdpbus \Device\RdpBus
  2453. ffffb005e4f0b710: 00000000 \Driver\SoftwareDevice \Device\0000007d
  2454. ffffb005e4f1ca40: 00000000 \Driver\SoftwareDevice \Device\00000082
  2455. ffffb005e4f1aa40: 00000000 \Driver\SoftwareDevice \Device\0000007e
  2456. ffffb005e4f1f710: 00000000 \Driver\SoftwareDevice \Device\0000008c
  2457. ffffb005c510dac0: 00000000 \Driver\volmgr \Device\VolMgrControl
  2458. ffffb005e4f10a40: 00000000 \Driver\SoftwareDevice \Device\0000007f
  2459. ffffb005e4f240f0: 00000000 \Driver\SoftwareDevice \Device\00000081
  2460. ffffb005e9ed7ac0: 00000000 \Driver\SoftwareDevice \Device\0000008f
  2461. ffffb005e4f113e0: 00000000 \Driver\SoftwareDevice \Device\0000008d
  2462. ffffb005e4f11a40: 00000000 \Driver\SoftwareDevice \Device\00000090
  2463. ffffb005ea0c5ac0: 00000000 \Driver\SoftwareDevice \Device\00000091
  2464. ffffb005f9f26b10: 00000000 \Driver\SoftwareDevice \Device\0000009d
  2465. ffffb005df28ec40: 00000000 \Driver\volsnap
  2466. ffffb005df269c90: 00000000 \Driver\volsnap \Device\HarddiskVolumeShadowCopy2
  2467. ffffb005df258c90: 00000000 \Driver\volsnap \Device\HarddiskVolumeShadowCopy3
  2468. ffffb005efe04d60: 00000000 \Driver\volsnap \Device\HarddiskVolumeShadowCopy4
  2469. Level 0 (ffffb005c83ec120) 35/35 Non-Paged, PnP
  2470. WaitSleep:
  2471. ffffb005c51e2ac0: 00000000 \Driver\ACPI_HAL
  2472. ffffb005c553fd10: 00000000 \Driver\UEFI
  2473. ffffb005dfc5f0f0: 00000000 \Driver\UEFI \Device\00000058
  2474. ffffb005c50bf0b0: 00000000 \Driver\ACPI
  2475. ffffb005c81e4c70: 00000000 \Driver\ACPI \Device\0000001d
  2476. ffffb005c8185940: 00000000 \Driver\ACPI \Device\00000020
  2477. ffffb005c8270940: 00000000 \Driver\WmiAcpi
  2478. ffffb005c8270c70: 00000000 \Driver\intelpep
  2479. ffffb005c81e5940: 00000000 \Driver\ACPI \Device\0000001e
  2480. ffffb005c8272c70: 00000000 \Driver\ACPI \Device\00000027
  2481. ffffb005c8185c70: 00000000 \Driver\WmiAcpi
  2482. ffffb005c8272940: 00000000 \Driver\WmiAcpi
  2483. ffffb005c8274c70: 00000000 \Driver\ACPI \Device\0000002b
  2484. ffffb005c8206940: 00000000 \Driver\pci
  2485. ffffb005c8273940: 00000000 \Driver\ACPI \Device\00000028
  2486. ffffb005c8188d40: 00000000 \Driver\pci \Device\NTPNP_PCI0000
  2487. ffffb005c8271940: 00000000 \Driver\ACPI \Device\00000024
  2488. ffffb005c820fd40: 00000000 \Driver\pci \Device\NTPNP_PCI0004
  2489. ffffb005c81eed40: 00000000 \Driver\iaStorAC \Device\RaidPort0
  2490. ffffb005c8224b00: 00000000 \Driver\pci \Device\NTPNP_PCI0018
  2491. ffffb005c81dfb00: 00000000 \Driver\ACPI
  2492. ffffb005c8292b40: 00000000 \Driver\msisadrv
  2493. ffffb005c828ab00: 00000000 \Driver\ACPI \Device\0000002c
  2494. ffffb005c81ceb00: 00000000 \Driver\WmiAcpi
  2495. ffffb005c8183ac0: 00000000 \Driver\WmiAcpi
  2496. ffffb005c82745c0: 00000000 \Driver\ACPI \Device\0000002f
  2497. ffffb005c81e6480: 00000000 \Driver\ACPI \Device\00000030
  2498. ffffb005c82730b0: 00000000 \Driver\ACPI \Device\00000031
  2499. ffffb005c82733e0: 00000000 \Driver\ACPI \Device\00000032
  2500. ffffb005c82090b0: 00000000 \Driver\ACPI \Device\00000033
  2501. ffffb005c82093e0: 00000000 \Driver\ACPI \Device\00000034
  2502. ffffb005c50db0b0: 00000000 \Driver\partmgr
  2503. ffffb005c81850b0: 00000000 \Driver\ACPI \Device\00000037
  2504. ffffb005c8211d40: 00000000 \Driver\pci
  2505. ffffb005c818ed40: 00000000 \Driver\pci
  2506.  
  2507. 🌽 Some output above was a bit confusing, so let's examine power structures manually.
  2508. ✨✨✨
  2509. 2: kd> dt nt!_POP_POWER_ACTION fffff80114e23240 🌽🔋 address from above
  2510. +0x000 Updates : 0 ''
  2511. +0x001 State : 0x3 ''
  2512. +0x002 Shutdown : 0 ''
  2513. +0x004 Action : 2 ( PowerActionSleep )
  2514. +0x008 LightestState : 5 ( PowerSystemHibernate )
  2515. +0x00c Flags : 0x80000024
  2516. +0x010 Status : 0n0
  2517. +0x014 DeviceType : 6 ( PolicyImmediateDozeS4 )
  2518. +0x018 DeviceTypeFlags : 0
  2519. +0x01c IrpMinor : 0x2 ''
  2520. +0x01d Waking : 0x1 ''
  2521. +0x020 SystemState : 5 ( PowerSystemHibernate ) 🌽 okay, so system is "still" in Hibernate state?
  2522. +0x024 NextSystemState : 1 ( PowerSystemWorking ) 🌽 but the next state should be the Working state
  2523. +0x028 EffectiveSystemState : 5 ( PowerSystemHibernate )
  2524. +0x02c CurrentSystemState : 5 ( PowerSystemHibernate )
  2525. +0x030 ShutdownBugCode : (null)
  2526. +0x038 DevState : 0xffffb005`c83ec0e0 _POP_DEVICE_SYS_STATE 🌽🖱️1
  2527. +0x040 HiberContext : 0xffffb005`e46dfd70 _POP_HIBER_CONTEXT 🌽🖱️2
  2528. +0x048 WakeTime : 0x0000054c`ae3f2d59 🌽⌚ we can trust it: InterruptTime when system woke up
  2529. +0x050 SleepTime : 0x0000051e`3539a223 🌽💤 InterruptTime when system went to sleep (long ago)
  2530. +0x058 WakeFirstUnattendedTime : 0
  2531. +0x060 WakeAlarmSignaled : 3 ( PoConditionMaximum )
  2532. +0x068 WakeAlarm : [3] <anonymous-tag>
  2533. +0x0b0 WakeAlarmPaused : 0x1 ''
  2534. +0x0b8 WakeAlarmLastTime : 0x0000051d`fe45117d
  2535. +0x0c0 DozeDeferralStartTime : 0x00000504`d8f7d97d
  2536. +0x0c8 FilteredCapabilities : SYSTEM_POWER_CAPABILITIES
  2537. +0x118 WatchdogLock : 0
  2538. +0x120 WatchdogDpc : _KDPC
  2539. +0x160 WatchdogTimer : _KTIMER
  2540. +0x1a0 WatchdogInitialized : 0x1 ''
  2541. +0x1a4 WatchdogState : 2 ( PopPowerActionWatchdogStateResuming )
  2542. +0x1a8 WatchdogStartTime : 0x0000051e`3546dcd0
  2543. +0x1b0 ActionWorkerThread : 0xffffb005`ebd76040 _KTHREAD
  2544. +0x1b8 PromoteActionWorkerThread : (null)
  2545. +0x1c0 UnlockAfterSleepWorkerThread : (null)
  2546.  
  2547. ✨✨✨
  2548. 2: kd> !time
  2549. CURRENT TIME:
  2550. System: 01d768b3`64a7f260 (2021 Jun 24 04:42:52.788)
  2551. Interrupt: 0000054c`aea24ca6 (6 days, 17:51:30.552) 🌽⏰ current InterruptTime
  2552. Unbiased Interrupt: 000001ba`87a64e35 (2 days, 04:47:45.136)
  2553.  
  2554. 2: kd> ? 0000054c`aea24ca6-0x0000054c`ae3f2d59 🌽 delta: current time⏰ - wakeup time⌚
  2555. Evaluate expression: 6496077 = 00000000`00631f4d
  2556. 2: kd> ? (0000054c`aea24ca6-0x0000054c`ae3f2d59)/(0n10*0n1000) 🌽 yeah, same but in milliseconds
  2557. Evaluate expression: 649 = 00000000`00000289 🌽💎 system has woken up only 649 ms ago!
  2558.  
  2559. ✨✨✨
  2560. 🌽💎 That's interesting: system has woken up 649 ms ago, then 618 ms ago there was power up request for the device in
  2561. disk stack; 258 ms ago that device state change to D0, but only 0 ms ago (now) the power IRP has completed.
  2562.  
  2563. 🌽 Let's click some links in the just dumped structure to explore more stuff, just for reference.
  2564. ✨✨✨✨✨✨✨✨
  2565. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 ((ntkrnlmp!_POP_DEVICE_SYS_STATE *)0xffffb005c83ec0e0) 🌽🖱️1
  2566. ((ntkrnlmp!_POP_DEVICE_SYS_STATE *)0xffffb005c83ec0e0) : 0xffffb005c83ec0e0 [Type: _POP_DEVICE_SYS_STATE *]
  2567. [+0x000] IrpMinor : 0x2 [Type: unsigned char]
  2568. [+0x004] SystemState : PowerSystemWorking (1) [Type: _SYSTEM_POWER_STATE] 🌽 this one tells OS is working :)
  2569. [+0x008] SpinLock : 0x0 [Type: unsigned __int64]
  2570. [+0x010] Thread : 0xffffb005ebd76040 [Type: _KTHREAD *]
  2571. [+0x018] AbortEvent : 0x0 [Type: _KEVENT *]
  2572. [+0x020] ReadySemaphore : 0xffffe50a03816db8 [Type: _KSEMAPHORE *]
  2573. [+0x028] FinishedSemaphore : 0xffffe50a03816d98 [Type: _KSEMAPHORE *]
  2574. [+0x030] Order [Type: _PO_DEVICE_NOTIFY_ORDER]
  2575. [+0x1b0] Pending [Type: _LIST_ENTRY]
  2576. [+0x1c0] Status : 0 [Type: long]
  2577. [+0x1c8] FailedDevice : 0x0 [Type: _DEVICE_OBJECT *]
  2578. [+0x1d0] Waking : 0x1 [Type: unsigned char]
  2579. [+0x1d1] Cancelled : 0x0 [Type: unsigned char]
  2580. [+0x1d2] IgnoreErrors : 0x1 [Type: unsigned char]
  2581. [+0x1d3] IgnoreNotImplemented : 0x0 [Type: unsigned char]
  2582. [+0x1d4] TimeRefreshLockAcquired : 0x0 [Type: unsigned char]
  2583.  
  2584. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 ((ntkrnlmp!_POP_HIBER_CONTEXT *)0xffffb005e46dfd70) 🌽🖱️2
  2585. ((ntkrnlmp!_POP_HIBER_CONTEXT *)0xffffb005e46dfd70) : 0xffffb005e46dfd70 [Type: _POP_HIBER_CONTEXT *]
  2586. [+0x000] Reset : 0x0 [Type: unsigned char]
  2587. [+0x001] HiberFlags : 0x1 [Type: unsigned char]
  2588. [+0x002] WroteHiberFile : 0x1 [Type: unsigned char]
  2589. [+0x003] KernelPhaseVerificationActive : 0x0 [Type: unsigned char]
  2590. [+0x004] InitializationFinished : 0x1 [Type: unsigned char]
  2591. [+0x008] NextTableLockHeld : 0 [Type: long]
  2592. [+0x00c] BootPhaseFinishedBarrier : 0 [Type: long]
  2593. [+0x010] KernelResumeFinishedBarrier : 8 [Type: long]
  2594. [+0x014] HvCaptureReadyBarrier : 0 [Type: long]
  2595. [+0x018] HvCaptureCompletedBarrier : 0 [Type: long]
  2596. [+0x01c] MapFrozen : 0x1 [Type: unsigned char]
  2597. [+0x020] DiscardMap [Type: _RTL_BITMAP]
  2598. [+0x020] KernelPhaseMap [Type: _RTL_BITMAP]
  2599. [+0x030] BootPhaseMap [Type: _RTL_BITMAP]
  2600. [+0x040] ClonedRanges [Type: _LIST_ENTRY]
  2601. [+0x050] ClonedRangeCount : 0x5 [Type: unsigned long]
  2602. [+0x058] ClonedPageCount : 0x6 [Type: unsigned __int64]
  2603. [+0x060] CurrentMap : 0xffffb005e46dfda0 [Type: _RTL_BITMAP *]
  2604. [+0x068] NextCloneRange : 0xffffb005e46dfdb0 [Type: _LIST_ENTRY *]
  2605. [+0x070] NextPreserve : 0x18619e [Type: unsigned __int64]
  2606. [+0x078] LoaderMdl : 0xffffb0060009b000 [Type: _MDL *]
  2607. [+0x080] AllocatedMdl : 0xffffb005fff18c10 [Type: _MDL *]
  2608. [+0x088] PagesOut : 0xa1f0 [Type: unsigned __int64]
  2609. [+0x090] IoPages : 0xffffd3011f748000 [Type: void *]
  2610. [+0x098] IoPagesCount : 0x29 [Type: unsigned long]
  2611. [+0x0a0] CurrentMcb : 0xffffe50a039e3090 [Type: void *]
  2612. [+0x0a8] DumpStack : 0xffffb005eca4b010 [Type: _DUMP_STACK_CONTEXT *]
  2613. [+0x0b0] WakeState : 0xfffff80114eecb90 [Type: _KPROCESSOR_STATE *]
  2614. [+0x0b8] IoProgress : 0xa [Type: unsigned long]
  2615. [+0x0bc] Status : 1073742484 [Type: long]
  2616. [+0x0c0] GraphicsProc : 0x7 [Type: unsigned long]
  2617. [+0x0c8] MemoryImage : 0xffff9c894a010000 [Type: PO_MEMORY_IMAGE *] 🌽🖱️3
  2618. [+0x0d0] PerformanceStats : 0x0 [Type: unsigned long *]
  2619. [+0x0d8] BootLoaderLogMdl : 0xffff9c894a14a000 [Type: _MDL *]
  2620. [+0x0e0] FirmwareRuntimeInformationMdl : 0xffff9c894a14a1a0 [Type: _MDL *]
  2621. [+0x0e8] FirmwareRuntimeInformationVa : 0xffff9c894a029000 [Type: void *]
  2622. [+0x0f0] ResumeContext : 0xffffb005feb97000 [Type: void *]
  2623. [+0x0f8] ResumeContextPages : 0x4 [Type: unsigned long]
  2624. [+0x0fc] SecurePages : 0x0 [Type: unsigned long]
  2625. [+0x100] ProcessorCount : 0x8 [Type: unsigned long]
  2626. [+0x108] ProcessorContext : 0xffffb00600800000 [Type: _POP_PER_PROCESSOR_CONTEXT *]
  2627. [+0x110] ProdConsBuffer : 0xffffb00602c1b000 : "g???" [Type: char *]
  2628. [+0x118] ProdConsSize : 0x190000 [Type: unsigned long]
  2629. [+0x11c] MaxDataPages : 0x10 [Type: unsigned long]
  2630. [+0x120] ExtraBuffer : 0xffffb00600800000 [Type: void *]
  2631. [+0x128] ExtraBufferSize : 0x26ab000 [Type: unsigned __int64]
  2632. [+0x130] ExtraMapVa : 0xffffd3011f66c000 [Type: void *]
  2633. [+0x138] BitlockerKeyPFN : 0x0 [Type: unsigned __int64] 🌽🔑 😽
  2634. [+0x140] IoInfo [Type: _POP_IO_INFO]
  2635. [+0x1b0] IoChecksums : 0xffffb006012cf000 : "" [Type: wchar_t *]
  2636. [+0x1b8] IoChecksumsSize : 0x194bcd0 [Type: unsigned __int64]
  2637. [+0x1c0] HardwareConfigurationSignature : 0xaacd9f3 [Type: unsigned long]
  2638. [+0x1c4] IumEnabled : 0x0 [Type: unsigned char]
  2639. [+0x1c5] SecureBoot : 0x1 [Type: unsigned char]
  2640.  
  2641. 2: kd> dx -id 0,0,ffffb005e91bc040 -r1 ((ntkrnlmp!PO_MEMORY_IMAGE *)0xffff9c894a010000) 🌽🖱️3
  2642. ((ntkrnlmp!PO_MEMORY_IMAGE *)0xffff9c894a010000) : 0xffff9c894a010000 [Type: PO_MEMORY_IMAGE *]
  2643. [+0x000] Signature : 0x52424948 [Type: unsigned long] 🌽 'HIBR'
  2644. [+0x004] ImageType : 0x9 [Type: unsigned long]
  2645. [+0x008] CheckSum : 0x285d [Type: unsigned long]
  2646. [+0x00c] LengthSelf : 0x3e0 [Type: unsigned long]
  2647. [+0x010] PageSelf : 0x2c55b8 [Type: unsigned __int64]
  2648. [+0x018] PageSize : 0x1000 [Type: unsigned long]
  2649. [+0x020] SystemTime : {132689634122483680} [Type: _LARGE_INTEGER]
  2650. [+0x028] InterruptTime : 0x51e3539a223 [Type: unsigned __int64] 🌽💤 InterruptTime when went to sleep
  2651. [+0x030] FeatureFlags : 0x1fa4efbd9bbfff [Type: unsigned __int64]
  2652. [+0x038] HiberFlags : 0x1 [Type: unsigned char]
  2653. [+0x039] HiberSimulateFlags : 0xca [Type: unsigned char]
  2654. [+0x03a] spare [Type: unsigned char [2]]
  2655. [+0x03c] NoHiberPtes : 0xa [Type: unsigned long]
  2656. [+0x040] HiberVa : 0xffffd3011e7e0000 [Type: unsigned __int64]
  2657. [+0x048] NoFreePages : 0xa1b1 [Type: unsigned long]
  2658. [+0x04c] FreeMapCheck : 0xe249 [Type: unsigned long]
  2659. [+0x050] WakeCheck : 0xc641 [Type: unsigned long]
  2660. [+0x058] NumPagesForLoader : 0x214e6 [Type: unsigned __int64]
  2661. [+0x060] FirstSecureRestorePage : 0x0 [Type: unsigned __int64]
  2662. [+0x068] FirstBootRestorePage : 0x2f [Type: unsigned __int64]
  2663. [+0x070] FirstKernelRestorePage : 0x64cf [Type: unsigned __int64]
  2664. [+0x078] FirstChecksumRestorePage : 0x1555df [Type: unsigned __int64]
  2665. [+0x080] NoChecksumEntries : 0xaaaeb6 [Type: unsigned __int64]
  2666. [+0x088] PerfInfo [Type: _PO_HIBER_PERF]
  2667. [+0x280] FirmwareRuntimeInformationPages : 0x1 [Type: unsigned long]
  2668. [+0x288] FirmwareRuntimeInformation [Type: unsigned __int64 [1]]
  2669. [+0x290] SpareUlong : 0x0 [Type: unsigned long]
  2670. [+0x294] NoBootLoaderLogPages : 0x18 [Type: unsigned long]
  2671. [+0x298] BootLoaderLogPages [Type: unsigned __int64 [24]]
  2672. [+0x358] NotUsed : 0x0 [Type: unsigned long]
  2673. [+0x35c] ResumeContextCheck : 0x4add [Type: unsigned long]
  2674. [+0x360] ResumeContextPages : 0x4 [Type: unsigned long]
  2675. [+0x364] Hiberboot : 0x0 [Type: unsigned char]
  2676. [+0x365] SecureLaunched : 0x0 [Type: unsigned char]
  2677. [+0x366] SecureBoot : 0x1 [Type: unsigned char]
  2678. [+0x368] HvPageTableRoot : 0x0 [Type: unsigned __int64]
  2679. [+0x370] HvEntryPoint : 0x0 [Type: unsigned __int64]
  2680. [+0x378] HvReservedTransitionAddress : 0x0 [Type: unsigned __int64]
  2681. [+0x380] HvReservedTransitionAddressSize : 0x0 [Type: unsigned __int64]
  2682. [+0x388] BootFlags : 0x0 [Type: unsigned __int64]
  2683. [+0x390] RestoreProcessorStateRoutine : 0x0 [Type: unsigned __int64]
  2684. [+0x398] HighestPhysicalPage : 0x47ffff [Type: unsigned __int64]
  2685. [+0x3a0] BitlockerKeyPfns [Type: unsigned __int64 [4]] 🌽🔑 😽 (all four PFNs are zero)
  2686. [+0x3c0] HardwareSignature : 0xaacd9f3 [Type: unsigned long]
  2687. [+0x3c8] SMBiosTablePhysicalAddress : {1977802752} [Type: _LARGE_INTEGER]
  2688. [+0x3d0] SMBiosTableLength : 0xc12 [Type: unsigned long]
  2689. [+0x3d4] SMBiosMajorVersion : 0x3 [Type: unsigned char]
  2690. [+0x3d5] SMBiosMinorVersion : 0x2 [Type: unsigned char]
  2691. [+0x3d6] HiberResumeXhciHandoffSkip : 0x1 [Type: unsigned char]
  2692. [+0x3d7] InitializeUSBCore : 0x0 [Type: unsigned char]
  2693. [+0x3d8] ValidUSBCoreId : 0x0 [Type: unsigned char]
  2694. [+0x3d9] USBCoreId : 0x0 [Type: unsigned char]
  2695. [+0x3da] SkipMemoryMapValidation : 0x0 [Type: unsigned char]
  2696.  
  2697.  
  2698. 🌽 Okay, we're done with this section. What else can we check? Let's check callstacks of runningthreads.
  2699. ✨✨✨✨✨✨✨✨
  2700. 2: kd> !running -t
  2701.  
  2702. System Processors: (00000000000000ff) 🌽 there are 8 cpus, and 8 threads are running
  2703. Idle Processors: (0000000000000000)
  2704.  
  2705. 🌽 cpu0: nothing sus
  2706. Prcbs Current (pri) Next (pri) Idle
  2707. 0 fffff8010f533180 ffffb005e8039240 (16) fffff80114f27a00 ................
  2708. # Child-SP RetAddr Call Site
  2709. 00 fffff801`1946de20 fffff801`34e97898 IntcAudioBus+0xb16c
  2710. 01 fffff801`1946de60 fffff801`16ab3ca5 IntcAudioBus+0x27898
  2711. 02 fffff801`1946df00 fffff801`1447e795 Wdf01000!FxInterrupt::_InterruptThunk+0x75 [minkernel\wdf\framework\shared\irphandlers\pnp\km\interruptobjectkm.cpp @ 369]
  2712. 03 fffff801`1946df40 fffff801`145f844f nt!KiCallInterruptServiceRoutine+0xa5
  2713. 04 fffff801`1946df90 fffff801`145f8717 nt!KiInterruptSubDispatch+0x11f
  2714. 05 ffffe509`f8436bb0 fffff801`3500c7ee nt!KiInterruptDispatch+0x37
  2715. 06 ffffe509`f8436d40 fffff801`3500ea46 nvlddmkm+0xcc7ee
  2716. 07 ffffe509`f8436de0 fffff801`350d446c nvlddmkm+0xcea46
  2717. 08 ffffe509`f8436e10 fffff801`350acc9e nvlddmkm+0x19446c
  2718. 09 ffffe509`f8436e40 fffff801`35274d15 nvlddmkm+0x16cc9e
  2719. 0a ffffe509`f8436e70 fffff801`350e79f1 nvlddmkm+0x334d15
  2720. 0b ffffe509`f8436ed0 fffff801`350acbf2 nvlddmkm+0x1a79f1
  2721. 0c ffffe509`f8436f00 fffff801`350aae20 nvlddmkm+0x16cbf2
  2722. 0d ffffe509`f8436fc0 fffff801`350ab05e nvlddmkm+0x16ae20
  2723. 0e ffffe509`f84371c0 fffff801`350ab0fe nvlddmkm+0x16b05e
  2724. 0f ffffe509`f8437220 fffff801`34ff708d nvlddmkm+0x16b0fe
  2725. 10 ffffe509`f84374c0 fffff801`34ff6fff nvlddmkm+0xb708d
  2726. 11 ffffe509`f8437570 fffff801`35753d59 nvlddmkm+0xb6fff
  2727. 12 ffffe509`f84375b0 fffff801`35753fb1 nvlddmkm+0x813d59
  2728. 13 ffffe509`f8437640 fffff801`34f46608 nvlddmkm+0x813fb1
  2729. 14 ffffe509`f84376a0 fffff801`35c70f08 nvlddmkm+0x6608
  2730. 15 ffffe509`f84376f0 fffff801`35b9a0e2 nvlddmkm+0xd30f08
  2731. 16 ffffe509`f8437810 fffff801`2d97212e nvlddmkm+0xc5a0e2
  2732. 17 ffffe509`f84378f0 fffff801`2d82d849 dxgkrnl!ADAPTER_RENDER::DdiUnmapCpuHostAperture+0xba
  2733. 18 ffffe509`f84379a0 fffff801`2ee896f5 dxgkrnl!ADAPTER_RENDER_DdiUnmapCpuHostAperture+0x9
  2734. 19 ffffe509`f84379d0 fffff801`2eefe2c8 dxgmms2!VIDMM_CPU_HOST_APERTURE::UnmapRange+0xad
  2735. 1a ffffe509`f8437a20 fffff801`2eed18af dxgmms2!VIDMM_PAGE_TABLE_BASE::FreeCpuVisibleAddress+0x2a488
  2736. 1b ffffe509`f8437a50 fffff801`2eebfb12 dxgmms2!VIDMM_PAGE_TABLE_BASE::FreeDriverUpdateAddress+0x37
  2737. 1c ffffe509`f8437a80 fffff801`2eefe578 dxgmms2!VIDMM_PAGE_TABLE::UpdatePageTableInvalidate+0x10e
  2738. 1d ffffe509`f8437b50 fffff801`2eed21ad dxgmms2!VIDMM_PAGE_TABLE::CommitVirtualAddressRange+0x2a6a0
  2739. 1e ffffe509`f8437bf0 fffff801`2eefd3bc dxgmms2!VIDMM_PAGE_DIRECTORY::CommitVirtualAddressRange+0x77d
  2740. 1f ffffe509`f8437d60 fffff801`2eed146a dxgmms2!VIDMM_PAGE_DIRECTORY::CommitVirtualAddressRange+0x2b98c
  2741. 20 ffffe509`f8437ed0 fffff801`2ef0e18c dxgmms2!CVirtualAddressAllocator::CommitVirtualAddressRange+0x2f6
  2742. 21 ffffe509`f8438050 fffff801`2eefefe4 dxgmms2!VIDMM_GLOBAL::InitPagingProcessVaSpace+0x187cc
  2743. 22 ffffe509`f84381c0 fffff801`2eeecb52 dxgmms2!VIDMM_GLOBAL::ProcessSystemCommand+0x2a3b0
  2744. 23 ffffe509`f8438340 fffff801`2eef6739 dxgmms2!VIDMM_WORKER_THREAD::Run+0x1462
  2745. 24 ffffe509`f8438520 fffff801`144f53b5 dxgmms2!VidMmWorkerThreadProc+0x9
  2746. 25 ffffe509`f8438550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  2747. 26 ffffe509`f84385a0 00000000`00000000 nt!KiStartSystemThread+0x28
  2748.  
  2749. 🌽 cpu1: nothing sus
  2750. 1 ffffd30119a5c180 ffffb005f534d080 (10) ffffd30119a67240 ................
  2751. # Child-SP RetAddr Call Site
  2752. 00 ffffe50a`027697c0 fffff801`144730e2 nt!ExAcquireSpinLockExclusiveAtDpcLevel+0x2
  2753. 01 ffffe50a`027697d0 fffff801`1440aed9 nt!KiAbEntryRemoveFromTree+0xb2
  2754. 02 ffffe50a`02769860 fffff801`145000ef nt!KeAbPostRelease+0x1e9
  2755. 03 ffffe50a`027698c0 fffff801`147fb4f2 nt!MiZeroSectionObjectPointer+0x33
  2756. 04 ffffe50a`027698f0 fffff801`147faa64 nt!MiCreateImageOrDataSection+0x3b2
  2757. 05 ffffe50a`027699e0 fffff801`14817b32 nt!MiCreateSection+0xf4
  2758. 06 ffffe50a`02769b60 fffff801`14814ca9 nt!MmCreateSpecialImageSection+0xc6
  2759. 07 ffffe50a`02769c10 fffff801`146085b5 nt!NtCreateUserProcess+0x709
  2760. 08 ffffe50a`0276a3d0 00007ffb`b7b6e714 nt!KiSystemServiceCopyEnd+0x25
  2761. 09 0000005f`a977e218 00000000`00000000 0x00007ffb`b7b6e714
  2762.  
  2763. 🌽 cpu2: that's our current CPU, we've explored this BSOD'ed thread in details already
  2764. 2 ffffd30119b17180 ffffb005df2c8480 (19) ffffd30119b22240 ................
  2765. # Child-SP RetAddr Call Site
  2766. 00 ffffe509`f8198568 fffff801`14794dee nt!KeBugCheckEx
  2767. 01 ffffe509`f8198570 fffff801`14610023 nt!SMKM_STORE<SM_TRAITS>::SmStUnhandledExceptionFilter+0x7e
  2768. 02 ffffe509`f81985c0 fffff801`145cca4f nt!`SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue'::`1'::filt$0+0x22
  2769. 03 ffffe509`f81985f0 fffff801`145ffadf nt!_C_specific_handler+0x9f
  2770. 04 ffffe509`f8198660 fffff801`14487547 nt!RtlpExecuteHandlerForException+0xf
  2771. 05 ffffe509`f8198690 fffff801`14486136 nt!RtlDispatchException+0x297
  2772. 06 ffffe509`f8198db0 fffff801`14608cac nt!KiDispatchException+0x186
  2773. 07 ffffe509`f8199470 fffff801`14604e43 nt!KiExceptionDispatch+0x12c
  2774. 08 ffffe509`f8199650 fffff801`144d10d0 nt!KiPageFault+0x443
  2775. 09 ffffe509`f81997e8 fffff801`144a6bf0 nt!RtlDecompressBufferXpressLz+0x50
  2776. 0a ffffe509`f8199800 fffff801`144a6938 nt!RtlDecompressBufferEx+0x60
  2777. 0b ffffe509`f8199850 fffff801`144a67c5 nt!ST_STORE<SM_TRAITS>::StDmSinglePageCopy+0x150
  2778. 0c ffffe509`f8199910 fffff801`144a5ffc nt!ST_STORE<SM_TRAITS>::StDmSinglePageTransfer+0xa5
  2779. 0d ffffe509`f8199960 fffff801`144a5e2c nt!ST_STORE<SM_TRAITS>::StDmpSinglePageRetrieve+0x180
  2780. 0e ffffe509`f8199a00 fffff801`144a5c79 nt!ST_STORE<SM_TRAITS>::StDmPageRetrieve+0xc8
  2781. 0f ffffe509`f8199ab0 fffff801`144a5b31 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadIssue+0x85
  2782. 10 ffffe509`f8199b30 fffff801`14498bc8 nt!SMKM_STORE<SM_TRAITS>::SmStDirectReadCallout+0x21
  2783. 11 ffffe509`f8199b60 fffff801`144a368f nt!KeExpandKernelStackAndCalloutInternal+0x78
  2784. 12 ffffe509`f8199bd0 fffff801`14559934 nt!SMKM_STORE<SM_TRAITS>::SmStDirectRead+0xc7
  2785. 13 ffffe509`f8199ca0 fffff801`14559368 nt!SMKM_STORE<SM_TRAITS>::SmStWorkItemQueue+0x1ac
  2786. 14 ffffe509`f8199cf0 fffff801`144a4117 nt!SMKM_STORE_MGR<SM_TRAITS>::SmIoCtxQueueWork+0xc0
  2787. 15 ffffe509`f8199d80 fffff801`144fa96b nt!SMKM_STORE_MGR<SM_TRAITS>::SmPageRead+0x167
  2788. 16 ffffe509`f8199df0 fffff801`1445e0a0 nt!SmPageRead+0x33
  2789. 17 ffffe509`f8199e40 fffff801`1445bb4d nt!MiIssueHardFaultIo+0x10c
  2790. 18 ffffe509`f8199e90 fffff801`14528278 nt!MiIssueHardFault+0x29d
  2791. 19 ffffe509`f8199f50 fffff801`1442a4d2 nt!MmAccessFault+0x468
  2792. 1a ffffe509`f819a0f0 fffff801`14460836 nt!MiDeleteSubsectionPages+0x172
  2793. 1b ffffe509`f819a1b0 fffff801`14808d56 nt!MiDeleteSegmentPages+0x56
  2794. 1c ffffe509`f819a210 fffff801`14584a89 nt!MiSegmentDelete+0x92
  2795. 1d ffffe509`f819a260 fffff801`145ba179 nt!MiProcessDereferenceList+0xc1
  2796. 1e ffffe509`f819a320 fffff801`144f53b5 nt!MiDereferenceSegmentThread+0x129
  2797. 1f ffffe509`f819a550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  2798. 20 ffffe509`f819a5a0 00000000`00000000 nt!KiStartSystemThread+0x28
  2799.  
  2800. 🌽 cpu3: nothing sus
  2801. 3 ffffd30119bd2180 ffffb005f32b7080 ( 9) ffffd30119bdd240 ................
  2802. # Child-SP RetAddr Call Site
  2803. 00 ffffe509`fd17a500 fffff801`167b5b37 FLTMGR!FltpPerformPreCallbacksWorker+0x584
  2804. 01 ffffe509`fd17a620 fffff801`167b4b46 FLTMGR!FltpPassThroughInternal+0xc7
  2805. 02 ffffe509`fd17a670 fffff801`167b48bb FLTMGR!FltpPassThrough+0x1d6
  2806. 03 ffffe509`fd17a710 fffff801`145185b5 FLTMGR!FltpDispatch+0x8b
  2807. 04 ffffe509`fd17a770 fffff801`14428073 nt!IofCallDriver+0x55
  2808. 05 ffffe509`fd17a7b0 fffff801`144dbff0 nt!IoSynchronousPageWriteEx+0x13b
  2809. 06 ffffe509`fd17a7f0 fffff801`14535852 nt!MiIssueSynchronousFlush+0x70
  2810. 07 ffffe509`fd17a870 fffff801`1454b199 nt!MiFlushSectionInternal+0x862
  2811. 08 ffffe509`fd17ab40 fffff801`1468eb7a nt!MmFlushSection+0x155
  2812. 09 ffffe509`fd17abf0 fffff801`144bec7f nt!CcFlushCachePriv+0x1471aa
  2813. 0a ffffe509`fd17ad30 fffff801`181ef81d nt!CcCoherencyFlushAndPurgeCache+0x6f
  2814. 0b ffffe509`fd17ad80 fffff801`181ef6db Ntfs!NtfsCoherencyFlushAndPurgeCache+0x55
  2815. 0c ffffe509`fd17adc0 fffff801`181ef58d Ntfs!NtfsFlushUserStream+0xa3
  2816. 0d ffffe509`fd17ae50 fffff801`181eff72 Ntfs!NtfsPerformOptimisticFlush+0xa1
  2817. 0e ffffe509`fd17aea0 fffff801`181efd19 Ntfs!NtfsCommonFlushBuffers+0x1d6
  2818. 0f ffffe509`fd17afc0 fffff801`14498bc8 Ntfs!NtfsCommonFlushBuffersCallout+0x19
  2819. 10 ffffe509`fd17aff0 fffff801`14498b3d nt!KeExpandKernelStackAndCalloutInternal+0x78
  2820. 11 ffffe509`fd17b060 fffff801`181c7f6b nt!KeExpandKernelStackAndCalloutEx+0x1d
  2821. 12 ffffe509`fd17b0a0 fffff801`181c7e95 Ntfs!NtfsCommonFlushBuffersOnNewStack+0x67
  2822. 13 ffffe509`fd17b110 fffff801`145185b5 Ntfs!NtfsFsdFlushBuffers+0xe5
  2823. 14 ffffe509`fd17b180 fffff801`167b6ccf nt!IofCallDriver+0x55
  2824. 15 ffffe509`fd17b1c0 fffff801`167b48d3 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x28f
  2825. 16 ffffe509`fd17b230 fffff801`145185b5 FLTMGR!FltpDispatch+0xa3
  2826. 17 ffffe509`fd17b290 fffff801`148e2748 nt!IofCallDriver+0x55
  2827. 18 ffffe509`fd17b2d0 fffff801`14898f19 nt!IopSynchronousServiceTail+0x1a8
  2828. 19 ffffe509`fd17b370 fffff801`14898d06 nt!NtFlushBuffersFileEx+0x1f9
  2829. 1a ffffe509`fd17b400 fffff801`146085b5 nt!NtFlushBuffersFile+0x16
  2830. 1b ffffe509`fd17b440 00007ffb`b7b6d784 nt!KiSystemServiceCopyEnd+0x25
  2831. 1c 0000002e`abe7eb08 00000000`00000000 0x00007ffb`b7b6d784
  2832.  
  2833. 🌽 cpu4: nothing sus (graphics adapter is waking up); we'll 🍗 recheck it in a bit anyway
  2834. 4 ffffd30119c8e180 ffffb005f3cbe040 (13) ffffd30119c99240 ................
  2835. # Child-SP RetAddr Call Site
  2836. 00 ffffe50a`0320fef0 fffff801`310a4cda igdkmdn64+0x315447
  2837. 01 ffffe50a`0320ff00 fffff801`310a0765 igdkmdn64+0x314cda
  2838. 02 ffffe50a`0320ffe0 fffff801`310652ac igdkmdn64+0x310765
  2839. 03 ffffe50a`03210050 fffff801`30dd6a93 igdkmdn64+0x2d52ac
  2840. 04 ffffe50a`03210080 fffff801`30dd9db9 igdkmdn64+0x46a93
  2841. 05 ffffe50a`032100f0 fffff801`30dfb250 igdkmdn64+0x49db9
  2842. 06 ffffe50a`03210120 fffff801`30dfa268 igdkmdn64+0x6b250
  2843. 07 ffffe50a`032101f0 fffff801`30de661c igdkmdn64+0x6a268
  2844. 08 ffffe50a`032102a0 fffff801`30f45618 igdkmdn64+0x5661c
  2845. 09 ffffe50a`032102d0 fffff801`30f453a8 igdkmdn64+0x1b5618
  2846. 0a ffffe50a`03210370 fffff801`30f3cc37 igdkmdn64+0x1b53a8
  2847. 0b ffffe50a`03210620 fffff801`30f3844f igdkmdn64+0x1acc37
  2848. 0c ffffe50a`03210650 fffff801`30f312c8 igdkmdn64+0x1a844f
  2849. 0d ffffe50a`032106a0 fffff801`30f0a7f8 igdkmdn64+0x1a12c8
  2850. 0e ffffe50a`032106d0 fffff801`30ee0787 igdkmdn64+0x17a7f8
  2851. 0f ffffe50a`03210720 fffff801`30ef7578 igdkmdn64+0x150787
  2852. 10 ffffe50a`03210960 fffff801`30dcaf1c igdkmdn64+0x167578
  2853. 11 ffffe50a`03210e10 fffff801`30da0e60 igdkmdn64+0x3af1c
  2854. 12 ffffe50a`03210e40 fffff801`30da1172 igdkmdn64+0x10e60
  2855. 13 ffffe50a`03210e70 fffff801`30d99221 igdkmdn64+0x11172
  2856. 14 ffffe50a`032110c0 fffff801`2d973b6e igdkmdn64+0x9221
  2857. 15 ffffe50a`032111a0 fffff801`2d9730bf dxgkrnl!DpiDxgkDdiSetPowerState+0x5a
  2858. 16 ffffe50a`03211200 fffff801`2d973fee dxgkrnl!DpiFdoSetAdapterPowerState+0x15f
  2859. 17 ffffe50a`032112c0 fffff801`2d973441 dxgkrnl!DpiFdoHandleDevicePower+0x2ee
  2860. 18 ffffe50a`03211360 fffff801`2d974991 dxgkrnl!DpiFdoDispatchPower+0x21
  2861. 19 ffffe50a`03211390 fffff801`1459ce49 dxgkrnl!DpiDispatchPower+0xe1
  2862. 1a ffffe50a`032114b0 fffff801`144f53b5 nt!PopIrpWorker+0x1d9
  2863. 1b ffffe50a`03211550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  2864. 1c ffffe50a`032115a0 00000000`00000000 nt!KiStartSystemThread+0x28
  2865.  
  2866. 🌽 cpu5: 💣💣💣 ntfs read error?
  2867. 5 ffffd30119d48180 ffffb005f7fb8080 ( 9) ffffd30119d53240 ................
  2868. # Child-SP RetAddr Call Site
  2869. 00 ffffe509`f7a772c8 fffff801`18f9b252 CLASSPNP!ClasspGetDeviceIdData+0x5
  2870. 01 ffffe509`f7a772d0 fffff801`18f9b04a CLASSPNP!ClasspWriteScsiSrbErrorResultEvent+0x1f2
  2871. 02 ffffe509`f7a77400 fffff801`18f9614f CLASSPNP!ClasspWriteSrbErrorResultEvent+0x22
  2872. 03 ffffe509`f7a77440 fffff801`18f954eb CLASSPNP!ClasspWriteIOResultEvent+0x13f
  2873. 04 ffffe509`f7a77520 fffff801`1450dede CLASSPNP!TransferPktComplete+0x28b
  2874. 05 ffffe509`f7a77730 fffff801`1450dda7 nt!IopfCompleteRequest+0x11e
  2875. 06 ffffe509`f7a77820 fffff801`17df99e1 nt!IofCompleteRequest+0x17
  2876. 07 ffffe509`f7a77850 fffff801`17dfbf8d storport!RaidCompleteRequestEx+0x91
  2877. 08 ffffe509`f7a77920 fffff801`17df9398 storport!RaUnitAsyncError+0x145
  2878. 09 ffffe509`f7a77990 fffff801`17e08f1c storport!RaidUnitCompleteRequest+0xfa8
  2879. 0a ffffe509`f7a77b10 fffff801`145230ce storport!RaidpAdapterRedirectDpcRoutine+0x8c
  2880. 0b ffffe509`f7a77bb0 fffff801`145223b4 nt!KiExecuteAllDpcs+0x30e
  2881. 0c ffffe509`f7a77d20 fffff801`145fdc65 nt!KiRetireDpcList+0x1f4
  2882. 0d ffffe509`f7a77fb0 fffff801`145fda50 nt!KxRetireDpcList+0x5
  2883. 0e ffffe509`fc6ab6a0 fffff801`145fd11e nt!KiDispatchInterruptContinue
  2884. 0f ffffe509`fc6ab6d0 fffff801`180fc56a nt!KiDpcInterrupt+0x2ee 🌽 (async DPC arrived, error fix "suspended")
  2885. 10 ffffe509`fc6ab860 fffff801`180dd9fa Ntfs!NtfsFixDataError+0x29f5a 🌽💣 read failed and ntfs tries to fix it
  2886. 11 ffffe509`fc6ab9f0 fffff801`180dc6ff Ntfs!NtfsNonCachedIo+0xaca
  2887. 12 ffffe509`fc6abcb0 fffff801`180da62c Ntfs!NtfsCommonRead+0x1e9f
  2888. 13 ffffe509`fc6abeb0 fffff801`145185b5 Ntfs!NtfsFsdRead+0x1fc
  2889. 14 ffffe509`fc6abf80 fffff801`167b6ccf nt!IofCallDriver+0x55
  2890. 15 ffffe509`fc6abfc0 fffff801`167b48d3 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x28f
  2891. 16 ffffe509`fc6ac030 fffff801`145185b5 FLTMGR!FltpDispatch+0xa3
  2892. 17 ffffe509`fc6ac090 fffff801`14427e97 nt!IofCallDriver+0x55
  2893. 18 ffffe509`fc6ac0d0 fffff801`1445e04a nt!IoPageReadEx+0x1d7
  2894. 19 ffffe509`fc6ac140 fffff801`1445bb4d nt!MiIssueHardFaultIo+0xb6
  2895. 1a ffffe509`fc6ac190 fffff801`14528278 nt!MiIssueHardFault+0x29d
  2896. 1b ffffe509`fc6ac2a0 fffff801`14604d5e nt!MmAccessFault+0x468
  2897. 1c ffffe509`fc6ac440 00007ffb`b534b028 nt!KiPageFault+0x35e
  2898. 1d 0000008d`37d7cba8 00000000`00000000 0x00007ffb`b534b028
  2899.  
  2900. 🌽 cpu6: nothing sus
  2901. 6 ffffd30119e03180 ffffb005c589d040 (15) ffffd30119e0e240 ................
  2902. # Child-SP RetAddr Call Site
  2903. 00 ffffe509`f7c2f390 fffff801`14871e58 nt!KeAbPostRelease+0xe2
  2904. 01 ffffe509`f7c2f3f0 fffff801`14871cfc nt!EtwpGetNextGuidEntry+0xd0
  2905. 02 ffffe509`f7c2f440 fffff801`14871bdb nt!EtwpDisableTraceProviders+0x58
  2906. 03 ffffe509`f7c2f490 fffff801`14874ee6 nt!EtwpStopLoggerInstance+0x4b
  2907. 04 ffffe509`f7c2f4d0 fffff801`144f53b5 nt!EtwpLogger+0x386
  2908. 05 ffffe509`f7c2f550 fffff801`145fe278 nt!PspSystemThreadStartup+0x55
  2909. 06 ffffe509`f7c2f5a0 00000000`00000000 nt!KiStartSystemThread+0x28
  2910.  
  2911. 🌽 cpu7: nothing sus (appears to be a regular hard pagefault)
  2912. 7 ffffd30119ec0180 ffffb005f2605040 ( 9) ffffd30119ecb240 ................
  2913. # Child-SP RetAddr Call Site
  2914. 00 ffffe509`faa82ab0 fffff801`180dc6ff Ntfs!NtfsNonCachedIo+0x124c
  2915. 01 ffffe509`faa82cb0 fffff801`180da62c Ntfs!NtfsCommonRead+0x1e9f
  2916. 02 ffffe509`faa82eb0 fffff801`145185b5 Ntfs!NtfsFsdRead+0x1fc
  2917. 03 ffffe509`faa82f80 fffff801`167b6ccf nt!IofCallDriver+0x55
  2918. 04 ffffe509`faa82fc0 fffff801`167b48d3 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x28f
  2919. 05 ffffe509`faa83030 fffff801`145185b5 FLTMGR!FltpDispatch+0xa3
  2920. 06 ffffe509`faa83090 fffff801`14427e97 nt!IofCallDriver+0x55
  2921. 07 ffffe509`faa830d0 fffff801`1445e04a nt!IoPageReadEx+0x1d7
  2922. 08 ffffe509`faa83140 fffff801`1445bb4d nt!MiIssueHardFaultIo+0xb6
  2923. 09 ffffe509`faa83190 fffff801`14528278 nt!MiIssueHardFault+0x29d
  2924. 0a ffffe509`faa832a0 fffff801`14604d5e nt!MmAccessFault+0x468
  2925. 0b ffffe509`faa83440 00007ffb`b7b02c62 nt!KiPageFault+0x35e
  2926. 0c 0000008d`cc27cda0 00000000`00000000 0x00007ffb`b7b02c62
  2927.  
  2928.  
  2929. 🌽💎 We've learned that CPU5 was running a thread that encountered 💣 ntfs read error, and was attempting to fix it.
  2930. But DPC has arrived, so error fix attempt was "paused" for the duration of the DPC.
  2931. Now let's check callstacks of threads which are ready to execute.
  2932. ✨✨✨✨✨✨✨✨
  2933. 2: kd> !ready 6 🌽 using flags "6" to show callstacks
  2934. KSHARED_READY_QUEUE fffff8010f53ba00: (00) ****------------------------------------------------------------
  2935. SharedReadyQueue fffff8010f53ba00: Ready Threads at priority 9
  2936. THREAD ffffb005fa34f040 Cid 077c.80bc Teb: 0000006395787000 Win32Thread: 0000000000000000 READY on processor 80000002
  2937. Impersonation token: ffff9c895df760a0 (Level Impersonation)
  2938. DeviceMap ffff9c89494c6b60
  2939. Owning Process ffffb005e64e60c0 Image: svchost.exe
  2940. Attached Process ffffb005e961a080 Image: taskhostw.exe
  2941. Wait Start TickCount 37292195 Ticks: 0
  2942. Context Switch Count 301 IdealProcessor: 1
  2943. UserTime 00:00:00.000
  2944. KernelTime 00:00:00.031
  2945. Win32 Start Address 0x00007ffbb7b22ad0
  2946. Stack Init ffffe50a011aa5d0 Current ffffe50a011a9090
  2947. Base ffffe50a011ab000 Limit ffffe50a011a4000 Call 0000000000000000
  2948. Priority 9 BasePriority 8 IoPriority 2 PagePriority 5
  2949. Child-SP RetAddr : Call Site
  2950. ffffe50a`011a90d0 fffff801`145fd11e : nt!KxDispatchInterrupt+0x144
  2951. ffffe50a`011a9210 fffff801`14514f2a : nt!KiDpcInterrupt+0x2ee (TrapFrame @ ffffe50a`011a9210)
  2952. ffffe50a`011a93a0 fffff801`14459aaa : nt!RtlRbInsertNodeEx+0xba
  2953. ffffe50a`011a93b0 fffff801`145124c2 : nt!RtlpHpSegPageRangeShrink+0x1ba
  2954. ffffe50a`011a9420 fffff801`14bb3149 : nt!ExFreeHeapPool+0x6b2
  2955. ffffe50a`011a9500 fffff801`148c98fc : nt!ExFreePool+0x9
  2956. ffffe50a`011a9530 fffff801`148c98c9 : nt!ExpFreeTablePagedPool+0x20
  2957. ffffe50a`011a9560 fffff801`148c97b1 : nt!ExpFreeLowLevelTable+0x2d
  2958. ffffe50a`011a9590 fffff801`148cb4ac : nt!ExpFreeHandleTable+0x3d
  2959. ffffe50a`011a95e0 fffff801`148c9380 : nt!ObKillProcess+0x54
  2960. ffffe50a`011a9610 fffff801`1481146e : nt!PspRundownSingleProcess+0x204
  2961. ffffe50a`011a96a0 fffff801`14814fb7 : nt!PspAllocateProcess+0x101e
  2962. ffffe50a`011a9c10 fffff801`146085b5 : nt!NtCreateUserProcess+0xa17
  2963. ffffe50a`011aa3d0 00007ffb`b7b6e714 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffffe50a`011aa440)
  2964. 00000063`958fcb08 00000000`00000000 : 0x00007ffb`b7b6e714
  2965.  
  2966. THREAD ffffb005ebe70080 Cid 1454.27b4 Teb: 0000005693c67000 Win32Thread: ffffb005eb62fca0 READY on processor 80000002
  2967. IRP List:
  2968. ffffb005f1160250: (0006,0478) Flags: 00060043 Mdl: ffffb005fffa9d70
  2969. Not impersonating
  2970. DeviceMap ffff9c8947636600
  2971. Owning Process ffffb005e9d230c0 Image: svchost.exe
  2972. Attached Process N/A Image: N/A
  2973. Wait Start TickCount 37292195 Ticks: 0
  2974. Context Switch Count 19953 IdealProcessor: 2
  2975. UserTime 00:00:00.765
  2976. KernelTime 00:00:00.843
  2977. Win32 Start Address 0x00007ffbb741ae80
  2978. Stack Init ffffe509fa0575d0 Current ffffe509fa056630
  2979. Base ffffe509fa058000 Limit ffffe509fa051000 Call 0000000000000000
  2980. Priority 9 BasePriority 8 IoPriority 2 PagePriority 5
  2981. Child-SP RetAddr : Call Site
  2982. ffffe509`fa056670 fffff801`1440c970 : nt!KiSwapContext+0x76
  2983. ffffe509`fa0567b0 fffff801`1440be9f : nt!KiSwapThread+0x500
  2984. ffffe509`fa056860 fffff801`1440b743 : nt!KiCommitThreadWait+0x14f
  2985. ffffe509`fa056900 fffff801`180dd41a : nt!KeWaitForSingleObject+0x233
  2986. ffffe509`fa0569f0 fffff801`180dc6ff : Ntfs!NtfsNonCachedIo+0x4ea
  2987. ffffe509`fa056cb0 fffff801`180da62c : Ntfs!NtfsCommonRead+0x1e9f
  2988. ffffe509`fa056eb0 fffff801`145185b5 : Ntfs!NtfsFsdRead+0x1fc
  2989. ffffe509`fa056f80 fffff801`167b6ccf : nt!IofCallDriver+0x55
  2990. ffffe509`fa056fc0 fffff801`167b48d3 : FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x28f
  2991. ffffe509`fa057030 fffff801`145185b5 : FLTMGR!FltpDispatch+0xa3
  2992. ffffe509`fa057090 fffff801`14427e97 : nt!IofCallDriver+0x55
  2993. ffffe509`fa0570d0 fffff801`1445e04a : nt!IoPageReadEx+0x1d7
  2994. ffffe509`fa057140 fffff801`1445bb4d : nt!MiIssueHardFaultIo+0xb6
  2995. ffffe509`fa057190 fffff801`14528278 : nt!MiIssueHardFault+0x29d
  2996. ffffe509`fa0572a0 fffff801`14604d5e : nt!MmAccessFault+0x468 🌽 pagefault, by now it should be resolved
  2997. ffffe509`fa057440 00007ffb`7ea6d5c0 : nt!KiPageFault+0x35e (TrapFrame @ ffffe509`fa057440)
  2998. 00000056`9437f6a8 00000000`00000000 : 0x00007ffb`7ea6d5c0
  2999.  
  3000. THREAD ffffb005f7c08080 Cid 0754.5688 Teb: 000000fd079e6000 Win32Thread: 0000000000000000 READY on processor 80000003
  3001. IRP List:
  3002. ffffb005f6994a20: (0006,0478) Flags: 00060043 Mdl: ffffb005ee2ebb00
  3003. Not impersonating
  3004. DeviceMap ffff9c894c921810
  3005. Owning Process ffffb005e64ca080 Image: svchost.exe
  3006. Attached Process N/A Image: N/A
  3007. Wait Start TickCount 37292195 Ticks: 0
  3008. Context Switch Count 4541 IdealProcessor: 3
  3009. UserTime 00:00:00.531
  3010. KernelTime 00:00:00.140
  3011. Win32 Start Address 0x00007ffbac7cae30
  3012. Stack Init ffffe50a015645d0 Current ffffe50a01563630
  3013. Base ffffe50a01565000 Limit ffffe50a0155e000 Call 0000000000000000
  3014. Priority 9 BasePriority 8 IoPriority 2 PagePriority 5
  3015. Child-SP RetAddr : Call Site
  3016. ffffe50a`01563670 fffff801`1440c970 : nt!KiSwapContext+0x76
  3017. ffffe50a`015637b0 fffff801`1440be9f : nt!KiSwapThread+0x500
  3018. ffffe50a`01563860 fffff801`1440b743 : nt!KiCommitThreadWait+0x14f
  3019. ffffe50a`01563900 fffff801`180dd41a : nt!KeWaitForSingleObject+0x233
  3020. ffffe50a`015639f0 fffff801`180dc6ff : Ntfs!NtfsNonCachedIo+0x4ea
  3021. ffffe50a`01563cb0 fffff801`180da62c : Ntfs!NtfsCommonRead+0x1e9f
  3022. ffffe50a`01563eb0 fffff801`145185b5 : Ntfs!NtfsFsdRead+0x1fc
  3023. ffffe50a`01563f80 fffff801`167b6ccf : nt!IofCallDriver+0x55
  3024. ffffe50a`01563fc0 fffff801`167b48d3 : FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x28f
  3025. ffffe50a`01564030 fffff801`145185b5 : FLTMGR!FltpDispatch+0xa3
  3026. ffffe50a`01564090 fffff801`14427e97 : nt!IofCallDriver+0x55
  3027. ffffe50a`015640d0 fffff801`1445e04a : nt!IoPageReadEx+0x1d7
  3028. ffffe50a`01564140 fffff801`1445bb4d : nt!MiIssueHardFaultIo+0xb6
  3029. ffffe50a`01564190 fffff801`14528278 : nt!MiIssueHardFault+0x29d
  3030. ffffe50a`015642a0 fffff801`14604d5e : nt!MmAccessFault+0x468 🌽 another pagefault
  3031. ffffe50a`01564440 00007ffb`b5818b5e : nt!KiPageFault+0x35e (TrapFrame @ ffffe50a`01564440)
  3032. 000000fd`07b7f018 00000000`00000000 : 0x00007ffb`b5818b5e
  3033.  
  3034. THREAD ffffb005e637b080 Cid 0408.05d0 Teb: 0000008d3650c000 Win32Thread: 0000000000000000 READY on processor 80000002
  3035. Not impersonating
  3036. DeviceMap ffff9c8947636600
  3037. Owning Process ffffb005e60d6080 Image: svchost.exe
  3038. Attached Process N/A Image: N/A
  3039. Wait Start TickCount 37292195 Ticks: 0
  3040. Context Switch Count 1873 IdealProcessor: 2
  3041. UserTime 00:00:00.078
  3042. KernelTime 00:00:00.000
  3043. Win32 Start Address 0x00007ffbaf489a90
  3044. Stack Init ffffe509f88805d0 Current ffffe509f887fce0
  3045. Base ffffe509f8881000 Limit ffffe509f887a000 Call 0000000000000000
  3046. Priority 9 BasePriority 8 IoPriority 2 PagePriority 5
  3047. Child-SP RetAddr : Call Site
  3048. ffffe509`f887fd20 fffff801`1440c970 : nt!KiSwapContext+0x76
  3049. ffffe509`f887fe60 fffff801`1440be9f : nt!KiSwapThread+0x500
  3050. ffffe509`f887ff10 fffff801`1440b743 : nt!KiCommitThreadWait+0x14f
  3051. ffffe509`f887ffb0 fffff801`1445d988 : nt!KeWaitForSingleObject+0x233
  3052. ffffe509`f88800a0 fffff801`1445bc75 : nt!MiWaitForInPageComplete+0x1f8
  3053. ffffe509`f8880190 fffff801`14528278 : nt!MiIssueHardFault+0x3c5
  3054. ffffe509`f88802a0 fffff801`14604d5e : nt!MmAccessFault+0x468 🌽 and another
  3055. ffffe509`f8880440 00007ffb`b7afca51 : nt!KiPageFault+0x35e (TrapFrame @ ffffe509`f8880440)
  3056. 0000008d`36eff430 00000000`00000000 : 0x00007ffb`b7afca51
  3057.  
  3058. THREAD ffffb005f36e3080 Cid 0408.6fbc Teb: 0000008d36574000 Win32Thread: 0000000000000000 READY on processor 80000003
  3059. Not impersonating
  3060. DeviceMap ffff9c8947636600
  3061. Owning Process ffffb005e60d6080 Image: svchost.exe
  3062. Attached Process N/A Image: N/A
  3063. Wait Start TickCount 37292195 Ticks: 0
  3064. Context Switch Count 1 IdealProcessor: 3
  3065. UserTime 00:00:00.000
  3066. KernelTime 00:00:00.000
  3067. Win32 Start Address 0x00007ffbb7b22ad0
  3068. Stack Init ffffe509fe60d5d0 Current ffffe509fe60cce0
  3069. Base ffffe509fe60e000 Limit ffffe509fe607000 Call 0000000000000000
  3070. Priority 9 BasePriority 8 Priority Floor 9 IoPriority 2 PagePriority 5
  3071. Child-SP RetAddr : Call Site
  3072. ffffe509`fe60cd20 fffff801`1440c970 : nt!KiSwapContext+0x76
  3073. ffffe509`fe60ce60 fffff801`1440be9f : nt!KiSwapThread+0x500
  3074. ffffe509`fe60cf10 fffff801`1440b743 : nt!KiCommitThreadWait+0x14f
  3075. ffffe509`fe60cfb0 fffff801`1445d988 : nt!KeWaitForSingleObject+0x233
  3076. ffffe509`fe60d0a0 fffff801`1445bc75 : nt!MiWaitForInPageComplete+0x1f8
  3077. ffffe509`fe60d190 fffff801`14528278 : nt!MiIssueHardFault+0x3c5
  3078. ffffe509`fe60d2a0 fffff801`14604d5e : nt!MmAccessFault+0x468 🌽 fourth page fault
  3079. ffffe509`fe60d440 00007ffb`b7b1836e : nt!KiPageFault+0x35e (TrapFrame @ ffffe509`fe60d440)
  3080. 0000008d`39cff680 00000000`00000000 : 0x00007ffb`b7b1836e
  3081.  
  3082. SharedReadyQueue fffff8010f53ba00: Ready Threads at priority 8
  3083. THREAD ffffb005ea032080 Cid 1368.158c Teb: 0000009371b65000 Win32Thread: 0000000000000000 READY on processor 80000003
  3084. Not impersonating
  3085. DeviceMap ffff9c8947636600
  3086. Owning Process ffffb005e9ce2080 Image: ICEsoundService64.exe
  3087. Attached Process N/A Image: N/A
  3088. Wait Start TickCount 37292195 Ticks: 0
  3089. Context Switch Count 794952 IdealProcessor: 3
  3090. UserTime 00:00:01.265
  3091. KernelTime 00:00:09.031
  3092. Win32 Start Address 0x00007ff7c4ccfd40
  3093. Stack Init ffffe509fa8c75d0 Current ffffe509fa8c7200
  3094. Base ffffe509fa8c8000 Limit ffffe509fa8c1000 Call 0000000000000000
  3095. Priority 8 BasePriority 8 IoPriority 2 PagePriority 5
  3096. Child-SP RetAddr : Call Site
  3097. ffffe509`fa8c7240 fffff801`145451db : nt!KiSwapContext+0x76
  3098. ffffe509`fa8c7380 fffff801`1490408f : nt!KeDelayExecutionThread+0x37b
  3099. ffffe509`fa8c7410 fffff801`146085b5 : nt!NtDelayExecution+0x5f
  3100. ffffe509`fa8c7440 00007ffb`b7b6d4a4 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffffe509`fa8c7440)
  3101. 00000093`71ff74a8 00000000`00000000 : 0x00007ffb`b7b6d4a4
  3102.  
  3103. Processor 0: No threads in READY state
  3104. Processor 1: No threads in READY state
  3105. Processor 2: No threads in READY state
  3106. Processor 3: No threads in READY state
  3107. KSHARED_READY_QUEUE ffffd30119c96a00: (00) ----****--------------------------------------------------------
  3108. SharedReadyQueue ffffd30119c96a00: Ready Threads at priority 7
  3109. THREAD ffffb005f4ea80c0 Cid 0a9c.6114 Teb: 0000007e2165e000 Win32Thread: 0000000000000000 READY on processor 80000005
  3110. Not impersonating
  3111. DeviceMap ffff9c8947636600
  3112. Owning Process ffffb005e6710080 Image: taskhostw.exe
  3113. Attached Process N/A Image: N/A
  3114. Wait Start TickCount 37292195 Ticks: 0
  3115. Context Switch Count 23 IdealProcessor: 0
  3116. UserTime 00:00:00.000
  3117. KernelTime 00:00:00.000
  3118. Win32 Start Address 0x00007ffbb7b22ad0
  3119. Stack Init ffffe509fafaf5d0 Current ffffe509fafaf2c0
  3120. Base ffffe509fafb0000 Limit ffffe509fafa9000 Call 0000000000000000
  3121. Priority 7 BasePriority 6 IoPriority 2 PagePriority 5
  3122. Child-SP RetAddr : Call Site
  3123. ffffe509`fafaf300 fffff801`145fd11e : nt!KxDispatchInterrupt+0x144
  3124. ffffe509`fafaf440 00007ffb`b7ae2cb9 : nt!KiDpcInterrupt+0x2ee (TrapFrame @ ffffe509`fafaf440)
  3125. 0000007e`21c7f458 00000000`00000000 : 0x00007ffb`b7ae2cb9
  3126.  
  3127. Processor 4: Ready Threads at priority 12
  3128. THREAD ffffb005f6dcf040 Cid 0004.63b4 Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 4
  3129. Not impersonating
  3130. DeviceMap ffff9c8947636600
  3131. Owning Process ffffb005c509f100 Image: System
  3132. Attached Process N/A Image: N/A
  3133. Wait Start TickCount 37292192 Ticks: 3 (0:00:00:00.046)
  3134. Context Switch Count 174 IdealProcessor: 4 NoStackSwap
  3135. UserTime 00:00:00.000
  3136. KernelTime 00:00:00.000
  3137. Win32 Start Address nt!ExpWorkerThread (0xfffff80114541120)
  3138. Stack Init ffffe50a035495d0 Current ffffe50a03548a50
  3139. Base ffffe50a0354a000 Limit ffffe50a03543000 Call 0000000000000000
  3140. Priority 12 BasePriority 12 IoPriority 2 PagePriority 5
  3141. Child-SP RetAddr : Call Site
  3142. ffffe50a`03548a90 fffff801`14414232 : nt!KiSwapContext+0x76
  3143. ffffe50a`03548bd0 fffff801`1449240a : nt!KiProcessDeferredReadyList+0x112
  3144. ffffe50a`03548c10 fffff801`145753b3 : nt!KeSetSystemGroupAffinityThread+0x13a
  3145. ffffe50a`03548c80 fffff801`1457527c : nt!KiAcquireInterruptConnectLock+0x43
  3146. ffffe50a`03548cc0 fffff801`145750d9 : nt!KiConnectInterrupt+0x90
  3147. ffffe50a`03548d30 fffff801`1494ea8c : nt!KeConnectInterrupt+0x5d
  3148. ffffe50a`03548d90 fffff801`1494e341 : nt!IopConnectInterrupt+0x330
  3149. ffffe50a`03548ee0 fffff801`16ac7691 : nt!IoConnectInterruptEx+0x1e1
  3150. ffffe50a`03548fd0 fffff801`16ac7520 : Wdf01000!FxInterrupt::ConnectInternal+0xd5 [minkernel\wdf\framework\shared\irphandlers\pnp\km\interruptobjectkm.cpp @ 174]
  3151. ffffe50a`03549050 fffff801`16ac74af : Wdf01000!FxInterrupt::Connect+0x5c [minkernel\wdf\framework\shared\irphandlers\pnp\interruptobject.cpp @ 1187]
  3152. ffffe50a`035490e0 fffff801`16ac7403 : Wdf01000!FxPkgPnp::NotifyResourceObjectsD0+0x5b [minkernel\wdf\framework\shared\irphandlers\pnp\fxpkgpnp.cpp @ 6437]
  3153. ffffe50a`03549140 fffff801`16ac25f6 : Wdf01000!FxPkgPnp::PowerWakingConnectInterrupt+0x13 [minkernel\wdf\framework\shared\irphandlers\pnp\powerstatemachine.cpp @ 3653]
  3154. ffffe50a`035491a0 fffff801`16ac23aa : Wdf01000!FxPkgPnp::PowerEnterNewState+0x156 [minkernel\wdf\framework\shared\irphandlers\pnp\powerstatemachine.cpp @ 1699]
  3155. ffffe50a`03549300 fffff801`16ac1ad8 : Wdf01000!FxPkgPnp::PowerProcessEventInner+0xea [minkernel\wdf\framework\shared\irphandlers\pnp\powerstatemachine.cpp @ 1615]
  3156. ffffe50a`03549380 fffff801`16ac6793 : Wdf01000!FxPkgPnp::_PowerProcessEventInner+0x68 [minkernel\wdf\framework\shared\irphandlers\pnp\powerstatemachine.cpp @ 1447]
  3157. ffffe50a`035493c0 fffff801`16ac66ec : Wdf01000!FxEventQueue::EventQueueWorker+0x9b [minkernel\wdf\framework\shared\irphandlers\pnp\eventqueue.cpp @ 279]
  3158. ffffe50a`03549410 fffff801`144748e5 : Wdf01000!FxThreadedEventQueue::_WorkItemCallback+0xc [minkernel\wdf\framework\shared\irphandlers\pnp\eventqueue.cpp @ 437]
  3159. ffffe50a`03549440 fffff801`14541225 : nt!IopProcessWorkItem+0x135
  3160. ffffe50a`035494b0 fffff801`144f53b5 : nt!ExpWorkerThread+0x105
  3161. ffffe50a`03549550 fffff801`145fe278 : nt!PspSystemThreadStartup+0x55
  3162. ffffe50a`035495a0 00000000`00000000 : nt!KiStartSystemThread+0x28
  3163.  
  3164. Processor 5: No threads in READY state
  3165. Processor 6: No threads in READY state
  3166. Processor 7: No threads in READY state
  3167.  
  3168. 🌽 I can't say ready-threads look sus. Four pagefaults are fine.
  3169. But let's go back CPU4 and 🍗 recheck what it was doing. In the original analysis it was stated that this thread is
  3170. closely related to the BSOD - and that made me really, really angry 😹.
  3171. ✨✨✨✨✨✨✨✨
  3172. 2: kd> ~s4 🌽 switch to CPU4
  3173. 4: kd> !thread -1 🌽 display current thread info with callstack (notice "4:" instead of "2:")
  3174. THREAD ffffb005f3cbe040 Cid 0004.8114 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 4
  3175. Not impersonating
  3176. DeviceMap ffff9c8947636600
  3177. Owning Process ffffb005c509f100 Image: System
  3178. Attached Process N/A Image: N/A
  3179. Wait Start TickCount 37292189 Ticks: 6 (0:00:00:00.093)
  3180. Context Switch Count 171 IdealProcessor: 6 🌽 ctx switches count is a bit more than expected
  3181. UserTime 00:00:00.000
  3182. KernelTime 00:00:00.312 🌽 thread total runtime is only 312 ms (really, 0x14 osticks)
  3183. Win32 Start Address nt!PopIrpWorker (0xfffff8011459cc70)
  3184. Stack Init ffffe50a032115d0 Current ffffe50a0320fbe0
  3185. Base ffffe50a03212000 Limit ffffe50a0320b000 Call 0000000000000000
  3186. Priority 13 BasePriority 13 IoPriority 2 PagePriority 5
  3187.  
  3188. 4: kd> ?? @$thread->CreateTime 🌽 check thread creation time
  3189. union _LARGE_INTEGER 🌽 (was really silly to use SystemTime here, should have been InterruptTime)
  3190. 0x01d7686b`8d8cc02a
  3191. +0x000 LowPart : 0x8d8cc02a
  3192. +0x004 HighPart : 0n30894187
  3193. +0x000 u : <anonymous-tag>
  3194. +0x000 QuadPart : 0n132689525176320042 🌽 use this...
  3195.  
  3196. 4: kd> .formats 0n132689525176320042 🌽 ...here
  3197. Evaluate expression:
  3198. Hex: 01d7686b`8d8cc02a
  3199. Decimal: 132689525176320042
  3200. Decimal (unsigned) : 132689525176320042
  3201. Octal: 0007273206561543140052
  3202. Binary: 00000001 11010111 01101000 01101011 10001101 10001100 11000000 00101010
  3203. Chars: ..hk...*
  3204. Time: Wed Jun 23 20:08:37.632 2021 (UTC + 0:00) 🌽 ok, got thread creation time
  3205. Float: low -8.67443e-031 high 7.91284e-038
  3206. Double: 8.73829e-300
  3207.  
  3208. 4: kd> !time
  3209. CURRENT TIME:
  3210. System: 01d768b3`64a7f260 (2021 Jun 24 04:42:52.788) 🌽 ...so thread was created long ago
  3211. Interrupt: 0000054c`aea24ca6 (6 days, 17:51:30.552)
  3212. Unbiased Interrupt: 000001ba`87a64e35 (2 days, 04:47:45.136)
  3213.  
  3214. 4: kd> kv 🌽 dump callstack, with "Args to Child"
  3215. # Child-SP RetAddr : Args to Child : Call Site
  3216. 00 ffffe50a`0320fef0 fffff801`310a4cda : 00000000`000c7200 00000000`00000000 ffffb005`df8c7224 00000000`00000000 : igdkmdn64+0x315447
  3217. 01 ffffe50a`0320ff00 fffff801`310a0765 : 00000000`000c7200 00000000`00000000 ffff3b03`42fdcfb2 ffffb005`df872000 : igdkmdn64+0x314cda
  3218. 02 ffffe50a`0320ffe0 fffff801`310652ac : 00000000`00000000 ffffe50a`03210189 ffffb005`dfc5add0 00000000`00000000 : igdkmdn64+0x310765
  3219. 03 ffffe50a`03210050 fffff801`30dd6a93 : 00000000`00000000 ffffe50a`03210189 ffffb005`00000000 ffffb005`df872000 : igdkmdn64+0x2d52ac
  3220. 04 ffffe50a`03210080 fffff801`30dd9db9 : ffffe50a`03210190 fffff801`1441f670 00000000`00064040 fffff801`310a08ec : igdkmdn64+0x46a93
  3221. 05 ffffe50a`032100f0 fffff801`30dfb250 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : igdkmdn64+0x49db9
  3222. 06 ffffe50a`03210120 fffff801`30dfa268 : 00000000`00040300 00000000`00000000 00000000`00000001 fffff801`30dd9db9 : igdkmdn64+0x6b250
  3223. 07 ffffe50a`032101f0 fffff801`30de661c : ffffe50a`032102f0 00000000`00000000 00000000`00000000 ffffe50a`032102f0 : igdkmdn64+0x6a268
  3224. 08 ffffe50a`032102a0 fffff801`30f45618 : 00000000`00000000 ffffb005`c9e0b350 ffffe50a`03210420 fffff801`30dd77e8 : igdkmdn64+0x5661c
  3225. 09 ffffe50a`032102d0 fffff801`30f453a8 : ffffb005`e4dfcc00 ffffe50a`03210470 ffffb005`c9e0b210 ffffb005`e4f85b20 : igdkmdn64+0x1b5618
  3226. 0a ffffe50a`03210370 fffff801`30f3cc37 : 00000000`00000000 00000000`00000000 00000000`00000000 ffffb005`e4f85b20 : igdkmdn64+0x1b53a8
  3227. 0b ffffe50a`03210620 fffff801`30f3844f : 00000000`00000000 ffffb005`df7c0410 ffffb005`df7c0410 ffffb005`e4f85b20 : igdkmdn64+0x1acc37
  3228. 0c ffffe50a`03210650 fffff801`30f312c8 : 00000000`00000000 ffffb005`df7c0410 ffffb005`c9e0b200 fffff801`14e51b70 : igdkmdn64+0x1a844f
  3229. 0d ffffe50a`032106a0 fffff801`30f0a7f8 : ffffb005`c9e0b200 ffffb005`c9e0b200 ffffb005`c9d80000 00000000`00000000 : igdkmdn64+0x1a12c8
  3230. 0e ffffe50a`032106d0 fffff801`30ee0787 : ffffe50a`03210820 ffffb005`c9d80000 00000000`00000008 fffff801`14e50b00 : igdkmdn64+0x17a7f8
  3231. 0f ffffe50a`03210720 fffff801`30ef7578 : ffffb005`e4ef9cb0 ffffb005`e4ef9cb0 ffffb005`dfc5add0 ffffb005`e4f37090 : igdkmdn64+0x150787
  3232. 10 ffffe50a`03210960 fffff801`30dcaf1c : ffffe50a`03210e70 000002ee`00000003 ffffe50a`03210e70 fffff801`30dbeaf9 : igdkmdn64+0x167578
  3233. 11 ffffe50a`03210e10 fffff801`30da0e60 🍗 : 00000003`00000001 00000000`c0000001 fffff801`311f56c0 00000000`00000000 : igdkmdn64+0x3af1c
  3234. 12 ffffe50a`03210e40 fffff801`30da1172 : 00000003`00000001 ffffb005`df872000 00000000`00000000 fffff801`311f5830 : igdkmdn64+0x10e60
  3235. 13 ffffe50a`03210e70 fffff801`30d99221 : ffffb005`df870407 00000000`00000002 ffffb005`df872000 ffffb005`df872000 : igdkmdn64+0x11172
  3236. 14 ffffe50a`032110c0 fffff801`2d973b6e : 464c3635`314d4c00 11000a34`ffffffff ffffb005`df5cc030 00000001`00000780 : igdkmdn64+0x9221
  3237. 15 ffffe50a`032111a0 fffff801`2d9730bf : 00000000`00000005 ffffe50a`03211261 00000000`00000000 ffffb005`df5cc180 : dxgkrnl!DpiDxgkDdiSetPowerState+0x5a
  3238. 16 ffffe50a`03211200 fffff801`2d973fee : 00000000`00000000 ffffb005`e41b2360 00000000`00000001 00000000`00000001 : dxgkrnl!DpiFdoSetAdapterPowerState+0x15f
  3239. 17 ffffe50a`032112c0 fffff801`2d973441 : 00000000`40190000 00000000`00000000 fffff801`19452000 ffffb005`e41b2550 : dxgkrnl!DpiFdoHandleDevicePower+0x2ee
  3240. 18 ffffe50a`03211360 fffff801`2d974991 : ffffb005`e41b2360 ffffb005`df5cc180 ffffb005`df5cc030 ffffe50a`03211480 : dxgkrnl!DpiFdoDispatchPower+0x21
  3241. 19 ffffe50a`03211390 fffff801`1459ce49 : ffffb005`f3cbe040 ffffb005`00000000 ffffb005`f3cbe040 ffffb005`00000001 : dxgkrnl!DpiDispatchPower+0xe1
  3242. 1a ffffe50a`032114b0 fffff801`144f53b5 : ffffffff`fa0a1f00 fffff801`1459cc70 ffffb005`f49f9430 00000000`00000092 : nt!PopIrpWorker+0x1d9
  3243. 1b ffffe50a`03211550 fffff801`145fe278 : fffff801`0f533180 ffffb005`f3cbe040 fffff801`144f5360 7e7e7e7e`7f7f7f7f : nt!PspSystemThreadStartup+0x55
  3244. 1c ffffe50a`032115a0 00000000`00000000 : ffffe50a`03212000 ffffe50a`0320b000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28
  3245.  
  3246. ✨✨✨✨✨✨✨✨
  3247. 🌽 Now, the rant about homeparams (args to child) and 0xC0000001 in the frame 0x11 🍗 in the original analysis. It was
  3248. claimed that the thread runs deep inside igdkmdn64, does some setup, and returns an error, which is shown in frame 0x11:
  3249. 0xC0000001, STATUS_UNSUCCESSFUL. I believe that claim is incorrect.
  3250. 🌽 First, the thread is still running. It's running some igdkmdn64 code right now.
  3251. And it has only run (in total) for about 312 ms, which is not that much - considering the thread was created even
  3252. before the wakeup from hibernation, and the wakeup itself happened 649 ms ago. Since Intel igdkmdn64 routines are still
  3253. executing, it is premature to say anything about returned values!
  3254. 🌽 Second, the value 0xC0000001 is situated in HOMEPARAMS. It's the 0x20-byte space right next to the return address,
  3255. required by the x64 ABI. Unless the program was compiled with `/homeparams` (rare), there can be any trash in there.
  3256. Most often compiler uses this space to save some unrelated non-volatile registers, or just leaves the space (or part of
  3257. it) non-initialized. It's especially non-deterministic in kernel mode, where interrupts can use the same stack as
  3258. currently running thread - thus any random ISR may put any random value on stack (while that space is still free ofc).
  3259. 🌽 So without examining disasm and determining where the value came from, in almost all cases it's absolutely useless
  3260. to examine the "Args to Child" block on x64. May as well run `dps @rsp L6000/8` and pick any "error" value from there.
  3261. <end of rant>
  3262. 💎 Verdict: nothing sus about igdkmdn64 in this case.
  3263.  
  3264.  
  3265. ✨✨✨✨✨✨✨✨
  3266. I think we're almost done. But since I'm planning to fingerpoint at storage driver issues, let's check interesting set
  3267. of storage-related commands.
  3268.  
  3269. 4: kd> !storagekd.storhelp 🌽 like the power-related commands, `storagekd` is really well-made extension
  3270. Storage Debugger Extension
  3271. ===============================================
  3272. General Commands
  3273. ----------------
  3274. !storhelp - Displays complete help of the commands provided in this KD extension
  3275. !storclass - Dumps all class devices managed by classpnp
  3276. !storadapter - Dumps all adapters managed by Storport
  3277. !storunit - Dumps all disks managed by Storport
  3278.  
  3279. STORPORT specific commands
  3280. --------------------------
  3281. !storlogirp <args> - displays internal log entries that reference the specified IRP.
  3282. See '!storhelp storlogirp' for details.
  3283. !storloglist <args> - displays internal log entries. See '!storhelp storloglist' for details.
  3284. !storlogsrb <args> - displays internal log entries that reference the specified SRB.
  3285. See '!storhelp storlogsrb' for details.
  3286. !storsrb <address> - display details for the specified SCSI or STORAGE request block
  3287.  
  3288. STORPORT commands for NVMeoF functionality
  3289. ------------------------------------------
  3290. !nvmeofhg <host gateway address> - displays information about a NVMeoF Initiator Host Gateway
  3291. !nvmeofsp <subsystem port address> - displays information about a NVMeoF Initiator Subsystem Port
  3292. !nvmeofct <nvme controller address> - displays information about a NVMe Controllers
  3293.  
  3294. STORPORT commands for NVMe functionality
  3295. ----------------------------------------
  3296. !stornamespace <nvme namespace address> - displays information about a NVMe Namespace
  3297. !stornvmequeueslots <nvme queue address> - displays information about non-empty slots in a NVMe queue
  3298. !stornvmequeuerequests <nvme queue address> - displays information about outstanding requests in a NVMe queue
  3299.  
  3300. ✨✨✨
  3301. 🌽 Since we've run `!storagekd.storhelp`, the `storagekd` built-in extension is now loaded, and its commands became
  3302. available without `!storagekd.` prefix. So I'll be skipping the prefix.
  3303.  
  3304. 4: kd> !storclass
  3305. Storage Class Devices
  3306. Usage Legend: B = Boot, P = Paging, D = Dump, H = Hiber, R = Removable
  3307.  
  3308. FDO # Device ID Usage UP DN FL 🌽🌋 "FL" is for FaiLures
  3309. -------------------------------------------------------------------------------
  3310. ffffb005df271170 [1,2] 0 WDC WD10SPZX-80Z10T2 ? ? 0
  3311. ffffb005df273060 [1,2] 1 WDC PC SN530 SDBPNPZ-256G-1002 BPDH ? ? 16 🌽🌋💿 16 fails for our drive!
  3312.  
  3313. Usage: !storclass <class device> <level [0-2]>
  3314. !storclass 0 <level [1-2]> to display summary with upper-level and down-level IRP counts
  3315.  
  3316. Optical devices, such as DVD drives, can be listed with !wdfkd.wdfdriverinfo cdrom, and further explored
  3317. using the "!wdfkd.wdfdevice <device_handle>" and "!wdfkd.wdfdevicequeues <device_handle>" commands.
  3318.  
  3319. ✨✨✨
  3320. 4: kd> !storagekd.storclass ffffb005df273060 🌽🌋💿 explore our device
  3321. Storage class device ffffb005df273060 with extension at ffffb005df2731b0
  3322. Classpnp Internal Information at ffffb005df299040
  3323.  
  3324. Failed Requests: 🌽🌋 here's the 16 failed requests
  3325. Srb Scsi
  3326. Opcode Status Status Sense Code Sector/ListId Time Stamp
  3327. ------ ------ ------ ---------- --------------- ------------
  3328. 1b 06 02 05 24 00 11:56:05.366
  3329. 1b 06 02 05 24 00 16:53:23.397
  3330. 1b 06 02 05 24 00 20:08:40.475
  3331. 1b 07 00 00 00 00 04:42:52.788 🌽🕰️ failure time
  3332. 2a 07 00 00 00 00 08856c00 04:42:52.788
  3333. 28 07 00 00 00 00 026d6f38 04:42:52.788
  3334. 28 07 00 00 00 00 00ffc744 04:42:52.788
  3335. 2a 07 00 00 00 00 087e7398 04:42:52.788
  3336. 28 07 00 00 00 00 01a654a8 04:42:52.788
  3337. 28 07 00 00 00 00 026ed412 04:42:52.788
  3338. 28 07 00 00 00 00 07560d60 04:42:52.788
  3339. 28 07 00 00 00 00 02dfd615 04:42:52.788
  3340. 28 07 00 00 00 00 07657810 04:42:52.788
  3341. 28 07 00 00 00 00 0a0f2e58 04:42:52.788
  3342. 28 07 00 00 00 00 00690f28 04:42:52.788
  3343. 28 07 00 00 00 00 0a0f3000 04:42:52.788
  3344.  
  3345. Classpnp External Information at ffffb005df2731b0
  3346. WDC PC SN530 SDBPNPZ-256G-1002 21106000 E823_8FA6_BF53_0001_001B_448B_4630_F5CA.
  3347. Minidriver information at ffffb005df273670
  3348. Attached device object at ffffb005c5106b00 🌽🖱️1
  3349. Physical device object at ffffb005cc218050 🌽🖱️2
  3350. Media Geometry:
  3351. Bytes in a Sector = 512
  3352. Sectors per Track = 63
  3353. Tracks / Cylinder = 255
  3354. Media Length = 256060514304 bytes = ~238 GB
  3355.  
  3356. dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION ffffb005df2731b0
  3357. To get port level details, please use !storunit ffffb005cc218050 🌽🖱️3
  3358.  
  3359. ✨✨✨
  3360. 4: kd> !time
  3361. CURRENT TIME:
  3362. System: 01d768b3`64a7f260 (2021 Jun 24 04:42:52.788) 🌽🕰️ now: same as the failure time ^^
  3363. Interrupt: 0000054c`aea24ca6 (6 days, 17:51:30.552)
  3364. Unbiased Interrupt: 000001ba`87a64e35 (2 days, 04:47:45.136)
  3365.  
  3366. ✨✨✨
  3367. 4: kd> !devobj ffffb005c5106b00 🌽🖱️1 Attached Device Object
  3368. Device object (ffffb005c5106b00) is for:
  3369. \Driver\ACPI DriverObject ffffb005c50d65f0
  3370. Current Irp 00000000 RefCount 0 Type 00000032 Flags 00000010
  3371. SecurityDescriptor ffff9c8949435e60 DevExt ffffb005c80d5be0 DevObjExt ffffb005c5106c50
  3372. ExtensionFlags (0000000000)
  3373. Characteristics (0x00000100) FILE_DEVICE_SECURE_OPEN
  3374. AttachedDevice (Upper) ffffb005df273060 \Driver\disk
  3375. AttachedTo (Lower) ffffb005cc218050 \Driver\iaStorAC
  3376. Device queue is not busy.
  3377.  
  3378. ✨✨✨
  3379. 4: kd> !devobj ffffb005cc218050 🌽🖱️2 Physical Device Object...
  3380. Device object (ffffb005cc218050) is for:
  3381. 0000003e \Driver\iaStorAC DriverObject ffffb005c81d0c60 🌽☣️ ...owned by iaStorAC
  3382. Current Irp 00000000 RefCount 0 Type 00000007 Flags 00001050
  3383. SecurityDescriptor ffff9c8949279860 DevExt ffffb005cc2181a0 DevObjExt ffffb005cc219120 DevNode ffffb005c50db010
  3384. ExtensionFlags (0000000000)
  3385. Characteristics (0x00000180) FILE_AUTOGENERATED_DEVICE_NAME, FILE_DEVICE_SECURE_OPEN
  3386. AttachedDevice (Upper) ffffb005c5106b00 \Driver\ACPI
  3387. Device queue is not busy.
  3388.  
  3389. 4: kd> !storunit ffffb005cc218050 🌽🖱️3
  3390. DO: ffffb005cc218050 Ext: ffffb005cc2181a0 Adapter: ffffb005c81d21a0 Working
  3391. ERROR: Unable to retrieve InquiryData offset
  3392. Vendor: Product: SCSI ID: (3, 0, 0)
  3393. Claimed Enumerated
  3394. SlowLock: Free RemLock: 268435456 PageCount: 3
  3395. QueueTagList: ffffb005cc2183a0 Outstanding: Head: 0000000000000000 Tail: 0000000000000000 Timeout: 0 (Ticking Down)
  3396. DeviceQueue: ffffb005cc218430 Depth: 254 Status: Not Frozen PauseCount: 0 BusyCount: 0
  3397. IO Gateway: Busy Count: 0 Pause Count: 0
  3398. Requests: Outstanding: 2 Device: 61 ByPass: 0 🌽 # of requests is somewhat large?
  3399.  
  3400. [Device-Queued Requests]
  3401. IRP SRB Type SRB XRB Command MDL SGList Timeout
  3402. -----------------------------------------------------------------------------------------------------------------------------------
  3403. ffffb005eb70da50 [SCSI] ffffb005f5d44e50 n/a SCSI/WRITE (10) ffffb005edb42a90 n/a 60s
  3404. ffffb005f5faca20 [SCSI] ffffb005f5d45630 n/a SCSI/WRITE (10) ffffb005edb41c20 n/a 60s
  3405. ffffb005f5115010 [SCSI] ffffb005f5d44de0 n/a SCSI/READ (10) ffffb005fc3031f0 n/a 60s
  3406. ffffb005e67144c0 [SCSI] ffffb005f5d45470 n/a SCSI/READ (10) ffffb005fe8b0390 n/a 60s
  3407. ffffb005f24f0a20 [SCSI] ffffb005f6fc5a30 n/a SCSI/READ (10) ffffb005fe8a84b0 n/a 60s
  3408. ffffb005ee879250 [SCSI] ffffb005f0369470 n/a SCSI/READ (10) ffffb005c504a110 n/a 60s
  3409. ffffb005e62c04c0 [SCSI] ffffb005fa24b3a0 n/a SCSI/WRITE (10) ffffb005ec204d50 n/a 60s
  3410. ffffb005ece77a20 [SCSI] ffffb005f0113590 n/a SCSI/READ (10) ffffb005c504bb50 n/a 60s
  3411. ffffb005f9f14ac0 [SCSI] ffffb005f5d44210 n/a SCSI/READ (10) ffffb005c504b610 n/a 60s
  3412. ffffb005f62f1a20 [SCSI] ffffb005f5d441a0 n/a SCSI/READ (10) ffffb005c504af10 n/a 60s
  3413. ffffb005f37eeb20 [SCSI] ffffb005f5d43f70 n/a SCSI/READ (10) ffffb005c504ad50 n/a 60s
  3414. ffffb005efff2a20 [SCSI] ffffb005f5d449f0 n/a SCSI/READ (10) ffffb005c504b450 n/a 60s
  3415. ffffb005edb74010 [SCSI] ffffb005f0368f30 n/a SCSI/READ (10) ffffb005c504a2d0 n/a 60s
  3416. ffffb005f7164a60 [SCSI] ffffb005fa2a7ee0 n/a SCSI/READ (10) ffffb005c504a650 n/a 60s
  3417. ffffb005f4ee7730 [SCSI] ffffb005fa2a8340 n/a SCSI/READ (10) ffffb005c504b290 n/a 60s
  3418. ffffb005ece74b20 [SCSI] ffffb005fa2a8420 n/a SCSI/READ (10) ffffb005c504a9d0 n/a 60s
  3419. ffffb005ed786a70 [SCSI] ffffb005f6dac980 n/a SCSI/READ (10) ffffb005e93792e0 n/a 60s
  3420. ffffb005fbe65820 [SCSI] ffffb005f7f13db0 n/a SCSI/READ (10) ffffb005c504b990 n/a 60s
  3421. ffffb005eed73010 [SCSI] ffffb005f5d440c0 n/a SCSI/READ (10) ffffb00600545ba0 n/a 60s
  3422. ffffb005f9305a20 [SCSI] ffffb005f0368b40 n/a SCSI/READ (10) ffffb005fc3021a0 n/a 60s
  3423. ffffb005ef4f97e0 [SCSI] ffffb005eb3a0250 n/a SCSI/READ (10) ffffb005fef369d0 n/a 60s
  3424. ffffb005f57d3010 [SCSI] ffffb005f6dacd70 n/a SCSI/WRITE (10) ffffb005ef67a390 n/a 60s
  3425. ffffb005fbc49a20 [SCSI] ffffb005f5d43f00 n/a SCSI/READ (10) ffffb005fac03ef0 n/a 60s
  3426. ffffb005ea6224b0 [SCSI] ffffb005f0368670 n/a SCSI/READ (10) ffffb00600544b50 n/a 60s
  3427. ffffb005f9b46a20 [SCSI] ffffb005f5d44980 n/a SCSI/READ (10) ffffb00600543cd0 n/a 60s
  3428. ffffb005ea627010 [SCSI] ffffb005f0368a60 n/a SCSI/READ (10) ffffb00600544980 n/a 60s
  3429. ffffb005fc1c7a20 [SCSI] ffffb005f5d44c90 n/a SCSI/READ (10) ffffb006005462e0 n/a 60s
  3430. ffffb005ee084a20 [SCSI] ffffb005f5d44910 n/a SCSI/READ (10) ffffb00600543760 n/a 60s
  3431. ffffb005f547ea20 [SCSI] ffffb005f5d44670 n/a SCSI/READ (10) ffffb005fc3028e0 n/a 60s
  3432. ffffb005f930ba20 [SCSI] ffffb005f5d44c20 n/a SCSI/WRITE (10) ffffb005f6248dd0 n/a 60s
  3433. ffffb005f9babb20 [SCSI] ffffb005f5d44830 n/a SCSI/WRITE (10) ffffb005fce65330 n/a 60s
  3434. ffffb005f5cf3a20 [SCSI] ffffb005f5d44600 n/a SCSI/WRITE (10) ffffe509fc8219a0 n/a 60s
  3435. ffffb005ed8f2010 [SCSI] ffffb005f5d43a30 n/a SCSI/READ (10) ffffb005fce636c0 n/a 60s
  3436. ffffb005e9feea20 [SCSI] ffffb005f5d43410 n/a SCSI/READ (10) ffffb005fc3047b0 n/a 60s
  3437. ffffb005f6904010 [SCSI] ffffb005f5d439c0 n/a SCSI/READ (10) ffffb00600545290 n/a 60s
  3438. ffffb005eafbca20 [SCSI] ffffb005f36eb8f0 n/a SCSI/READ (10) ffffb00600544d20 n/a 60s
  3439. ffffb005f5355b20 [SCSI] ffffb005f5d457f0 n/a SCSI/READ (10) ffffb005fc4462e0 n/a 60s
  3440. ffffb005ece05af0 [SCSI] ffffb005f5d431e0 n/a SCSI/READ (10) ffffb005fc302e50 n/a 60s
  3441. ffffb005f27f0760 [SCSI] ffffb005f5d43100 n/a SCSI/READ (10) ffffb005fac052e0 n/a 60s
  3442. ffffb005f7fb5a20 [SCSI] ffffb005f5d436b0 n/a SCSI/READ (10) ffffb005ebfb79d0 n/a 60s
  3443. ffffb005f5160a20 [SCSI] ffffb005f5d44f30 n/a SCSI/READ (10) ffffb005fc443ea0 n/a 60s
  3444. ffffb005f2ce8b20 [SCSI] ffffb005f5d43b80 n/a SCSI/READ (10) ffffb005e9371900 n/a 60s
  3445. ffffb005f19eab20 [SCSI] ffffb005fa2a7f50 n/a SCSI/READ (10) ffffb00600545f40 n/a 60s
  3446. ffffb005eeddcb00 [SCSI] ffffb005f6dac280 n/a SCSI/READ (10) ffffb00600546680 n/a 60s
  3447. ffffb005ec091a20 [SCSI] ffffb005f5d443d0 n/a SCSI/READ (10) ffffb006005459d0 n/a 60s
  3448. ffffb005ea696b10 [SCSI] ffffb005f5d43800 n/a SCSI/READ (10) ffffb005f01c7d20 n/a 60s
  3449. ffffb005f6d10010 [SCSI] ffffb005f5d43720 n/a SCSI/READ (10) ffffb00600543590 n/a 60s
  3450. ffffb005f5ceea20 [SCSI] ffffb005f5d44520 n/a SCSI/READ (10) ffffb00600546850 n/a 60s
  3451. ffffb005fbc34a20 [SCSI] ffffb005f5d44280 n/a SCSI/READ (10) ffffb006007bb850 n/a 60s
  3452. ffffb005ee628a20 [SCSI] ffffb005f5d43cd0 n/a SCSI/READ (10) ffffb00600545d70 n/a 60s
  3453. ffffb005e9e0aa40 [SCSI] ffffb005f5d43c60 n/a SCSI/READ (10) ffffb006005464b0 n/a 60s
  3454. ffffb005f2037010 [SCSI] ffffb005f5d43bf0 n/a SCSI/READ (10) ffffb006005447b0 n/a 60s
  3455. ffffb005eec18250 [SCSI] ffffb005f5d435d0 n/a SCSI/READ (10) ffffb006005431f0 n/a 60s
  3456. ffffb005f9b51010 [SCSI] ffffb005edd65d40 n/a SCSI/READ (10) ffffb006005433c0 n/a 60s
  3457. ffffb005ec2db010 [SCSI] ffffb005f36edf00 n/a SCSI/READ (10) ffffb005ff112350 n/a 60s
  3458. ffffb005f6d78a20 [SCSI] ffffb005f5d44bb0 n/a SCSI/READ (10) ffffb00600547c40 n/a 60s
  3459. ffffb005f3ca8a20 [SCSI] ffffb005f4aa98e0 n/a SCSI/READ (10) ffffb00600547a70 n/a 60s
  3460. ffffb005f7fb8a20 [SCSI] ffffb005f36f13f0 n/a SCSI/READ (10) ffffb006005495a0 n/a 60s
  3461. ffffb005f24f1a20 [SCSI] ffffb005f36f1d20 n/a SCSI/READ (10) ffffb00600546dc0 n/a 60s
  3462. ffffb005f0edfa20 [SCSI] ffffb005f36f2030 n/a SCSI/READ (10) ffffb00600548e60 n/a 60s
  3463. ffffb005f5544a20 [SCSI] ffffb005f36f1770 n/a SCSI/WRITE (10) ffffb005fe35de30 n/a 60s
  3464.  
  3465. [Bypass-Queued Requests]
  3466. IRP SRB Type SRB XRB Command MDL SGList Timeout
  3467. -----------------------------------------------------------------------------------------------------------------------------------
  3468.  
  3469. [Outstanding Requests]
  3470. IRP SRB Type SRB XRB Command MDL SGList Timeout
  3471. -----------------------------------------------------------------------------------------------------------------------------------
  3472.  
  3473. [Completed Requests]
  3474. IRP SRB Type SRB XRB Command MDL SGList Timeout
  3475. -----------------------------------------------------------------------------------------------------------------------------------
  3476. ffffb005f3cbba20 [SCSI] ffffb005f5d45780 ffffd3011a80a020 SCSI/READ (10) ffffb005fa6a02a0 ffffd3011a80a108 60s
  3477.  
  3478. NOTE: 1 request(s) not found on the completed or outstanding list. The
  3479. requests are probably being transferred from one list to another. On an
  3480. MP machine this could be happening on a separate processor.
  3481.  
  3482.  
  3483. ✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨
  3484. ✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨
  3485. And I guess it's time to conclude this dump analysis.
  3486.  
  3487. We conducted an extensive analysis of the code flow for the thread that invoked KeBugCheckEx, including SEH exception
  3488. filter code recovery, the use of compressed memory, and PTE and PFN analysis. We identified that the thread was
  3489. deleting a segment (a section object) and had manually invoked MmAccessFault to page-in the prototype PTE before
  3490. locking it. We identified the PFN and displayed the original PTE, which indicated that PageFile #2 was needed,
  3491. corresponding to the fake pagefile used by the Store Manager (with compressed memory). To decompress memory,
  3492. the thread attached to the MemCompression process; yet the decompression attempt incurred a real page fault - because
  3493. the compressed data itself had been paged out.
  3494.  
  3495. The attempt to read compressed data from the pagefile failed: MmAccessFault, invoked from the nt!KiPageFault #PF
  3496. handler, returned STATUS_NO_SUCH_DEVICE (0xC000000E).
  3497. We then identified system pagefiles, and learned that there was only one relevant real pagefile: C:\pagefile.sys.
  3498. While looking for what could have caused the STATUS_NO_SUCH_DEVICE failure, we checked volume and filesystem device
  3499. stacks. During this process, we identified McAfee mfehidk.sys filesystem minifilter, but deemed it an unlikely cause.
  3500.  
  3501. We then identified the disk device and its stack. This revealed that a storage device using the Intel iaStorAC.sys
  3502. driver was deeply involved. That driver is related to Intel Rapid Storage Technology and is a known cause of BSODs.
  3503.  
  3504. Using `!fxdevice` command and the name of the HDD device found earlier, we dumped the device power requests log and
  3505. learned that a request for the D0 state was issued to the storage device only 618 ms ago - before that, the device was
  3506. in the D3 low-power state. We also learned that the storage device changed to the D0 state just 258 ms ago, yet the
  3507. related power IRP completed only now, 0 ms ago. We then used `!powertriage` command to validate the system power state,
  3508. and learned that the system started to wake up from hibernation only about 649 ms ago.
  3509.  
  3510. In search of more clues, we dumped the call stacks of all currently running and all ready threads, and found that a
  3511. thread running on CPU5 was in the process of attempting recovery from an NTFS error, in `ntfs!NtfsFixDataError`.
  3512.  
  3513. After that, we used the built-in Windbg extension storagekd. It revealed that there were 16 storage errors on the
  3514. relevant device stack, including 13 errors timed to the current moment with millisecond precision. The device stack
  3515. directly involved the Intel iaStorAC.sys driver. We then used the `!storunit` command and learned that there were 61
  3516. device-queued requests (mostly read requests) and one outstanding request missing.
  3517.  
  3518. ✨✨✨
  3519. So the most likely culprit is the Intel iaStorAC.sys driver, possibly in combination with its related hardware
  3520. controller, if any. The failure is likely related to mishandling device read requests during early power state
  3521. transitions. Alternatively, the driver may have been overwhelmed by the inrush of I/O requests during the system wakeup,
  3522. or it might have been a combination of both factors.
  3523.  
  3524. ✨✨✨
  3525. The recommendation for the affected user is to disable Intel Rapid Storage Technology in the BIOS. This may involve
  3526. either explicitly turning off such option, or changing the storage mode from RAID to AHCI. Most likely that'll solve the
  3527. problem.
  3528. A secondary recommendation is to check the disk device for any SMART warnings, e.g. using the free CrystalDiskInfo tool,
  3529. and replace the disk if serious issues are found.
  3530. A tertiary recommendation - in case it's impossible to disable Intel Rapid Storage for some reason - is to update
  3531. drivers via Windows Update.
  3532. A quaternary recommendation: while the final part of our analysis shows the McAfee mfehidk.sys driver is unrelated to
  3533. the crash, removing this piece of product can be a good measure for system stability.
  3534.  
Tags: Dump BSOD CrashDump dump analysis iaStorAC.sys IntelRST
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!