API tools faq
paste
Advertisement
Guest User

Dump analysis of radom w3wp.exe ThreadAbortException crash

a guest
Oct 16th, 2015
358
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
C# 18.87 KB | None | 0 0
raw download clone embed print report
  1. The email chain below shows an analysis of one of the IIS crash dump files that I provided to Microsoft Developer Support in the course of troubleshooting the random w3wp.exe crash issue detailed here: http://stackoverflow.com/questions/33001299/intermittent-crash-of-w3wp-exe-with-threadabortexception-after-net-4-6-upgrade. (Note that I have removed personal names and email addresses of MS employees.)
  2.  
  3. The obstacle here is that MS only provided this detailed analysis of one dump file. I have strong reasons to believe that a similar analysis of a different dump file would show a completely different stack trace and different affected page, proving that the issue is random:
  4. • Hammering the supposed culprit page produced no error.
  5. • The page has no custom recursive code. In fact it takes no user input at all, and just renders static content.
  6. • The <%=Me.MyBasePage.HiddenSessionID()%> call just returns an empty string. The other <%-- --%> comment blocks are also irrelevant. However, just to be sure, I removed them from the Master page (which is used by many pages) and verified that we still saw crashes.
  7. • If we roll back .NET 4.6 and recent Windows Updates on a web server, it no longer experiences random crashes.
  8.  
  9. So far, despite repeated requests and an escalation to their technical lead, MS Support has failed to provide us with a similarly-detailed analysis of any of the other 8 crash dumps that I sent them.
  10.  
  11. I am posting this in the hopes that someone can verify my analysis and/or troubleshoot their own occurrence of this issue if they are also affected.
  12.  
  13. We are now in the process of reapplying the updates in groups and waiting several days to watch for crashes, in an effort to narrow down the faulty update.
  14.  
  15. From: Jordan Rieger [mailto:jordan@webnames.ca]
  16. Sent: Friday, October 09, 2015 5:07 PM
  17. To: Microsoft Support
  18. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  19.  
  20. Would it be possible for you to analyze a different dump that also had the ThreadAbortException crash? Then you could tell me if it's the same classes (Webnames4_5 / press_release_021505) affected or different classes. I think that would be very helpful in isolating the problem.
  21.  
  22. From: Jordan Rieger [mailto:jordan@webnames.ca]
  23. Sent: Friday, October 09, 2015 5:00 PM
  24. To: Microsoft Support
  25. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  26.  
  27. Oh, I didn't see that! Sorry, thank you.
  28.  
  29. But now what puzzles me is that this page is working fine: https://www.webnames.ca/about/press_releases/press_release_021505.aspx
  30.  
  31. I can hit the URL multiple times with no crash. And there is nothing in the code of that page that does recursion. In fact it has no code-behind, just markup.
  32.  
  33. From: Microsoft Support
  34. Sent: Friday, October 09, 2015 4:49 PM
  35. To: Jordan Rieger
  36. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  37.  
  38. Hi Jordan,
  39.  
  40. Thanks for your question.
  41. But the crash is initiated from ASP.about_press_releases_press_release_021505_aspx as seen from the memory dumps.
  42.  
  43. With Regards
  44.  
  45. From: Jordan Rieger [mailto:jordan@webnames.ca]
  46. Sent: Saturday, October 10, 2015 4:19 AM
  47. To: Microsoft Support
  48. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  49.  
  50. Hi,
  51.  
  52. Thank you for the detailed stack trace. I have attached the source code for the Master page that is referenced. The code in question is front-end markup on a Master page, and I don't see how it could generate any recursion. Furthermore, that part of the code has not changed in years. Many pages on our site use this master page, and if I browse to them right now they show no problem, e.g.: https://www.webnames.ca/Login.aspx. Here is the markup:
  53.  
  54.            <form runat="server" id="Form1">
  55.                <ajaxtoolkit:ToolkitScriptManager ID="sm" runat="server" EnablePageMethods="true"
  56.                    AsyncPostBackTimeout="60" />
  57.                <%-- PlaceHolder is here to seal off "code blocks" (<% ... %>) so that controls can be dynamically added --%>
  58.                <asp:PlaceHolder ID="phForm" runat="server">
  59.                    <%=Me.MyBasePage.HiddenSessionID()%>
  60.                    <%-- Somewhere in the master page, a variable named "first" is declared, which conflicts with a
  61.            variable with the same name in the ASP.NET validator functions, so we have to wipe it here.
  62.            Also, we add a global variable to store the ClientID prefix needed to access controls in the main
  63.            content place holder.
  64.                    --%>
  65.                    <div id="contentwrap">
  66.                        <asp:ContentPlaceHolder ID="ucContent" runat="server">
  67.                        </asp:ContentPlaceHolder>
  68.                        <br />
  69.                        <br />
  70.                    </div>
  71.                    <!-- /#contentwrap -->
  72.                </asp:PlaceHolder>
  73.            </form>
  74.  
  75. The ASP.Net front end comments using <%-- --%> tags are actually old an obsolete, so you can ignore those. MyBasePage.HiddenSessionID() returns an empty string.
  76.  
  77. Is there any way you can pull out of the dump the URL of the request at the time of the crash? That might really help in pinpointing the problem. Right now it seems to be intermittent, and according to our logs, not associated with any particular request. This leads me to believe that something is going wrong in the .NET 4.6 JIT compiler or the ASP.NET page/control rendering framework that is causing recursion where it shouldn't happen.
  78.  
  79.  
  80.  
  81. From: Microsoft Support
  82. Sent: Friday, October 09, 2015 3:14 PM
  83. To: Jordan Rieger
  84. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  85.  
  86. Hi Jordan,
  87.  
  88. Please find the analysis
  89.  
  90. Dump: w3wp__Webnames__PID__5376__Date__10_08_2015__Time_10_05_05PM__82__Second_Chance_Exception_E0434352.dmp
  91.  
  92. Crashing stack 40 and dumping the stack
  93.  
  94. ChildEBP RetAddr  
  95. 00 349eef98 73d61975 KERNELBASE!RaiseException+0x6c
  96. 01 349ef034 73c6fcd4 clr!RaiseTheExceptionInternalOnly+0x27c
  97. 02 349ef04c 73c6fc65 clr!RaiseTheException+0x86
  98. 03 349ef074 73c6fbfe clr!RealCOMPlusThrowWorker+0x72
  99. 04 (Inline) -------- clr!RealCOMPlusThrow+0x23
  100. 05 349ef0a8 73e63302 clr!RealCOMPlusThrow+0x2f
  101. 06 349ef1c8 73e6287c clr!Thread::RaiseCrossContextException+0x36d
  102. 07 349ef290 73d63c48 clr!Thread::DoADCallBack+0x2d1
  103. 08 349ef2e8 0234e41b clr!UM2MDoADCallBack+0x92
  104. 09 349ef31c 74469063 CLRStub[StubLinkStub]@2177be760234e41b
  105. 0a 349ef340 744690bd webengine4!W3_MGD_HANDLER::ProcessNotification+0x5a
  106. 0b 349ef350 73d5661d webengine4!ProcessNotificationCallback+0x32
  107. 0c 349ef398 73d6465f clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x1ce
  108. 0d 349ef3ac 73d6479e clr!ThreadpoolMgr::ExecuteWorkRequest+0x42
  109. 0e 349ef414 73d59fd1 clr!ThreadpoolMgr::WorkerThreadStart+0x390
  110. 0f 349ef9ac 766386e3 clr!Thread::intermediateThreadProc+0x4d
  111. 10 349ef9b8 771ebd79 kernel32!BaseThreadInitThunk+0xe
  112. 11 349ef9fc 771ebd4c ntdll!__RtlUserThreadStart+0x72
  113. 12 349efa14 00000000 ntdll!_RtlUserThreadStart+0x1b
  114.  
  115. Dumping the exception
  116. Exception object: 1128b10c
  117. Exception type:   System.Threading.ThreadAbortException
  118. Message:          Thread was being aborted.
  119. InnerException:   <none>
  120. StackTrace (generated):
  121. <none>
  122. StackTraceString: <none>
  123. HResult: 80131530
  124.  
  125. The crash is due to cross context exception
  126.  
  127. Looking at thread 42
  128.  
  129. SP       IP       Function
  130. 34c2dd18 00000000 PrestubMethodFrame
  131. 34c2dd88 7093f57d System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
  132. 34c2ddb4 7093f59e System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)
  133. 34c2ddb8 7093f46d System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)
  134. 34c2ddc0 7093f6c3 System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  135. 34c2dde8 7093f625 System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  136. 34c2de08 7093f5ba System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)
  137. 34c2de18 1dc8ba6f ASP.app_masterpages_webnames4_5_master.__RenderphForm(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)
  138. 34c2de28 7093f57d System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
  139. 34c2de54 7093f59e System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)
  140. 34c2de58 7093f46d System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)
  141. 34c2de60 7093f6c3 System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  142. 34c2de88 7093f625 System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  143. 34c2dea8 7093f5ba System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)
  144. 34c2deb8 7093f4f1 System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
  145. 34c2dee4 7128f435 System.Web.UI.HtmlControls.HtmlForm.RenderChildren(System.Web.UI.HtmlTextWriter)
  146. 34c2df04 70950851 System.Web.UI.HtmlControls.HtmlContainerControl.Render(System.Web.UI.HtmlTextWriter)
  147. 34c2df14 7094e846 System.Web.UI.HtmlControls.HtmlForm.Render(System.Web.UI.HtmlTextWriter)
  148. 34c2df30 7093f6c3 System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  149. 34c2df58 7093f625 System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  150. 34c2df78 7094ef59 System.Web.UI.HtmlControls.HtmlForm.RenderControl(System.Web.UI.HtmlTextWriter)
  151. 34c2df88 1dc8ac44 ASP.app_masterpages_webnames4_5_master.__Render__control1(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)
  152. 34c2e4e4 7093f57d System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
  153. 34c2e510 7093f59e System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)
  154. 34c2e514 7093f46d System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)
  155. 34c2e51c 7093f6c3 System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  156. 34c2e544 7093f625 System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  157. 34c2e564 7093f5ba System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)
  158. 34c2e574 7093f4f1 System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
  159. 34c2e5a0 7093f59e System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)
  160. 34c2e5a4 708e7b7a System.Web.UI.Page.Render(System.Web.UI.HtmlTextWriter)
  161. 34c2e5b4 7093f6c3 System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  162. 34c2e5dc 7093f625 System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
  163. 34c2e5fc 7093f5ba System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)
  164. 34c2e60c 70947bf8 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
  165. 34c2e7bc 70947535 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
  166. 34c2e7f0 70947469 System.Web.UI.Page.ProcessRequest()
  167. 34c2e824 70947417 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
  168. 34c2e830 709473fb System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
  169. 34c2e844 34e5f575 ASP.about_press_releases_press_release_021505_aspx.ProcessRequest(System.Web.HttpContext)
  170. 34c2e848 7094a58d System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
  171. 34c2e88c 7091fad6 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
  172. 34c2e8cc 7092dcfb System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
  173. 34c2e9b0 70920440 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
  174. 34c2e9c8 7092c822 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
  175. 34c2ea18 709212a0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
  176. 34c2ea1c 00000000 InlinedCallFrame
  177. 34c2eb0c 70920f8f System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
  178. 34c2effc 00000000 InlinedCallFrame
  179. 34c2eff8 70974a0d DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
  180. 34c2effc 00000000 InlinedCallFrame
  181. 34c2f030 70921443 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
  182. 34c2f034 00000000 InlinedCallFrame
  183. 34c2f124 70920f8f System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
  184. 34c2f1fc 00000000 ContextTransitionFrame
  185. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  186.  
  187. Dumping the native stack:
  188.  
  189. # ChildEBP RetAddr  
  190. 00 34c2d498 6758b217 clrjit!Compiler::fgComputeLife+0x4e8
  191. 01 34c2d4e8 67589150 clrjit!Compiler::fgGlobalDataFlow+0x218
  192. 02 34c2d500 6758c433 clrjit!Compiler::compCompile+0x418
  193. 03 34c2d538 6758c890 clrjit!Compiler::compCompileHelper+0x2ff
  194. 04 34c2d5b0 6758bf5b clrjit!Compiler::compCompile+0x236
  195. 05 34c2d6a8 6758128b clrjit!jitNativeCode+0x1e9
  196. 06 34c2d6cc 73caab7b clrjit!CILJit::compileMethod+0x25
  197. 07 34c2d720 73cabf17 clr!invokeCompileMethodHelper+0xe0
  198. 08 34c2d760 73cabf5c clr!invokeCompileMethod+0x34
  199. 09 34c2d7cc 73cab87f clr!CallCompileMethodWithSEHWrapper+0x2d
  200. 0a 34c2db84 73cabb54 clr!UnsafeJitFunction+0x3dd
  201. 0b 34c2dc68 73cacac0 clr!MethodDesc::MakeJitWorker+0x398
  202. 0c 34c2dce0 73c9fe83 clr!MethodDesc::DoPrestub+0x5fc
  203. 0d 34c2dd58 73bd17f5 clr!PreStubWorker+0xe2
  204. 0e 34c2dd7c 7093f57d clr!ThePreStub+0x11
  205. 0f 34c2dda8 7093f59e System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)+0x10d
  206. 10 34c2ddb8 7093f46d System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)+0xe
  207. 11 34c2ddb8 7093f6c3 System_Web_ni!System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)+0xd
  208. 12 34c2dddc 7093f625 System_Web_ni!System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x43
  209. 13 34c2ddfc 7093f5ba System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x65
  210. 14 34c2de10 1dc8ba6f System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)+0x1a
  211. 15 34c2de1c 7093f57d App_Web_e0xyxym0!ASP.app_masterpages_webnames4_5_master.__RenderphForm(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)+0x6f
  212. 16 34c2de48 7093f59e System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)+0x10d
  213. 17 34c2ddb8 7093f46d System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)+0xe
  214. 18 34c2ddb8 7093f6c3 System_Web_ni!System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)+0xd
  215. 19 34c2dddc 7093f625 System_Web_ni!System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x43
  216. 1a 34c2ddfc 7093f5ba System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x65
  217. 1b 34c2de10 1dc8ba6f System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)+0x1a
  218.  
  219. <Trimmed due to recursion>
  220.  
  221. f5 34c2de1c 7093f57d App_Web_e0xyxym0!ASP.app_masterpages_webnames4_5_master.__RenderphForm(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)+0x6f
  222. f6 34c2de48 7093f59e System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)+0x10d
  223. f7 34c2ddb8 7093f46d System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)+0xe
  224. f8 34c2ddb8 7093f6c3 System_Web_ni!System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)+0xd
  225. f9 34c2dddc 7093f625 System_Web_ni!System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x43
  226. fa 34c2ddfc 7093f5ba System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)+0x65
  227. fb 34c2de10 1dc8ba6f System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)+0x1a
  228. fc 34c2de1c 7093f57d App_Web_e0xyxym0!ASP.app_masterpages_webnames4_5_master.__RenderphForm(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)+0x6f
  229. fd 34c2de48 7093f59e System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)+0x10d
  230. fe 34c2ddb8 7093f46d System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)+0xe
  231. ff 34c2ddb8 7093f6c3 System_Web_ni!System.Web.UI.Control.Render(System.Web.UI.HtmlTextWriter)+0xd
  232. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  233. Dumping the code:
  234.  
  235. private void __RenderphForm(HtmlTextWriter __w, Control parameterContainer)
  236. {
  237.      __w.Write(this.get_MyBasePage().HiddenSessionID());
  238.      __w.Write("\r\n                    \r\n                    <div id=\"contentwrap\">\r\n                        ");
  239.      parameterContainer.Controls[0].RenderControl(__w);
  240.      __w.Write("\r\n                        <br />\r\n                        <br />\r\n                    </div>\r\n                    <!-- /#contentwrap -->\r\n                ");
  241. }
  242.  
  243. The above code is going under recursion and causing w3wp to crash
  244. ===============================================================================================================================
  245.  
  246. Action Plan:
  247. We need to engage the developer to look into this further and why it’s going under recursion and causing stack overflow
  248. private void __RenderphForm(HtmlTextWriter __w, Control parameterContainer)
  249. {
  250.                 __w.Write(this.get_MyBasePage().HiddenSessionID());
  251.                 __w.Write("\r\n                    \r\n                    <div id=\"contentwrap\">\r\n                        ");
  252.                 parameterContainer.Controls[0].RenderControl(__w);
  253.                 __w.Write("\r\n                        <br />\r\n                        <br />\r\n                    </div>\r\n                    <!-- /#contentwrap -->\r\n                ");
  254. }
  255. ================================================================================================================================
  256. With Regards,
  257.  
  258.  
  259. From: Jordan Rieger [mailto:jordan@webnames.ca]
  260. Sent: Saturday, October 9, 2015 4:19 AM
  261. To: Microsoft Support
  262. Subject: RE: [REG:MS Support Request #] Intermittent crash of w3wp.exe with ThreadAbortException- Action plan
  263. Hi,
  264.  
  265. I have uploaded a .zip file containing 9 dumps that the Debug Diagnostic tool generated. Please investigate and advise.
  266.  
  267. Jordan
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!