Hi guys,

After Aktore’s e-mail I had a look at the site and servers yesterday evening. The disk which keeps the local resized image cache was full on all three servers in production. This caused issues with generation of PDF guides. I tried fixing the issue by removing any images from the disk cache that might have been corrupted, cycling the applications in Octopus Deploy and invalidating the Cloundfront cache. This solved the issue but also caused a 30m+ window of very high server response times for all applications.

We still had issues this morning, with Cloundfront still keeping broken images in cache, causing the PDF generator to hiccup. I have set up a new invalidation to mitigate this and will be monitoring today. This issue has been steadily getting worse since 27th of December but Outlook’s fancy e-mail filtering has left the alarm messages out of my mailbox.

To manage this moving forward, I think we should offload the image resizing / caching to a secondary system, leaving the web servers to serve requests.

Aktore, please give me a call if you are working today. I need again to explain the process to getting the Admin password for the windows instances.