MEMORY LEAKS SUCKS. If you’re not familiar with what this is, it’s when a specific function or operator within the game's code tries to access some memory address that it wasn't supposed to, causing the game to gradually become more and more unstable. These delayed effects make it insanely hard to pinpoint the true cause, and by simply existing, they can throw doubt on any bug/crash list - you won’t know which are true issues solvable on their own, and which are merely side effects of the memory leak.

Well, we’ve apparently had one for about 2 months now. And our list of mass client crashes was steadily growing. Sometimes people would crash during map loads, sometimes they’d just crash randomly in the middle of a match, but stranger still was the ghosts. Yea, ghosts. Essentially a player would silently crash, but instead of disconnecting, a corrupted ghost of him would be left behind, glitching out and rubber banding back and forth as it continues moving in whatever the player's last input was. It’d live for about 5 seconds before the server finally boots it out, triggering one of 3 out-comes: Crashing nobody except the ghosted-player, crashing nearby players as well, or crashing EVERYONE in the server.

I was investigating a SourceTV demo of one such case wherein everyone crashed in the middle of a game, when I noticed something interesting by pure chance: The demo wasn’t able to be rewound. This wasn’t really anything new, as some complex maps have caused this in the past, but a small map like the one this crash occurred on causing it just seemed odd. So on a whim i decided to see if maybe it was a specific point causing it. Sure enough, within the first few minutes I could rewind it just fine. I continued slowly browsing-> rewinding my way through the demo until I finally found the exact break point. And it was quite the good one.

You see, our game currently has two ways of scaling players: A full body scale, or just a head scale. We’ve given our mappers the ability to make portals that can trigger a scale effect when entered, and on this particular map, the mapper in question had made a slight over-sight and accidentally included an additional head scale effect on his shrinking portal, having assumed the headscale value was supposed to match the body scale value. On it’s own this is no big deal, but in comes the spectate monsters.

The spectate monsters (headcrabs and birds) are ways for dead players to continue interacting with a match in a harmless manner, not being actually tangible. But because they’re essentially player controlled NPC’s, they have different bones than a player model would. And lo and behold, a spectate monster went through a head-scaling portal at the exact moment the demo became corrupted. The portal tried to scale the headcrab’s head, but there wasn’t anything for it to scale. And moments, this very player later triggered a mass client crash when he turned into a ghost. Thus, a good candidate for a memory leak was found.

I quickly organized a playtest to investigate this hunch, with the goal of seeing what happens if a ton of people do this at once. And so, here’s the magical moment everything suddenly connected, forever immortalized in this ~60 second clip of pure stupidity.

https://www.youtube.com/watch?v=9IH74EVenjw

That's right. A giant toilet-portal saved the day. A god awful bug was flushed, and we were free from the torment of numerous mass client crashes.