Speaker 1: This New York Times piece on our our hero, Andreas, is not good. It is bad. No, it's terrible. It is really, really bad. And I'm like, who the hell wrote this thing? I'm like, Kevin, Bruce, I've got. I don't know you follow. I mean, I've got higher expectations for. Bruce's work, I got to tell you, Kevin, Bruce, if you listen to this, this thing is bad. Do it again. Right again.

Speaker 2: I got to read aloud the part that I found most.

Speaker 1: Hold on. So are we going to like the passage that absolutely like set me off? I want the same package. It's got to be the same one.

Speaker 2: So I want to.

Speaker 1: I'll turn it right down. Go for it. Read it aloud.

Speaker 2: Well, I already tweeted it out. So his job involves developing a piece of open source database software known as Postgres SQL, whose details would probably bore you to tears if I could explain them correctly, which I can't. 100 percent brain blew up. It's like you can't.

I mean, far like why? I mean, you you are just a journalist. Like, it's not like we expect you to understand and explain details. Like, let's not. Journalism has very little to do with the comprehension and explanation of details as far as I understand it.

Speaker 1: Also, like, how is like if this would bore you to tears, if I could explain the details to you, which I can't. You could apply. What would you not apply that to? You could apply that to anything. It's like the the the 2024 presidential election, which would bore you to tears. You understood that. Totally.

Speaker 2: I mean, it's like Kevin McCarthy ousted. I mean, details sound really tedious. So let's just let's just skip to the end of this whole thing. Right.

Speaker 1: Oh, man. It that it is that is absolutely atrocious. And I hope I mean, I hope to God you had an editor demanding that you that you insert that parenthetical Kevin Roos. Because it is you know, it is it is like I did something I was funny that the same thing set you off.

Adam, it is it is irresponsible. It is saying that there's something wrong with digging into the details and that by the way, things that you don't understand are boring. It's like, no, no, no, they're not actually things that you understand are actually really interesting because you actually don't understand the details. Did you keep reading this article? Because I have to say when I tell you when I read that article, like when I read that, then it's it got me got worse, I feel. You keep reading.

Speaker 2: I did. I did read the whole thing. Yes, I read the whole thing. Actually, I mean, I embarrassed to say multiple times because like I need to quote parts of it back to my mom, who was a the editor of her college newspaper, she had read it in print on the front page of the New York Times and found it just shocking.

Speaker 1: That's great. God bless you. So you called you up so you could you too could read passages a lot.

Speaker 2: Well, I talked to my I talked to my folks about it because I thought it was a really interesting one and because I was excited to have Andreis in the show and and my dad has listened to an episode of the show. So I wanted to get the heads up that there was so much excitement coming down the pike.

Speaker 1: Oh, you're right. Exactly. And this led to obviously discussing the story, which again is not very good. So the other thing that they they have a twist fit for Hollywood, tech leaders and cybersecurity researchers are hailing Mr. Frund as a hero. It's like, is the twist that they're hailing him as a hero? Me, is the writing really this bad? Andreis, are you are you with us?

Speaker 3: Yes, I am.

Speaker 1: I do not. Oh, welcome. Kevin, Roos, thanks you for showing up with working audio because this just was not the end. We just add them in me just taking swings at this terrible New York Times piece. So I'm glad you're here. She wasn't that bothered by it. So that's great, actually. That is here. No, honestly, that is great to hear. I'm glad because this is, you know, it's a big news. It is great to have you with us for what is a truly extraordinary story. And I mean, the folks have likely heard aspects of the story and but there's there's a whole lot of depth to it.

There are a bunch of interesting angles to cover. But I wonder if you if you might just start with with what your role is in Postgres and kind of the walk up to this, because I think your diligence here and investigating average behavior is such an interesting part of the story, obviously. So describe your your kind of your role with Postgres and what's your background with respect to Postgres and what were you doing that kind of walked you up to this?

Speaker 3: Sure, I've been working on Postgres for, I don't know, 15 years or so about. And as part of the overtime, more and more contributions got more and more technically involved or deeply technically involved. And if a lot of work around performance, replication, availability for the years and a lot of just.

Finding and. At some point, I became a committer in the project and I've been at for about 10 years, I would guess. And, yes, I've done a lot of reasonably sized pieces of work in Postgres and probably. One of the people in Postgres that.

Looks most on. Performance and particularly around microbenchmarking or benchmarking and profiling and seeing what's causing scalability issues and stuff like that. So I guess that I had a good starting point. Or finding any issues like this.

Speaker 1: Okay, so let me ask you if you don't mind, because when I actually first came to Sun years ago, I came into the Solaris performance group, which some it's used to being an oxymoron, but I don't think it was. And the but our I remember I was driving to lunch with our colleague Kevin Clark, who was optimizing every aspect of his route. And he had a line to me that's like performance is not just a job, it's a lifestyle. Andres, do you have that kind of or do you have that kind of is that a lifestyle for you in terms of. The way you think about a system and optimizing it, what drew you to performance in particular?

Speaker 3: And actually. No, I think. Started out doing a lot of Postgres consulting and. That being consulting for a bunch of bigger users of Postgres. And they had. Performance issues, because they were the bigger users of Postgres out there and.

We hadn't necessarily optimized for very large machines and at the time, like the number of cores were exploding and. I started to to to nice for those kinds of use cases and learn more about what to do that. I think I started out with a lot of knowledge and any of this.

Speaker 1: And interesting, so you as people are are hitting new levels of scale. I mean, that's a great way to discover performance problems because the system is being pushed into a new domain now and. You've got it. So what were some of the performance issues that you were that you were hitting and resolving over the years?

Speaker 3: Many, I think I the first really skill ability oriented work was around. And. Because of the internal locked implementation for that it calls lightweight locks, not actually that lightweight. So the name is a bit of a.

Or on that. It's always like a basic reader writer mutex lock and it didn't scale that well to many concurrent readers because there was a spin lock around the lock state, which obviously. Will not work if you have a lot of concurrent accesses and that I rewrote that to. Yeah, smart implementation and. Do that in a reasonably portable way. Postgres tries to be portable so we couldn't easily rely on few texts or any thing like that.

Speaker 1: So you were effectively doing your own synchronization primitive in shared memory and this is to allow for. I think what you're highlighting is a very important point. At the that people think of oh, I'm going to have like a couple of readers and only one writer. So they'll share this RW lock at least in the what is now the most car and it's like actually if you need to modify that state, it's a read to own on that cash line. So it's like really no different than a mutex if your old times are really short. But it sounds like you you wanted to build the data structure that would allow for true concurrent sharing.

Speaker 3: In. It's still like a cash line with state so. Right. Then it like a simple lock operations are. A single X add or. Like atomic sub like a lock release and stuff like that, but other than that, like they still like operations that are just a compare exchange on that cash line. So it's not a completely it doesn't remove like a lot of contention and we definitely could do better and like this few projects in the works to the lock scale better for like many waiters at the same time and stuff like that by but. It's become much less of an issue over time compared to when we didn't have the optimized locking. Since then there have been many other parts of performance work. I've done a lot of execution performance improvements and there's also again, lots of more to do because it turns out that if you take a university project. The 90s 80s it's not necessarily written be the fastest credit executor.

Speaker 1: Yeah, I feel that's true. The part of a great make software systems great is I feel like it's true of many software systems. It feels like with performance there's always more you can do and especially as the demands of the system shift over time. It feels like it just it's kind of an infinite supply of interesting problems.

Speaker 3: Yes, and there was also just nobody else that was really focusing on that. So I think actually I. Vigrated to avoid and what people were caring about so and that allowed me to become like. Learn more because I was doing it and I wasn't like. I think of somebody else's toes so on.

Speaker 1: Yeah, interesting. You know, the other great thing about performance work is it is as it's work that you can do totally asynchronously. You can just it's just you in the system.

You don't have to feel like you're which can be really helpful. I think if you're leading a project, I joined the kind of the PostgreSQL team there. So is it just your habit in terms of like the actual just to kind of walk us into this backdoor that you discovered? Were you looking at your own work or were you working on benchmarking a release? Or was there someone else's work? What performance were you were you measuring here?

Speaker 3: It was the patch of a colleague that in some cases improved performance, but in other cases, you knew from theoretical analysis that it would likely reduce performance a bit. You want to see how large that regression would be and the regression was in a fully cached workload. And we didn't want to get that slower because that's not an that's an important workload. So I was profiling like before after the change to see what the performance difference was. And the performance difference was very small by the time I was actually benchmarking. Like I think it was more in the one to two percent range.

Speaker 1: So one or one to two percent improvement. Regression, regression.

Speaker 3: OK, then we need because we couldn't that that patch was making IO path faster. So we couldn't potentially realistically improve the non IO path. But we wanted to make sure that the IO non IO path was not like harmed measurably. So he set on to micro optimize. To make sure that we put it down.

Speaker 1: Yeah, OK, that's interesting. And what was the nature of the work that was improving the IO path? What was it doing in particular?

Speaker 3: Work that a few colleagues and I a large project that a few colleagues and I have been working on over the last couple of years is to add proper as in Kona's IO and direct IO support into Postgres. And on part of that is to make the system aware of when you read like a table or something else that. Which order blocks are required and so that we can do like a smart read ahead or something around along those lines and read the data into shared buffers before we are blocked waiting on that. And the we don't haven't gotten all the necessary prerequisites into this release. But we have been working on at that point was infrastructure for like a helper that makes it very easy to provide all the necessary information to to smart read ahead from some reader. Some place needing all those blocks.

Speaker 1: OK, interesting. And that makes sense why you're really concerned about like, OK, we want to add these hooks that are going to allow us to actually be much smarter with respect to IO. But we really don't want to slow down the cashed path like we really need. And this is where I mean, I'm sure this is the kind of thing you're looking for where it's like a boy like a couple of extra cash line misses. And you can you can really easily or we perturb that cashed path.

Speaker 3: It's surprising actually how when the number of cash line accesses were was pretty much the same like even a few cycles showed up, which wow, somewhat surprised me because then all the expensive things in that path that wouldn't have expected to be quite so sensitive.

Speaker 1: Yeah, I kind of feel like my intuition over the effect of this stuff is like always wrong when I'm convinced that it's like, you know, it's like this has got to be a cashing issue. It's like, no, no, it's not that at all. And then when I'm like not even thinking about it, I'm getting my butt kicked by these kinds of things.

I've been there. And what are you using it to? So are you looking at kind of are you starting I guess you're starting out with a stopwatch, not by looking at hardware performance counters as you're doing this? Yeah.

Speaker 3: Once you start recording performance counters, like to pick like, particularly if you do it like a sampling based out to the profile, like that already disturbs the profile enough that I find the timing is pretty much not so anymore.

So I started out by just running in BD and what the total time is and I'm going to use profile to see, profile to see what is causing that slowdown.

Speaker 1: Okay. And so is this where you begin to see like, wait a minute, I'm seeing, I'm definitely seeing some aberrant activity on this. Like where, where do you get kind of the first hint that that something is something odd is going on?

Speaker 3: In that moment, I was the first hint was that I was seeing like repetitive repeated runs of the benchmark have a bit of noise in it and so like results varied a bit too much for my liking. So I was checking whether the benchmark system actually like in an idle state or whether some background process being too much CPU. And for that, I looked just the top and saw occasional SSHD processes using a fair bit of CPU and it's like, odd, like was directly exposed to the internet. So like you'd expect like the usual probing, but like should all be rejected like very quickly.

Speaker 2: Wasn't the case. That's super fishy. That's really fishy even on its face.

Speaker 1: Yeah, but also it also reminds you of out of the discussion we had last week in terms of like, this is the kind of thing where you're like, Hey, what's going on those SSHD, SSHD processes? Like that, that's, that can't be important right now. Like that's not what like pop up. I assume I'd raise like you pop up and then like disappear. You're like, okay, that's not, is that related? Wait a minute. Is that related?

Speaker 3: I mean, are you, I mean, I, I don't like not understanding what's going on. So I think it's very easy to distract me with random things popping up that have performance effects because I like, if it's like something that I understand why it's using CPU, then it's fine. But if it's like something I don't understand, and bothers me, and it wasn't like the first time I was seeing it, like, that's like the benchmarking was going on for like several hours. But just after a while is okay, this is something I should need, I need to look at.

Speaker 2: Yeah, you know, Andres, Brian and I spent years and years going around giving detrus demos and Brian, I don't know how often you did. Yeah, often my demo would just be like, what's going on right now on my box.

In fact, I was giving one of those demos, maybe a couple of years ago, maybe three years ago at Oxide. And I found this bizarre behavior in our VPN software, where it was like waking up and forking and exacting every second.

Speaker 3: So like, yeah, it turns out actually they said it was a known bug. So that's right, working as intended. But yeah, like when you start looking at the system at large, you see all kinds of weird stuff that each one of these things can merit its own incredibly deep dive.

Speaker 1: Well, I think it's also I'm so impressed that you because I feel like you see these things and you're like, okay, this is very strange behavior. And you start to dig into it and it gets kind of stranger. And like, there's a point like, okay, I've got to like go back to what I was trying to do here.

I need to and that hitting that balance of like, no, wait a minute, what is going on here? How did you I mean, you surely must have had that moment of just like, okay, is that is this what I want to go investigate and then realizing in a way that something actually very strange is going on here?

Speaker 3: Yeah, certainly, I think I stopped investigating and started investigating a couple times. Right, interesting. And I'm not sure that I would actually have investigated it if I hadn't seen some weird symptoms like a couple weeks earlier, in some automated testing of Postgres where suddenly, what all grind was after a few package updates was reporting some think rights that were not permissible like to supposedly inaccessible memory and couldn't figure out what was going on at the time.

There's definitely something right at the time. But I traveling so I had to stop looking and then when I now like doing the benchmarking was seeing that the profiles were showing time spent in lib LZMA, which should really not have any part of this workload. An SSH login should not be in the LZMA because like, SSHD does not use LZMA directly and no meaningful journaling or anything like logging going on.

So there is also no reason for like journal the system to log anything to like to use compression. So I was where like, okay, there's definitely something wrong because I'd seen the same symbols in the log grind reports that I now was seeing in the profile.

Speaker 1: Oh, interesting. Yeah. Interesting. Because there's always that moment when you're debugging a problem, you're like kind of drifting around a little bit and you're trying to like think about it from first principles and answer questions and so on. But then you get something where you're just like blood in the water. You know what I mean? Where you're just like, no, wait, what that this is?

I don't know if this is even related to what I'm investigating, but this is definitely apparent. And it sounds like Andres, is it reasonable to say that seeing LZMA there was like, wait a minute, that does not make sense.

Speaker 3: Yeah, I did not make sense. And I was first thinking it could just be like, I don't know, the profile of doing something weird.

Speaker 1: Totally.

Speaker 3: Everyone knows beginnings. And I was getting the debug symbols, or I don't know. But like, all those didn't really seem hidden. When I was entering the debugger, attaching with the debugger, which was it was just about long enough that I could script it to catch shortly after fork. And I would see, like, okay, this is in somewhere with a corrupted stack, or the there's no whatever, what's a debug symbol. The EH frame information to for the debugger to know where are the frame at frame, the stack frames, didn't make sense. And it was just showing somewhere like a bunch of unknown things. And it would occasionally lose track of the stack at all.

Like, it wouldn't need nowhere mainness anymore and stuff like that, or not main, because it never reached main, but like start. And that seemed just wrong. And then I just dug in deeper and deeper. And I definitely like lost track a couple of times because they ejected the backdoor into the release tar ball. But the first time I wrote the release tar ball, I actually accidentally didn't do that. But it generated tar GZ or tar, whatever it was, that source, not source, source, which GitHub generates. And that is just like the normal source, the kit checkout. And that doesn't have the problem. So I was trying to produce it with my own built app.

Speaker 1: And we at this point, like, what do you think you're trying to reproduce? Are you like, there is good? I mean, you're obviously going into this thinking this is just a software. It's a very weird bug of some flavor or another, where we're pulling in this thing that we shouldn't be pulling in.

Speaker 3: And I think it takes too long to think that Oh, is there something bad going on? Because really, that's no, if you ever like, I don't know, but it's very rare to see like, actual CPU usage be like before you reach main. And have that if you have like some if you're linking, I don't know, to live LVM or something with we just ginormous or Libre office, okay, then you could perhaps expect like, full time in the dynamic linker. But that wouldn't be the case for SSHD.

Because like the libraries that it needs to aren't that large. And there was a really good explanation for any of this. I thought for a while it could just be a bug in like, I don't know, the, the symbol that is injected is called, I don't know, underscore get underscore CPU ID, which I think is just named to similar to some because G lip see function to just underscore underscore get CPU ID. And that should never take long. And it was very quickly visible when single stepping that it was like, way more than just do a CPU ID computation. And then it was clear that there's something weird going on.

Speaker 1: Okay, so you're seeing just to kind of replay that. So you're seeing this CPU utilization, heavy CPU utilization before main, I think I would still assume that you've got to stuck in an it section of some shared object that is, that is being accidentally quadratic as our colleague

Speaker 3: Matt, he was actually before any of the init callbacks that call it was before.

Speaker 1: Okay. Yes.

Speaker 3: So that could be a while to figure out because I didn't actually know how part of elf works. Okay. Like the set a breakpoint in the C for the point where it calls the init callbacks. And those are actually after it calls all like loads all the shared libraries into memory. And has performed relocations because we could have cross calls across all the shared libraries. And so you have to first perform all those relocations. And this part and the way that I've the back door injected itself was that it's this Ivans are like a special kind of relocation basically where target address is computed. And that's was where it was like doing the initial start of the injection. That was what was so suspicious. Yeah.

Speaker 2: And when you when you say it was taken a while before we even got to me, like how long was it taking?

Speaker 3: It's those 500 milliseconds.

Speaker 1: Crazy. So I figured out that's performing crazy. It got a lot easier after I figured out that I could reproduce this without a running SSHD. Yeah, interesting. At some point I forked my investigation to see why I couldn't reproduce problem or critical problem and running SSHD from a console. Then everything was normal. Only when I started starting it with phone system D. The problem was also clearly suspicious.

Speaker 1: Okay. I was going to ask. When is that kind of first moment of like, I think I'm seeing something nefarious here versus something that's just very odd and just a very strange bug? When you were running it differently or when is that kind of that? Because that must have been a wild one.

Speaker 3: I don't think there was a single moment. It was like gradually increasing. To be honest, there's also some aspect of like after that, once I figured that out that it was nefarious, such a whirlwind. I kind of don't have the timing.

Speaker 1: Oh, I don't like down anymore. Oh, I know. Absolutely. Time would stand still. At that point, once you are like, I am looking at a back door, I can only imagine that you would not.

You would be so locked in at that point. What is the timeline here from first starting to experiment with this patch and benchmark it to wait a minute, they've got this extra CB utilization? I mean, I can imagine that happening over days or weeks or maybe hours, but it feels like this is over a slightly longer period of time.

Speaker 3: Yeah, so the first time I saw those wall grind errors that was about a month before I looked at the CPU utilization. I had kind of forgotten about that. Even the first half an hour of looking into the CPU utilization, it all looks very familiar in some way, but I couldn't actually remember even though it was only like a month. But after a while, it came back and luckily had like a shell history of what I hadn't looked at.

Speaker 1: Saved by the show history. God, so many times. My butt's been pulled out of the fire by my own show history. Thank you, past Brian.

Speaker 3: Interesting. It's okay to use- And I think when it was a couple hours of investigation that I was sure that something was weird, I wasn't yet sure where that problem was. That it was actually in LVMA, any of that was all clear. And figuring out that there was something weird to the report that was date, like a few days.

Speaker 1: A few, that was been some pretty extraordinary days, I would imagine. Yes. Pretty focused days. I don't know if you have- If you- This is where- I understand that my kids tell me that if you die in your house, your cats will eat you within a matter of days. I don't know if that's true or not, but I've never exactly looked at the cats the same way. Again, I don't know if you live alone or with pets or anything, but it just feels like you'd be like so locked in once you are realizing like I am on the path of something that's really big, I mean, and extraordinary.

Speaker 3: I tried to have a few days a week where I don't have any meetings. I think the first day I looked into it, I was looking in a no meeting day. And so I spent most of that day and then the next day was a meeting day.

So I remember sitting a bunch of meetings and I'd really being able to concentrate. Because it feels like I should- I need to continue looking into this. Also, I can't be a DIY at this point.

Speaker 2: Yeah, I have this open can of plutonium in my kitchen that really I should be attending to.

Speaker 1: Yeah, I'm sorry. Exactly. I need to actually save humanity over here. But no, this sounds like no, the state we have for the coming month, it sounds important. I think- Wow. So, 500 milliseconds also should be said. This is an extraordinary long amount of- and this is like 500 milliseconds of compute time it sounds like.

Speaker 3: Yeah, that was like nearly completely CPU-bound. CPU-bound. It was like a bit of a very few cache line misses, but like nearly all just- Number of it's- I don't remember the exact APC. It was like clearly bound by the number of instructions, nothing else. The reason that it was actually expensive is that it's all like due to obfuscation, not because it was doing expensive crypto or anything. They wanted to remove the- There's no strings visible in the binary in that injected object file. Instead, they have some weird obfuscated try of like very strings and string comparisons, it walks basically- and this is like where I am starting to not really understand.

This is like largely from other people looking into it after I volicized the report. Matches like this symbol names via these obfuscated strings and that turns out to be really expensive the way they implemented it. They do that for every symbol in the various shared libraries that are loaded.

That's where all the time is spent. If they have done less obfuscation, I probably would not have noticed anything was wrong because a few hundred microseconds I wouldn't have been able to notice.

Speaker 1: Yeah, that's it. I think that again, for software engineers, you basically know this, for Kevin Roos, for whom the details are apparently too boring, that's an extraordinary amount of CPU time. You almost certainly have something that is either compute intensive like crypto or what have you, or you've got something that is just iterating over a lot of data. It sounds like that's in this case, it's like going over every symbol obfuscated doing an expensive string comparison. It's like, yeah, that's going to add up if you have a lot of symbols.

Speaker 3: Yes. It's not even that there are that crazy amount of symbols. This is HD compared to some budget. But string-matching routine takes a few good cycles for each individual character like it does that.

Speaker 1: So at what point do you realize this is something nefarious? You don't know what it's doing. Where do you turn that dial of like, hey, I need to, it is time to let other people know about this versus I need to check my own work. That must have been balance in there as well. This is serious and I need to make security teams aware of this. What was that inner dialogue like?

Speaker 3: I think I was trying to, like initially I was wondering whether I should report it just at that stage. It was all unclear where the problem was coming from that I felt like I didn't know where exactly to report to and who to let know. So I waited and wanted to understand the, I know, is the source of the problem.

And at some point I figured out that it was like LZ or XZ, I guess. And actually I thought it was the Debian package maintainer that was where the source of all of this was from. It was visible in the Debian package sources, like the injected text thing. It wasn't visible in like the downloaded or in the Git checkout.

Speaker 2: So until I figured it out, I was like, okay, I'm going to report this to Debian. Yeah, right, right, right, because they must have built it wrong because when you build it from source, it's all fine.

Speaker 3: I mean, I was seeing that there was something malicious in the tarball or in the Debian sources. I was seeing that it was executing this weird thing.

Speaker 1: So your thinking is like a Debian build server has been rooted or the build process has been somehow corrupted. Is that kind of what you're thinking at that time?

Speaker 3: It was in the, like they have a Git repo from the maintainer of XZ, though, and it was visible in that Git checkout because they import the tarballs. So it wasn't, it was clear that it wasn't just the build process. It was like, either the maintainer committed something intentionally malicious or they didn't notice or the system was hacked. And then, oh, so I started writing up a report just to Debian. So I thought like, oh, I will need to let them know that something is going on. And then at some point I figured out, oh, this is not actually Debian specific. This happens. Like, it's not in the Debian package alone.

It's in the upstream tarball. I think as soon as I knew, like, how to realize that, I reported it. And people were working on coordinating, I know, what to do about it. I was like doing more investigation of what was happening. And the initial report, I didn't yet know that it was redirecting any symbols or any of that that was coming later. Oh, interesting.

Speaker 1: And so then when do you start to, because at some point, like, you start to think that that auto tools, that lib tool is to blame? Is that right?

Speaker 3: There was a weird thing that way they inject themselves into the build processes that they add a few lines before the main location that triggers, like, building the library via lib tool. And basically just adding something before a semicolon. And so it gets executed. And that then recursively, there's lib tool again, because like, part of the commands that it has, like, gets by environment, environment variables, is like the normal lib tool invocation that would also happen doing an un-bored build process that's like rebuilding the same file, like the same sum of the files multiple times. It was like, confusing me because not normally happened. And the way the output was that they added like, where in some parts of the build, they were like make, what's the add at the beginning of a make rule, so it wasn't printing it out. So I didn't even see that it was doing it twice. I was seeing like, you see invocations for the same file with different PADs.

It was very confused for a while. Because I haven't been able to figure out how to properly make make, actually also output the silenced rules. Turns out that it prints more if you do make dash n. That happened to show the problem. And there at some point, I figured out that it was double invocation of the tool rule coming from somewhere. And yeah, that made me realize that where what the modified part of the make file was.

Speaker 1: And I mean, there are, I mean, consuming 500 milliseconds in their auditing code, like I'm not sure how clever that is. This part of it feels really particularly clever. I'm not sure. Is that a fair read on it? Or they've taken, effectively, as I understand it, they have taken one of the like the corrupt data that they have to test it. And then they've perturbed that and then use that as the payload.

Speaker 3: Yeah. And the code to do that is not actually in the Git sources. That's just in the Reletar ball. So if you just look at either the release tar ball or the Git sources, their own, you can't actually figure any of it out. The notification to the build system. It looks plausible-ish. If you just use to look at Autocon, it's just like weird and for these. And only the test file really shows the problem.

Speaker 2: And it felt really- But as part of the same thing, the test data, it's all this weird fixture data that doesn't look like anything anyway. So it is just a perfect place to bury this kind of stuff.

Speaker 3: One more thing that made me really suspicious relatively early on was that I was seeing that in after 6-0, they added those test files. But there was not actually a test using them. Oh, interesting. It felt like a weird giveaway.

I don't understand. They added the test that actually used those files, but only in 5.6 .1. Whoops.

That's like, why would you add a test file 20 hours before the release and then only add a user after the release? That's off. That is off. I think they must have been hurryed because of the removal upstream system was removing the dependency on the LZMA at every load time. They were going to dynamically load the shared library only when they needed it and they wouldn't have needed it in this context. And it would have been much harder to do any redirection if you during DL Open. So I think they very hurriedly started to release it and do the injection. I think they were not prepared to go live at that point.

Speaker 2: So this is a really interesting part of the story because this is this endeavor. And we can link, we'll link some timelines in the notes, but this endeavor had been going on for multiple years. So then to see that both things were coming to fruition and there was this closing window, as you're saying Andres, of system de-changing, this PR being merged and the opportunity being missed. I mean, seems like tough to deadlines over in the state-sponsored exploit land.

Speaker 1: It's an exploit land. It's like, oh, God, it's scheduled over quality again around here.

Speaker 3: I also think that the backdoor works. There's a lot of really well done work. At the same time, it has this using the same really good obfuscation techniques. It's using way too, using it for too many things where it actually isn't required. And if it feels to me like somebody developed a lot of the techniques to, hey, how can you redirect something during a shared library load time where you can redirect symbols because later the memory for got UT, I don't know how to pronounce this.

Global offset table is not mapped read only. So you can't actually redirect anything anymore without like M protect or something. And M protect coming from some objective code, that's pretty suspicious. Wanted to do it at that time, but that required some really complicated hangups because when during the I think execution, you can't actually do any function calls.

It's like not of the sub the reader at the relocations haven't been performed completely. So you can't do easily call like a libc function or anything like that. And because of that, they needed to do like lots of complicated stuff. And if they had just done the minimum part, I think it would have been fine. Because that would have added like a 10s perhaps of milliseconds. Then they added more, looked more and more symbols up.

And like there's a list of like somebody wrote like a obfuscated list of strings and like the list of symbols it looks up that time is very large. But there was actually no need to do that. They could have just some heuristic and connect like SH connect time after they injected the back door and done more of the library initialization at that time, more of the symbol lookups, all the symbols they only needed to call themselves. And whole overhead would have been gone, would have gone and be present. And kind of feels like there was just some less experienced people using the techniques that other people built for them and overused them further and further. And it has a bit of a corporate effect.

Speaker 2: I love these people just going through the motions of like, I gotta use Bob's obfuscation tool because he's so proud of it. Now he's the VP of these exploits and just gonna do it his way, I guess.

Speaker 1: You know, I've said around that this nation state, we do not we need more mentoring, we need a better mentoring program and just no one wants to listen to me. It's always, you know, gotta get this thing out today because they're closed the door.

Speaker 2: So can we be a little more agile folks? Well, like what's with the waterfall schedule here?

Speaker 3: Well, if you are not the one making the timing, but like the projects control the timing, it's a bit harder to plan it.

Speaker 1: So you think so the this back door, because it was it clearly was important to them to backdoor SSH in particular, this all was a very long chain to get through to SSH. And I mean, pretty creatively to get to SSH through a library that it does not link to.

Speaker 3: I mean, that is, they must have had a lot of like somebody on the team must have had like a lot of UNIX experience or multiple people. Because like, stuff they knew they know about weird autoconf build issues and how to like plausibly, and they like had like weird comments in the eject like added code to build to host M4 at under like reasonable ish as good as justification as Fred and many other parts of autoconf. And let's not say it's good, but just like it's on the tech possible.

Speaker 2: And then like knowing about the way the injection works, you needed to know about ifanx. And then that was not actually enough to do anything because like it happens. invocation happens just in the the main is loaded, but not when the main binary is not loaded. So you can redirect anything for that. So then you then needed to know about the audit mechanism, just like itself like a copy from So Larissa, I don't actually remember.

Speaker 3: It is 100%. Yeah, no, no, it is. I saw that I'm like, oh, no, I'm so sorry. Because there was a no that that the that is the pletodding that that Rod and Mike added and Rod.

Speaker 2: These are these are colleagues of ours. Yeah, from the Solaris team. Yeah. And there was kind of an error. Actually, honestly, I appreciate it because there was an error. Well, because this is not true for all Linux facilities. I kind of wish that more Linux facilities would literally like just look at people that came before them and like and the the GNU linker was happy to be like, no, like we are Solaris linker is basically state of the art. So we're just going to implement everything they've got. Which I think was a good approach because there's a lot of good stuff that they did. And then and then we've got pletodding, which I actually really think is I don't think is bad.

Actually, it's kind of a hat and like I mean, because Adam so trust I think you used it used pletodding if I remember correctly. Hmm. I don't remember. I remember. I remember so trust, but I remember the specific mechanism there.

Speaker 1: He was a slightly different idea about how you might debug a system. And I mean, I'm not that Detroit's put it out of business, but it definitely made it. It took away a use case because they were kind of thinking of a debugging use case and Mike, if our memory serves, had built this thing called so trust as a shared object trust, trust being the kind of the the the old your next kind of yeah, they as trace it, but but using proper using Roger Faulkner's trace system calls and signals is kind of where where trust comes from. But the so that was kind of but yeah, I think you're right. So I think this was coming from can you all do I think took this from so whereas so I was like, I don't know. I'm sorry.

Speaker 3: I think might have made more sense back in the Solaris days, but number of symbols are actually so large these days and very common use cases that amount of information it gives you unless you add like a lot of filtering is just kind of too large. And yeah, but not something that a lot of people still need to look at and intrusion vector it provides seems like problematic to me. And I'm I haven't actually seen any like a lot of other try to look whether there's other backdoors or exploits or whatever using deal audit, but I didn't find much. But I was kind of surprised that there wasn't more because it seems like kind of made to defeat like a lot of the security mechanisms that were reduced introduced to make dumb exploitation harder. Yeah.

Speaker 1: And it also like it also I mean, I can see, you know, the kind of the genesis of the facility and why they've got it. We definitely use that I have got I've used it for things where you are able to because you're able to interpose on using a mechanism that isn't just LD preload. The problem is you're also like allowing people to kind of monkey patch programs and you're not necessarily encouraging a sovereign sharing. So the so you discover this thing is using the and you know, I kind of know that this is coming from upstream in XZ, which is super surprising. Do you at this point realize that like, there's actually a crooked committer here? When does that kind of like go on?

Speaker 3: I think that was as soon as I discovered it was coming from upstream. Yeah, interesting. I found that it referenced those test files. So I looked at who added the test files. At that point, it was pretty clear. Either the system was hijacked or that they were involved.

Yeah. And it seemed suspicious enough that it was leaning towards them being involved in some way because it not just a single day, it was multiple days. I hadn't yet like found like looking later, I found that in frame where they did weird things was a lot larger. But again, I said at that point, but it's clear that they were involved. So we couldn't be like, I couldn't really like report it to the I'm going to be able to see security contact, which was GIA at that point. Because it was clearly not going to be helpful for anybody.

Speaker 2: It was just going to tell them that, hey, you got a other machine set you can find in a scan now, we better exploit them right now. Right.

Speaker 1: Wow. Okay, because that didn't really occur to me, but of course that makes sense, that the security contact for the library is GIA. And fortunately, you know enough to realize like, I don't know if this person even exists, but this person is definitely involved in the nefarious activity. So I'm not going to, because it would be normally the responsible protocol would be to obviously get the security folks involved for a particular project. And now are you at this point working with but hat or debit, I mean debit engineers, I mean, who's kind of has a team kind of grown at this point working on this? Or are you still doing it during while you're in meetings?

Speaker 3: I'm not entirely sure what the exact precise timeline was because I was ordered it up to various security lists that are private security lists. And then continued as soon as I had sent those emails, I continued investigating and then back and forth discussions. But I don't, exact timeline there is a bit hard to, Yeah, of course. But I think I did like until the release of the public email, I did have most done most of the technical investigation of happening. I got to the point of like figuring out that it was detouring RSA public decrypt.

Speaker 1: Yeah, wow. So I mean, at this point, I mean, and maybe and presumably earlier, you've realized like this is a very big find. This is a very, very, very big story, a very big vulnerability.

I mean, at this point, are you thinking like I'm definitely deal. I mean, I guess we don't know for certain that this is a state actor, but I think the consensus is that it is likely someone who is doing this for their job. They're doing it over a some number of years. It feels like it's an intelligence agency.

Speaker 3: I mean, are you, I don't think I got any of that part. I saw that I didn't analyze what was happening upstream that carefully or anything. I was just seeing that there was those changes and were introduced by one of the maintainers and I didn't see at that point much need to look into anything further than that.

Because they was like, they're just being compromised or playing along con or I don't know, didn't really play a role for the reaction. So I pressed forward towards getting all the necessary information to actually be able to make a public closure of what's happening.

Speaker 1: And so now that the timeline really goes very, very quickly because you send that mail on, I mean, it's more or less public when you, I mean, it seems like a latency between you sending the kind of the mail to OSS security. Maybe or maybe that is at that moment it is public effectively.

Speaker 3: I mean, that's a public list.

Speaker 1: Public list, okay. So at that point, when it goes OSS security, that is the public announcement of this thing. And so this was not something that had been going on though for months behind the scenes.

Speaker 3: I mean, you are talking about a few days in total.

Speaker 1: But this is important because I think that like one of the things that was surprising certainly to those as I talked about before, but from Spectrum meltdown is that it had been Spectrum meltdown had been going on for for an extended period of time in an extremely small number of people before they kind of broaden that out. But this is not that this is really moving quickly.

Speaker 3: I think that's a very different situation because one is the back door. So somebody clearly intentionally introduced it and knew how to utilize it. As like with the Spectrum meltdown stuff, it was like researchers finding something a complicated where the real exposure of like how to exfiltrate data was like really complicated. And so I think that a lot of noise associated with it. So I think it's a lot more defensible to take a long time.

Oh, no. I don't think that you think at the time frame that it was taking it was good. But like, I think the trade-offs are very different than from something that might actually be exploited today.

Speaker 1: No, it's a very, very good point in that when you're finding like a potential vulnerability that like, oh, this is a theoretical vulnerability. There's not a known exploit in the wild. This wasn't nefarious.

And what we really want to do is we want to be sure that everyone has the opportunity to secure their systems and that you want to be of anything you want to proceed kind of as slowly as you reasonably can without divulging the vulnerability such that someone nefarious could take advantage of it. And this is really the exact opposite. It's like, you've got no, like the nefarious thing has already happened. The back door is there. We really need to be biased towards getting this thing public very, very quickly.

Speaker 3: The fix is much easier, right? Like the vector meltdown case, like you needed to build very complicated compiler and part of system modifications. Here you just needed to like go back to an already released version of a library. So it wasn't. I think the trade-offs really were kind of clear and worth noting that some people were kind of unhappy, even with like the very short disclosure.

Private to public disclosure window. They were saying, I kind of understand that once it's an back door that is in actual use, like holding back just event gives like the attacker more chances and everyone like less. Yeah.

Speaker 2: I didn't understand that. So the nature of this where my understanding is that it was attackers with a particular private key. So this was not like opening the floodgates to anyone who knew the secret combination or whatever. Everybody knew this at that site.

Speaker 3: It was like a week, like about a week or four days later or something till anybody got to that point, because. Gotcha. I think I was really the only one early on, look at what it was doing and just taking all my skills to figure out that redirecting one symbol. I'm not going to be able to reverse engineer some obfuscated, but reasonably well done. That's just where you're outside of my skill set. It also was just not necessary. Once you know that it's in the path of normal authentication, it doesn't really matter what the technical details of the rotations are. I was very worried that it was feasible for everyone because it actually weighed towards, not immediately, a small disclosure window to give the distribution the chance to have a package ready. I didn't know that somebody like some other bad people could quickly figure out what it was doing and just exploit everyone.

Speaker 1: What is your emotional state during this time? You've got to be a combination of, I mean, it's obviously excitement. You discovered something really important, really big, maybe a little bit of like, God, a little bit of fear for the damage that could be done using this. What are you kind of thinking in the kind of moments leading up to the public disclosure?

Speaker 3: It has changed a lot over the time. It was really being annoyed.

Speaker 1: I didn't actually, I didn't want to spend time on this. I'll take it as a lease out, actually.

Speaker 3: Once I started seeing that what it was, there was no way I could responsibly say, okay, I'm just going to defer this for a few days. It was like, but it wasn't a responsible option. Partially, it was really impatient to get this over with. And the other one was like, I don't know, really nervousness. Like, what is the right move? Right. Yeah. And some of like, I'm like giggling.

Speaker 1: That's great. And did you like seek, I mean, do you see counsel from someone in terms of like, what should I do? I mean, the path towards public disclosure does feel pretty clear. But, or did it feel clear actually? Did it feel clear that like, what I need to do is disclose this as broadly as possible, as quickly as possible?

Speaker 3: I think there's some, for the minutiae I don't quite want to talk about. Sure. It felt very clear that it needed to go out very broad. Yeah, that's good. Yeah. The mechanics of that, I was like wildly unsure about. Interesting.

Who do you inform? Like at that point, I hadn't even figured out that it was just like, was it like in all the BSDs? Or was it like, I don't know.

Speaker 1: Is it in the Lumos? Come on, that's not your next question. I mean, what are we doing over here? You're right.

Speaker 3: Sorry. There's so much pain associated for we with the Lumos. I kind of don't care.

Speaker 1: It's fine. We're used to it. It's fine. It's fine.

Speaker 3: All the real problems that we occasionally, or like the emulated system calls occasionally fail and stuff like that from the Postgres. That's mostly my angle of contact.

Speaker 1: It's fine. Look, Lumos is the latchkey kid of operating systems. Like we'll look at ourselves. We'll let ourselves in. We're fine. We're good. We're going to make ourselves then or we're used to it.

I don't know. Our parents haven't been here for weeks. It's fine. So, yeah, it's okay. So you don't know, but it's like, and I got it. I mean, if you actually was had a legit like God, if open BSD is vulnerable, that is actually would be a bit scary, honestly. So do you think that part?

Speaker 3: I don't remember when exactly was at some point it was become clear that the path was via system D. So that at some point used it to like to Linux. Although like, I don't know whether there was some other dependency on.

The may like, I don't know, could have been patched in like some other pulled in some other way to though. I didn't think like I didn't want to wait till I've used all those details to report, but between the public, the private and the public disclosure, there was enough time to figure out that there was this if Linux I just did actually did something. Initially, when I did the private report, I hadn't even figured out that that same script is called twice. And that's why it has like an if that's test for a sense of some files.

And then in that branch days, like Linux conditions, but then in the on that builds the injector backdoor, there's no best conditions. I hadn't figured any of that out initially.

Speaker 1: Yeah, wow. All right. So then you but now increasing confidence. Okay, this is this is Linux and the it's publicly disclosed. And I mean, you must know like this is going to be a big deal. Is it has the reaction? How has the reaction met in terms of your expectations of what you thought the reaction would be?

Speaker 3: Clean didn't quite match it. Kind of thought the initial reaction would be faster. It would be noisier in the first couple of hours and that it would be way less noisy after the first few days.

But like the got noisier and noisier and noisier over time. Like I never expected it to reach like, you know, the audience or anything like that. That's kind of thinking, okay, we know, although we get in post and some security besides well listed and stuff like that, but I didn't think that general media would write about it and.

Speaker 1: Right. Interesting. And of course, part of the reason that people are writing about it is because it's tacking into a bunch of different bigger narratives. It's certainly tacking into a narrative around open source and how we, I mean, a lot of different angles, how we maintain those projects. I mean, Adam, the sock puppet bullying that was happening. Oh, man. I mean, and because Andre's all that, I assume you, you all discovered along with us after the disclosure. I assume that you didn't see any of that before you discussed it. It's a, what was your reaction? What was your reaction to seeing some of that? Well, pretty cool in a way.

Speaker 3: And I initially already suspected that whenever I made the report that original maintainer might not be involved. Yeah, interesting. And like, I think that I saw that like, okay, now I understand the path of how that Gia actor, like, involved, got all those, like, became a maintainer reasonably quickly. It was like, explained by that. And it was like really bad to part of something that they kind of have cost a lot of hurt for Lassie's.

And it felt like in the maintainer for a long time now. And I know the feeling of like, you're overwhelmed, you can't keep up. And people are, you know, hiding you for like, not doing enough in their eyes, not helping to all the hoops that they want you to. And not very familiar in a way. Like, even though like, I don't think any of the people that have been, I think, yeah, it have been like, like really big organized attacks, but still felt like familiar.

Speaker 1: And super dark, as you're saying, in terms of the, for whatever reason, Adam, I mean, you just felt the same way that the engagement of these sock puppets and where they're doing good cop, bad cop with the sock puppets. Totally. Where, you know, they've invented Jigar Kumar as this just like absolute asshole who.

Speaker 2: But totally believable character, by the way.

Speaker 3: Like, oh yeah, totally. So many of you would, I mean, I've encountered that kind of person in, you know, the issues in PRs, you know, dozens of times, right? So you at no point are you raising red flags. Wow, this person seems over the top. Like, the person seems over the top in a normal kind of way.

Speaker 1: Yeah, you get a wonder that like inside of the nation state brainstorming session, they're like, this is a little over the top. Like now, trust me, this one is not sadly, you'd think so. You'd think this would be on the top, but this one is not even going to move the needle. No, there's like really, you're

Speaker 2: saying this person is yelling at this person who's not getting paid for their work and that is the norm. It's like, yeah, pretty much. Yeah, no, this will fly.

Speaker 3: And also like pressing way less nice interactions.

Speaker 2: You're right, you're giving a lot of good notes, but one of them is too respectful.

Speaker 1: Okay, good. Right, exactly. This is too respectful to be plausible.

Speaker 3: I wouldn't quite say that, but like I've seen people behave worse and like. You've made worse. We're like long-term email addresses and stuff like. Yeah, they get entities.

Speaker 1: Well, you know, if one outcome that comes out of this is that like people are a little less inclined to be an asshole, because they may be, people may assume that they're a nation state actor.

Like that would be a good outcome if like actually, you know, hey, why are you pressuring me to get this release out so badly on. And because the emails were, they're also like devoid of technical content. He's just like, hey, like integrate these patches. It's like, what are we talking about here? Like what are we make it more concrete? Like there's not a, it just, it felt very manipulative. And obviously it was very manipulative.

Speaker 2: And you can see them latch it. Like the moment Lassie says, you know, I'm having these long-term mental health issues, almost immediately the pressure campaign ramps up. I mean, it makes me wonder like how many more irons are in the fire? Right. How many other because like, you know, XZ can't be the only, you know, that's not a full-time job for this team, I assume. Right. Like how many other irons in the fire are they waiting? Are they kind of gently putting pressure and then waiting to see where softness emerges or where cracks emerge?

Speaker 1: Absolutely. I mean, I just, you must have had the same thought of like this is, this can't be the only one out there like this. There's got to be some other activity out there that is, I mean, would you have that same thought?

Speaker 3: I think it was, seems like big enough effort that you wouldn't try to find your way into only one project. So clearly at the beginning they must have like had multiple, let's become a maintainer type efforts.

Yeah. I don't know whether after doing that, like after getting access on XZ, then they said like, okay, we've achieved our goal because once you have backdoor SSH, like that's a, you don't really need much, many other backdoors at the same time and particularly not from the same team. So could it have been that they just stopped the other efforts and like, I think if they stopped the other efforts, like, I'm not sure that realistic to ever find that out because like, there are enough other people that are being asked holes online, like how we did that, right?

Speaker 1: Okay, like some very unfriendly that has no email identity that can't be a smaller pool. So I didn't know whether they

Speaker 3: call the concurrent efforts, but it's certainly something to be afraid of. Yeah. And I'm not sure that like it's only small projects that are at risk here. Like, it was one of the early themes. Again, the online reaction to all of this that there's so many products that are just a single trainer or quarter maintainer or something. It's not that hard to get become a computer in some big projects either. And if you look at complex projects, are in important parts of the chain, like building in open source OS, injecting yourself into a compiler or something like,

Speaker 1: Yeah, or or or or or a lib tool, honestly, I mean, not to sorry, right now, but like that would be because it's something that's kind of convoluted and scary and unloved and has a bunch of architectural issues. I mean, it just feels like, boy, you could do a lot of a lot of damage by infiltrating some of these things. And so then so we were surprised, I guess, as we all were about like, story, were you surprised is like the story kept getting kind of deeper and deeper. I think still like Adam, I didn't realize the private key business that they've so they had discovered that or tries to discover that there's a single private key that is that is basically gain access to these to compromise systems.

Speaker 3: Yes, that's, I think that took like three, four days or something. Wow. Investigation that somebody found that there's like some elliptic curve key that can be appended, I guess, to an RSA certificate. And that is how like, their backdoor decides, hey, I'm going to do in the backdoor path or the normal path.

And kind of grossness of it all. Like you send an RSA public key certificate, then somehow contains in like some data, like the curve key that then actually another set of payload, like actually contains the commands. There's like a lot of investigation of how all of this stuff works in SSHD. And it's like, it's tricky.

Speaker 1: Yeah. And then obviously another narrative that people were tacking into, I mean, whatever you previously thought about open source, this confirmed your view of it.

Speaker 3: It's a very common online discourse thing.

Speaker 1: Yeah, right. There were people just like, thank God it was open, which I think is my own probably my own personal bias is like, no, that's actually the openness here was a huge win. I think it also, I mean, I'd like to get your take on this, but I think the fact that it's all open required this attack to be way more sophisticated. I mean, like my read on this is there was a lot of mental energy spent on making the repo not match the artifact. And it would be a lot easier to do that if that were not open. The fact that that repo is open requires you to spend a lot of energy on that and that obfuscation and maybe leads to the CPU footprint that you see. I don't know, Andres, what do you kind of make of that?

Speaker 3: I don't think the way they injected the build to the build like had any impact on the runtime overhead. Oh, interesting. It was really the backdoor code purely self. They just needed away from the LZMA to SSHD and that was where it was of the challenge. Yeah. I think they could have actually committed it to the repo and nobody would have noticed, to be honest.

Speaker 1: Yeah, interesting. I mean, you look at the file, you're looking for a stray period here. So, you know...

Speaker 3: I mean, yeah. I don't think that actually played any role in the main path to exploitation. I'm kind of actually confused why they did any of that. I kind of feel like it's noisy but not really useful. Because they can actually make the whole authentication path like legitimately allow access. All the sandboxing is not very interesting. Because they can break out of the sandbox by just permitting access. I'm not sure that there's a good explanation yet why they tried that. And whether that perhaps was just a parallel path towards exploitation. They didn't get know how to utilize LZMA access to... Oh, interesting.

That's why they tried to... ...inject it, break out of the sandbox. Because it's actually not that easy to figure out how you can get to the RSA... ...function checking part without being passed some other checks first. Only happens in the SL certificate based authentication.

Which is not the most common way of doing it. Because in the normal key, just key-based authentication... ...SH first checks whether key is in authorized hosts. So, you could totally provide a fake key at that point. But you would need to know which the keys are in authorized hosts. Authorized keys.

And that would have been much less attractive, obviously. And the way that what they figured out at some point is that you can do... ...has all of that with the certificate based authentication. I suspect that they just figured that out later or concurrently. Interesting.

Speaker 1: And so you think that some of the sandboxing is kind of like... ...some tech debt they didn't bother to pay down. It's like, oh, yeah, that was an old project that didn't go anywhere.

Speaker 3: I don't know. It just confuses me. Yeah, interesting. I don't think we know yet.

Speaker 1: And so as... I mean, this was only 10 days ago. I'm sure this has been a rather intense 10 days for you. And of course, you're trying to get a release out as well. Congratulations. Did the release go out, by the way? Did you get the release out there?

Speaker 3: The feature freeze. The release will be released in a couple of moments. Okay. So you're scrambling... I've got some stuff in, but not quite all that I wanted. It turns out that it's fairly distracting.

Speaker 1: Well, yeah. And then were you expecting kind of... See the kind of the noise after the fact... ...or the fact that it kept growing was surprising to you. I did think it was great that... I mean, Satya had the... You know, your boss is boss is boss is boss. However, now how many levels up? I mean, the CEO of Microsoft.

Speaker 3: Or I think something like that. 25, however many. That must have felt good though, to see that... Just to see that it was... That your... And I love the fact that in particular, praised your craftsmanship and curiosity, which felt like it was awfully accurate. I'm not sure... Oh, wait a minute. Did you have to write that tweet for yourself? Did he...

Speaker 1: No, I didn't know anything about it. I just took a picture of it later.

Speaker 3: So I hadn't actually checked Twitter at that moment. And I think somebody sent to me a picture. And that's how I realised it. But very... It was already like, you know, overwhelmed and a bit like... over-stimulated I guess. Right. So they're like, oh, what? It's happening?

No. Yeah, it was also nice to... See that it also like, feel it... Oh, working in open source teams at Microsoft. Getting that recognition is just like... I think I couldn't have imagined. 20, 20 years ago.

Speaker 1: Yeah, that's a very interesting feeling. Yeah, that you have a company that was... that had actually kind of explicitly under the Balmer era, had pretty much positioned itself as a diametrically opposed open source. And now... you know, 20 years later as you say, not only are... implicitly supporting this effort because they're employing you and part of your job is working on the Postgres Core team, but then to see them really being explicitly encouraging it. I assume the reaction of Microsoft has weird Satya's reaction and people have been pretty positive. Is that a fair guess?

Speaker 3: Yeah, overwhelmingly. Some fusion about it... exists, part. I think... Oh, well, you're like... kind of... doing from lots of parts of Microsoft I've never had contact with because I've... I mean, I've been doing open source stuff at Microsoft. Our team has grown over the time, but it's still like an odd corner, I guess. And suddenly I was hearing from parts of Microsoft that I never... even existed.

Speaker 1: Well, hopefully there are parts of Microsoft that you need things out of because that's the... Oh, hey, look, oh, thanks. Thank you for the high-price framework. You know, there's an email that I sent you two months ago that you've been sitting on, so if you could actually please... now's the time, this is the opportunity, your window's open to get the things that you... that's great to hear. I mean, it's always, I think... I think people rightfully feel... and they should feel proud of your role and Microsoft's role in this because I think it's very important. One question I've got for you is the... because I think this is something that people are definitely curious about, is... it feels lucky. And my reaction is this is like... this is like... I'm not sure this is locked so much as it is someone who's really digging into aberrant behavior.

What's your kind of view? And I guess you had this kind of... the fact that you have the Valgrind artifacts. In other words, obviously it feels like you could have easily... not as investigated as deeply. What is your view of kind of the role of luck in this? And I think a lot of people have this like... would this have been discovered in the limit or do you think this could have hidden in perpetuity?

Speaker 3: Good luck, certainly by the role. If I get other releases, I had more of my own work and that's often then more... like doing code review, to stuff polishing and then... doing the eyes and stuff like that and not performance work. So I don't know if it looked at that time. And then... if I already had installed or... actually if the package maintainer had already integrated 5.6 .1... but sure I would have figured it out either. And the fact that Valgrind had warned me beforehand... I'm not sure that I would have figured it out without that and actually looked back and looked at the... port that we got from Valgrind in hindsight, that would have been enough to figure that. Because there was clearly some call stacks that were like... on the square start somewhere into the... in Glowder, into... LibLzma... didn't look right and... could have been enough to figure out that the problem was there but clearly they didn't. And...

I don't know how many other people had enough... chances to hit this multiple times. Yeah, interesting. If I had a fast... this was a server... CPU, so where the interval cores are a bit slower and I didn't have Turbo Boost enabled. So the CPU usage was really exaggerated compared to my laptop. For example.

Because that has a faster per core performance and... all along we have Turbo Boost enabled or whatever the AMD equivalent is. So that would have been... a much shorter... bike. So I think there was definitely lack involved. How long it would have not been caught, I've... I go back and forth I think. Interesting. Lots of people... lots of very high CPU usage... in... stress that nobody seems to care a whole lot about. Also... SSH is something that I think people are more... tuned to being weirded out by. But at the same time it's also so normal to have all these... random probing attacks... that everyone like in a public IP... press from.

And... I had disabled password authentication. But if I didn't have password authentication, disabled would have... totally been plausible that there... some... servered encryption is taking some time. Like running a lot of cycles and be crypt or whatever. And that that would have been... source of the CPU usage or something. Yeah, interesting.

So I really like... I don't think it would have been years. Because at some point somebody got to have... as I figured it out. But... on the other hand like... what if in 5.6 .3 or 2... they would have actually optimized the code... to only take a few milliseconds. And the story gets... is very different. Not sure that... that would have been... strictly found... many times in. Because like if somebody analyzed... actual intrusion, I think that's the most realistic path... as that they started to... utilize the backdoor.

Speaker 1: And then that somebody has to become noisy. Yeah, yeah. Yeah, yeah. So someone begins to see... So have you read the Cougar's Egg by Quiff Stahl?

Speaker 3: Like, funny enough, I saw a reference like a while ago. Like not... weeks ago. Where I'm like... didn't read... original book, but... I read about it a few weeks ago.

Speaker 1: So it's a great read. And Adam, have you read... I guess we decided that you can't remember... if you've read the Cougar's Egg or not. You and I were talking about this earlier. But the... Yeah.

The... you know, the Cougar's Egg is merit to reread. And in part because that is... just as you're saying, Andreas, the way Quiff Stahl is an astronomer, but is also a computer scientist. And I love his line about... the computer scientist.

Thought he wasn't much of a computer scientist. He was a great astronomer. Astronomer. Thought he wasn't much of a astronomer. He was a great computer scientist. But the... that he... he's in detecting it on the intrusion side. And because the intruder, like, just trips up a little bit. But then he's also... he's following his own curiosity. And I think... I guess, Andreas, that's kind of... dovetails my next question about, like, how... has this whole episode changed you as a technologist?

Speaker 3: I think it might have changed me more as a... person in the world than as a technologist. I always thought, like, important to... understand the system and... about problems, even though they are not necessarily, like, in my direct... you know, in sea chain to whatever I need to do.

At the same time, I think it's important to... I don't think as a technologist, there's that much of a change. But it also depends on what you define a technologist as. If I think... it does make me change my... has made me change my thoughts about, like, project maintainership of it. The threat model is for being a product maintainer. And... when you use to anybody that has done open source for a while, that we need to do something more about supporting some projects that are very crucial, like, it made it more apparent. I think... it has, you know...

I don't know whether it has really changed me that much, but... it's been very overwhelming in a way, is that suddenly, like, I don't know, school friends called me and said, I read about this. And those school friends have zero connection to tech, and that has never happened in that way. Right. That's the... probably being, like, somewhere in a... organization where that is very far outside of what I... ever thought I would be in. So I think what's, like... I wonder whether it's just changing me.

Speaker 1: Well, I think it... but in showing you, at least to me... so I'll tell you that from my perspective, I mean, I think that there's the shine's light on a bunch of different things. I think that one of the things that shines the light on just strictly positively is the... value of digging into an aberrant system. And this is something that I've always believed strongly in, that when you've got these wisps of smoke, that it's worth it to take the time and really investigate it and take that... the no meeting Wednesday or whatever you have, your meeting free day, Andreas, and really digging into this stuff, even though it's like, oh, I've got a deadline and so on. But, like, you know, when you have aberrant behavior in front of you, it may be your last opportunity to really understand that.

And I've always kind of thought of that as, like, it's your last opportunity to understand what is just misbehaving software rather than nefarious software. But I can tell you, Andreas, that you've already had an impact on me because I had a very strange bug over the weekend where we did a Rust toolchain update, and it induced an error that it should not have induced. Just the I-squared-C error talking to a serial presence detect on a dim. And I'm like... and I kind of, like, addressed the I-squared-C issue.

But I'm like, why did the toolchain induce this? And I was kind of... I was listening to some of the podcasts you've been on talking about this, and I'm like, you know, what would Andreas do right now? Andreas would, like, it's like, it's time to dig into, like, what is going on?

We got to completely understand this and ultimately discovered that it's like, yeah, the compiler chain update caused some stack growth, caused stack overflow and an unrelated thing, and drop a link to the... actually, it's all open source. I could drop a link to the ticket in the chat if people are curious on that. But it was a very, like, tangible moment of, like, I think we engineers need encouragement from one another at the... no, it is worth digging into the aberrant behavior. We did the episode with Dave Adam on the Cockroach DB data corruption issue.

Yeah, yeah. And I think that took a long, long time to debug. Obviously, not in a various at all, but very much just in the much more garden variety, but very gnarly data corruption bug. And it... like, engineers need that encouragement that, like, no, no, you should... Like, it's a computer.

It's supposed to be working the way we designed, and you should be investigating this aberrant behavior. And I think, Andreas, I got to tell you, I think you're a real inspiration to people who are at that decision point of, like, do I investigate this further or not? It's like, Andreas would investigate. I think that that is... And I think that there's a real potential to be an inspiration to other engineers in that regard.

Speaker 3: I'm trying to... I'm trying to see you up for your speaking tour, you know? I'm trying to...

Speaker 2: I want to be your... I want to... I want to... There's a movie here somewhere. I'm just having a movie. Oh, for sure. We can get a movie.

Speaker 3: One thing that has, like, made me wonder is, like, I figured that out if I started later as an engineer, because there's just so many low-level systems that are like, knowing how the dynamic linker works and, like, knowing some performance tools that, like, it's hard to get started with if you start out, like, at the current complexity, rather than at the complexity 20 or the years ago. And it worries me with, like, seeing how much more to learn these days to get to a very efficient...

Speaker 1: So, I think it's never too late to learn from first principles, I think.

Speaker 3: Yeah, I read it just harder, I think, than it used to be.

Speaker 1: I think, I'm glad you mentioned the tooling, because you used Perf to debug this. Obviously, Valgrind was very important. I mean, our tools are really important to help us understand our software. And it feels like they played an important role here.

Speaker 3: Yeah, and, like, interestingly enough, I... I had used to use Perf, not just a normal Perf, but, like, Intel PT processor trace. Yeah, yeah, yeah. Because it actually... Like, normal Perf isn't going to be granular enough for this. And the problem was that the profiler... like, the debugger didn't reliably work because they have some anti-debugger measures. So, the... like, being able to use that processor trace feature was... it could be powerful. And I've used it, like, a handful of time before. Pretty sure that I'm going to use it more now, because it's being able to, like, see the exact... path through a program at very low overhead in execution time. That's really powerful. And I think I was a bit surprised how little tooling is around it.

Speaker 1: Well, this... you know, we've discovered this, though, a lot about a lot of different aspects of detrace that sometimes when there are things that are a bit esoteric, but when you need them, you really need them. They're just not as broadly known as you might think.

I think they should be. It sounds like this might be an example of that, but obviously critical to help you kind of crack this case and do it relatively quickly.

Speaker 3: Back to the question earlier where you said, like, where they changed me as a technologist, I did think now of something that used to be way more suspicious of, like, soft security boundaries.

Yeah, okay. Now, I... really mentioned that more, like, things that could give you just, like, overheads up that something is going on, they're more like ease. It is that stuff is getting noticed. And given that we realistically weren't able to vent this kind of attack happening, there's just too many products and too many interdependencies, I think these kind of pretty boundaries that were, like, that aren't really hard to get over, like, add ability for noise, I think that's more powerful than I believe before end.

Speaker 1: And so, like, what do you mean, like, what are some of those boundaries concretely?

Speaker 3: If, example, that we can tell the linker these days to resolve all... the dynamic linker to resolve all symbols at program start, and because of that, we can remap the global offset table read-only. At that, they wouldn't have had to rely on any of this complicated machinery. And these shows have started to default to dash z now dash relver dash c relver for a while. And that's fine for long-running programs. It's actually not great for short-running stuff, because, like, most of the time, they're not going to use all the dynamic symbols that they were fencing. It actually adds overhead to the common case. Yeah. If they hadn't had to jump through all these hoops, it wouldn't have been, let us think, the complexity of providing, like, overhead of, defaulted to the overhead of this protection. I was more suspicious beforehand.

Now I'm less. And I think there's other kinds of, like, that, like, on the kernel level, doing more, always a bit of anoint about, like, mitigation measures that cause the pu overhead, because that's, I see, in the process. At the same time, I, like, a bit clear why it's perhaps worth doing that by default.

Speaker 1: Yeah, I know. And I feel like on some of these spectrum mitigations, in particular, it's like, do we have to, like, what are we not going to have speculative execution at all anymore?

I mean, I know what you mean. There's this, then what about the, someone asked in the chat about FNOW, that frame pointer, which did play a role, kind of a strange role in this, right? Because, yeah, you described the role that that played in the second.

Speaker 3: I don't think I understand the backdoor code quickly enough to technically, precisely point it. But my understanding is that it assumed a stack layout that have been the case with, without the frame pointer there. And you start to figure out, like, jump to fish stack, fish information from the stack somewhere, because I was compiling with FNOW mid frame pointer. And, and, like, the access to the wrong offset.

And because of that, there was the Morgan one warning. And I was seeing that because I was, have been running my test with FNOW mid frame pointer, just because it makes profiling easier. And after doing that for a while, and like, but it was the reason that it was noticed on Fedora was that they just changed the default to it. So I think that was also made it, like, easier for somebody else to discover this, because, like, 5 to 6, 6 to 0, on new Fedoras was suddenly, like, failing. Yeah. What you can see so far, nobody had picked up on that being a problem, and said, like, this warning was silence, just like I had silence, the warning initially. I think that this perhaps helped figure this out.

Speaker 1: Which is great. I mean, as, even though I don't like the double negative of no mid frame pointer, I definitely, I'm very pro frame pointers. And I do think it's interesting, Andres, for people to hear from you, that someone who takes performance very seriously and is looking at micro optimizations has frame pointers, because it is impossible to understand your software without frame pointers. Please use your frame pointers, or you need to.

Speaker 3: Yeah, I don't necessarily use it for an optimized build. I use it to say, like, it's noticeable, the overhead. I think it's totally worth the overhead, by default, in most programs. I'm not sure that every program needs to run by frame pointers in an optimized build, because it does.

Speaker 1: I, this is where I see it. If you don't have a frame pointer, inline it, which is what Russ does for the, the, so we're, but I think it's kind of a cancer-gill-in-stern kind of a role in this kind of FNOMIT frame pointer, kind of wandering by having an impact on this story.

Speaker 1: Well, Andres, this has been amazing. And we really, really, really appreciate you coming here. I know you've had a lot, I mean, obviously you're talking to the New York Times. You've got the, you're a celebrity now, rightfully so. You've got Satya praising you as you should be. So I know that you've got, and I know you've got a national speaking tour scheduled with Oprah and, you know, I know you're going to be here. So we really, really appreciate you coming on, coming here. It's been so great to get your perspective. This is such an extraordinary story in so many different dimensions. And obviously, you know, your role here is a real inspiration to anybody wondering if they should dig into that aberrant behavior, try to understand what's going on. I think you can really serve to inspire other technologists. So thank you very much.

Speaker 3: Hope I have destroyed too many, Boris, in a curious.

Speaker 1: No, it was great. And thanks everyone for joining us. And then Adam, I know you've got a national championship you need to run to. So we're going to let you do it. Adam is a Connecticut native.

That's right. Big Huskies fan. Big Huskies fan. You know, Connecticut having its day because, you know, you said Connecticut Yankee, King Arthur's Court, I always think of whenever there's a total of corruptions there today. So I feel this is a big, big Connecticut day. Big, big day for us nutmeggers, exactly.

Big day for the nutmeggers. We'll let you get to it. Just a reminder for folks that, or actually maybe not a reminder because we may have not publicized the date for the club discussion. Did I neglect to do that?

We have not. That's right. This is going to be the moment. All right. So on May 13th, so May 13th, we will be discussing how life works by and this is hopefully folks have, and you've started to read this or you found the audio book.

Speaker 2: I have started. Yeah. I started. No, there's no audio book. I was always planning to read. There is an audio book available only in the UK. I've discovered. I'm sure folks who need it can pirate it from their UK friends.

Speaker 1: The, but I'm really enjoying it. Adam, I hope you're enjoying it too. It's definitely, I think it's going to be a totally different direction for a lot of folks, hopefully in an interesting way.

So that is going to be on May 13th. And we're going to have my friend, Greg Cost, just going to join us and he, he's going to read the book, which is great. He's a microbiologist. He's like, I got to warn you, like I've read the blurb. This book, I, like, I'm, this book might be upsetting to me. Is it okay if I'm like, I have to like the book?

I'm like, no, definitely not. Like that's even better. Like come in hot. Like, well, I will invite Kevin Russo.

Treat him as a punching bag as well. We'll, we'll, so anyway, hopefully folks, if you, if you haven't started reading that, definitely read that. Join us for discussion or you'll be able to catch the recording there. And, and then I think we've got a, we've got a wild episode coming up in a week. Adam, I'm going to tease that a little bit. You'll look for more details. We've got a big, big crossover episode coming up next week.

Speaker 2: Stay tuned. Yeah. Very excited.

Speaker 1: Stay tuned. I am extremely excited. Probably a little too excited for the episode. We're going to have next week. Yep. Thanks again. Amazing story. Thank you everyone. And until the next backdoor, take care.