Job Talking Points

Trigger: WHAT ARE YOUR STRENGTHS?
Narrative: I know how OS and Kernel stuff works e2e. But I'm also primarily good at understanding problems in the low-level space and translating solutions in that space to applications in product development and growth hacking and team building in the context of the larger company.
Expanding: I was effective at juggling the expectations and needs of various classes of stakeholders: external ISPs, internal software developers and datacenter contractors. When I drove this effort, WE...
i.e. I saw an opportunity to vastly improve the OS distribution pipeline in Facebook Traffic and turned it into an XFN effort to publicize my team's accomplishments and spur the growth of our junior engineers.

Junior engineers are plants - they respond to sunlight. Mid-level engineers just respond - they define technical work and related impact. Senior engineers drive projects with e2e ownership over a problem space. Staff engineers clarify the core narrative underlying this.

Questions:
- At FB, we had this idea of "big bets" - large, concerted efforts between organizations to get something done. I.E., an in-house CDN platform. What are ___'s big bets? Roadmap items that folks are excited about?
- From my time at FB, I have grown accustomed to this hi-performance, bottom-up, work-hard/play-hard engineering culture, which had its own suite of benefits and drawbacks. How does your company's engineering culture compare/contrast?
- What's something you really appreciate about working at ____? And, what's something you wish was different?

"How was your day?" "Good! I spent a good deal of time today thinking about how to encapsulate the time I spent at FB - I'd love to have your feedback on the narrative I share with you. I was a PE on the TPE team as part of the larger Traffic org, and I thought about the security, performance, and scalability of FB's edge platform every day for 4.5 years. I spent a lot of time tackling super low-level problems, and I took these challenges as opportunities to grow my team, publicize my team's accomplishments, etc. I became quite adept at translating esoteric low-level OS context into something that various stakeholders, including ISPs, software engineers, and network contractors, could understand and empathize with."

"Think about a time you encountered something unfamiliar at work"
- "When I engage with a subject that I'm relatively new to, I like to observe and emulate masters in that subject. When I joined FB, I had intentionally made it one of my criteria that the team that I would select during bootcamp would have a high degree of tenure - time spent at FB - because then, I could emulate them, and hopefully transform myself from someone coming into the industry into someone who could be considered masterful in the industry."
- "There came a time when I was still a junior engineer that I bore witness to a great amount of churn on my team. Most of the senior engineers had decided to either take the next step in their careers, hop to a different internal organization, move back home for personal reasons, etc. This all happened in the span of about 3 months. My manager had sat down with me so we could brainstorm ways to mitigate this issue."
- "So I picked up a great deal of the work on my team, acting as captain for the projects that I was the SME for, and steward for the projects I was not. I also built a speedy yet comprehensive 4-week onboarding module to ramp up all of the new hires we were going to make. For the next year, I was almost always mentoring prospective hires, peering new hires, managing interns, etc, while steering the course for my team's roadmap. Many of the hires we made in that period became the new faces of the team as they took the projects I scoped out for them and really ran with them.
- "I also did bootcamp mentoring, bootcamp coaching, intern managing, and PE MLH scholarship mentoring."

TCP SACK Panic
- First oncall. Saturday night. Playing videogames. I get a call from a director asking me if I have time, otherwise they would go up to a more senior engineer.
- I get put in this war room chat where I find out that the director was having lunch with some buddies from Google...
- Netflix paper. TCP sack panic. Sequence of TCP packets that kernel panic your machine in 30s. Bad! Google patched their fleet.
- We spent a while trying to build a repro (reproduce) the panic.
- Then we spent a while iterating over kernel patches to mitigate the problem.
- Then we released the patch to the fleet ASAP. I did this quickly. This was bad for services for many reasons.
- This led to the Release Cycle project.

"What project are you most proud of"
Release Cycles
- When I joined the team, the Release Cycle for our custom OS (EdgeOS) was 30 days. 2 months if you consider the time for planning, building, validating, etc.
- We essentially had a bash script: a for loop iterating over all our servers across the world, SSHing in, dd'ing our build image to disk, and rebooting it without warning.
- This was bad. Draining production servers is important. Retrying failed SSH connections is important. Monitoring the health of servers and clusters and regions is important.
- I learned these lessons while writing the automation to replace that bash script, one of my first tasks on the team.
- Eventually, I realized through experimentation, that performing destructive maintenance on a machine at peak was very bad. Traffic redistribution mechanisms can't respond that quickly, even if they have an hour warning.
- However, taking down half of a given cluster, no matter the size, was fine, as long as it occurs during "off peak hours" i.e. the middle of the night wherever the cluster is in the world.
- My approach was: for a given cluster, look through its past week's peaks and troughs. Predict the next trough, publish that data somewhere, and schedule maintenance for that time (another problem).
- I iterated through a few different trough detection algorithms, but when all was said and done, our time to release a new kernel or initrd (assuming no egregious perf issues) went from 30d to 2d.
- This project also involved the whole other problem of regression detection. I ended up classifying everything that could go wrong by type of regression (system metrics like CPU/memory/IO util, network metrics like throughput, service metrics like cache hitrates, etc).
- Why 2d and not 1d? The major problem with destructive maintenances was not the actual maintenance event (1-2h, for various reasons). We tried to get that down, but the issue was cache hitrate recovery time.
- We designed our OS as a blank slate, because servers were in physically insecure locations. More technically, user data was stored on an overlayFS layer encrypted with a key stored in kernel memory (volatile).
- So the cache was wiped on every reboot. We had no infrastructure to restore it securely (another project). So we only considered a cluster "recovered" when its cache hitrate rose back to normal levels.

"You realize your team is sacrificing security/reliability to release a product, how do you bring this up"
NIC Investigation
- Let me preface this by saying that this was a long, complicated investigation into the Linux kernel network stack, but it eventually resolved in the prevention of a class of security CVEs through 2 small config changes.
- When a network packet comes into your machine, it gets received (RX) by the kernel as a softIRQ event and placed on the queue of one of your softIRQ cores. Maybe 16/100 cores are running this heavy load on network bound fleet.
- Around the time I started on the team, I solved one issue of high CPU util (redistribution wasn't fast enough) by upping the number of softIRQ cores from 16 to 32 across the fleet.
- IOMMU is a kernel cmdline setting that establishes a virtual memory mapping between PCIe devices and machine memory.
--> Demonstrate that there is a problem. Reproducible data does not lie.

hackNY
- I have been a Fellow in 2016, Mentor in 2017, Mentor in 2018, Executive Director in 2022, and Admissions Director in 2023.
- As a Mentor, I built community and resolved conflicts, and boy were there a lot of conflicts.
- As an Executive Director, I rearchitected the roles necessary to run the fellowship, documented everything we did, runbooked all of our built-up knowledge, and ran ~20 mock interviews for fellows over the summer.
- As an Admissions Director, I executed on my plan to improve the admissions process. Now, with the infrastructure I've set up for next year, running admissions will be little more than clicking a series of buttons and managing spreadsheets, when the executive board gives the go-ahead.

"Tell me about a time you disagreed with someone"
- hackNY Board: Executive Director or Community Directors
- FB: NIC Investigation (security vs. ISP throughput)

Strength: hyperfocus, never halfheartedly ship something, love responding to chaotic events, bringing order to chaos
Weakness: need a support network to be productive, need interesting projects to keep my interest

3:00pm - 4:00pm ET: Hai Ngo - Linux Round
4:00pm - 5:00pm ET: Jim Osial - System Design

RELEARN OOM STUFF AND VM
more about file systems: LVM
WIRESHARK
likes: large infra, dislikes: working with ppl o.o

TikTok design - Distribute an artifact to 5k servers in 1 DC with 2Gbps ToR and 2Gbps per server

FUN SEVs in my time at FB:
- TCP SACK Panic due to kernel bug
- L7 LB OOM due to tmpfs
- Global DNS Bounce due to backplane maintenance

maturity
- level 0: no infra
- level 1: hypershell on dev server
- level 2: script to reboot all machines
- level 3+ having solved all of those issues

- how did you help someone who was struggling?

1. Business Problem / Desired Outcome
2. What did you do?
3. How did you know that was the right move? Did you?
4. How did you accomplish it?