Giving back - how I cleared L6 System Design - Part 1
Long term prep strategy Part 1 is out: https://www.teamblind.com/post/YA2RKEqY
Coding Part 1 is out : https://www.teamblind.com/post/ikvVrRXa
Part 3 is out : https://www.teamblind.com/post/qubF6fS2
Part 2 is out : https://www.teamblind.com/post/rBrt5bV8
Edit 6: I will be posting the rest/commenting as user flung2 from Red Hat going forward.
Edit 5: I know I promised to reply to every comment/question but I left it at 90 comments and now it's 450 and grows faster than I can reply. I'll have to be selective. Sorry.
Edit 4: I will be posting the links of part 2 of SD and future coding post here.
Edit 3: "This is booky knowledge and not real life experience stuff". DDIA is condensed experience. Nishtala et al are as close to *relevant* real life experience as it gets. In fact, I had the feeling some of the experience of my interviewers was irrelevant experience in scaling glorified CRUD apps which had left them stale on knowledge. It's a balance, but I stand by my hints here if your goal is to clear the SD interview. This method got me 3 SH and 1 H in F/G interviews so like it or not, it gets the job done.
Edit 2: Some people feel this is too much detail and you never get asked these things. Wrong. It's all up to you to steer the conversation. The question is pretty vague. The key is to identify unassisted the main pain point(s) of the problem and once you do, dive as deep as possible after you cover the high level design. It is not uncommon that the detail you waved away turns your high-level design to dead end. Everything that I mentioned as examples here were actually brought up in my interviews. Less preparation could get you there but, as I mentioned, I go for the jugular.
Edit: Throw your questions in the comments. I'll get to everyone of them eventually, I promise. DMs also are welcome but it needs to be something you can't put on the comments.
1. Intro
I spent 6 months preparing overall. In the evenings before going to sleep I'd spend some time on Blind to get my hopes up and see a joke or two. There are some assholes here and there on Blind, but the majority are good people. I love you all. So now that I'm 'across the river' I decided to give back and provide some hints specifically on the System Design rounds.
I interviewed at Amazon, Twitter, Google, Facebook, Databricks and Elastic for L6 (or L6 equivalent). I cleared them all with exception of Twitter. I cleared Google and Facebook first then had 'emergency' interviews with the rest of the companies to ensure I was well-placed for the inevitable lowballing dance. One Friday I had half-onsite with Elastic in the morning and half-onsite with Twitter in the evening. I completely messed up the System Design for Twitter and that killed my prospects. I regret that though, as I liked Twitter.
I am not a genius, I'm just a hardworking tenacious guy. Don't let anyone tell you you're not smart enough for L6. It's all about mustering the motivation, channeling the work and executing it well.
I made the decision to try 6 months ago, and I prepared throughout that time while being at work. So my preparation hints will suppose long-term prep. If you're expecting a two-week crash course this is not for you.
I'm conscious some of this intro will read like a flex - it is genuinely not - but I'm sure the context will be helpful to many so I decided to make that tradeoff ;)
2. Content
Where do you start? Get the Designing Data Intensive Applications book. This will fill your theoretical gaps. Read it slowly. *Do not make fake progress*. If you don't understand something stop, use the references, research the subjects, get out of the book and back. There is *nothing* useless for you in that book. Nothing too much. It's all for you cover to cover. Properly grasping that book is half of the whole work. Part 1 is super useful to teach you how to pick data stores. Part 2 will dispel your fears of sharding and choosing a replication mechanism. Part 3 will give you a full idea on how to piece a big system together from smaller systems. The separation of System of record and derived data is key to understand there.
When you think you're done with DDIA, if I wake you up in the middle of the night you should be able to explain to me how LSMT-based databases use Red Black Trees to keep a sorted log in memory and why they do that, and then get back to sleep. I'm serious.
Next up, pay for Grokking the system design interview.
I know that Grokking can be too shallow. Nevertheless, I want you to be able to recite the solutions in Grokking down to the smallest details. Details are more important than the big picture for L6. Do you think your done with typeahead suggestions? Did you mention exponential moving average on how you weigh results? No? Failed.
You can leave the pastebin and bitly and that easy crap uncovered, but I'll need you to recite the other grokking solutions to the smallest details. Take extra care on the geospatial problems. If I show up at your lunch and ask you about quad trees I want you to be able to estimate the index size on the spot. You can't? Not ready yet.
Next up: videos. InfoQ and @scale videos about real life systems. Look up Nathan Bronson and suck up all his videos on Tao, Caches etc. Look up Nishtala and the memcache video. Look up Kulkarni and the Facebook live video. Look up Susheel Aroskar and his Zuul push video <- ultra useful.
Again, never make fake progress. Take this last one: Aroskar's Zuul push. He mentions Apache Netty. Read up on it, understand it. Go deeper, understand epoll and its red black trees. Go deeper and understand the TIMED-WAIT trick in TCP protocol which saves some web sockets connections. Another thing: he mentions some load balancers can't handle websockets. Why? Go figure it out. I did and I impressed my Facebook interviewer as I went deep until I was stopped as the interviewer lost it. L6 means depth, depth, depth. If you draw me a block that says 'Server' and leave it at that I'll slap you back to L3.
Now I went the extra mile. But I wasn't targeting barely making it - I was targeting mind-blowing performance. If you want the hardcore stuff, let me know and I'll give you content for that as well. If you're up for anecdotes, a Google interviewer challenged me on Paxos when I mentioned Spanner to him. I drew the multi-Paxos detailed flow for him with estimated latencies supposing a 5-replica configuration with the pivot replica on north-east US. He smiled and said 'ok' (Strong hire).
Next up, the Google SRE book. Did you say you're not interviewing for SRE? Don't care. You don't need it all. You do need the chapter on Non-Abstract Large System Design. You need to be able to recite that in your sleep. Especially the estimations part. You don't have NALSD interview? *rolls eyes* Don't care, learn the NALSD chapter or fail.
3. Speaking of Estimations
Ultra important. If you can't handle these, you're most likely ducked. How do you prepare for them? Practice power of 10 calculations. See how the NALSD chapter does it. Practice with fake numbers. Drag the units with you and reduce them when dividing/multiplying.
When calculating storage, consider cold storage. Also consider e.g. cassandra needs ±30% free space to do compacting (see, I told you to properly learn LSTMs in Part 1 of DDIA), also keep in mind that only ±85% of the disk space is usable (OS files, formatting tables etc.), also keep in mind that 3-5% of disks die in a year so account for that, also keep in mind that you need to multiply by replication factor, also keep in mind that cassandra says no more than 2TB disks otherwise it gets slow.
Have a strong grasp on the -bound concepts. Is something storage-bound (Log dumps)? Is something cpu-bound (live stream encoding)? Is something RAM bound (in-memory timeline serving)? Once you grasp those you design tiers with separate scaling techniques. That's why you need numbers, not to show off your math but to figure the -bound part and then decide how to scale a specific tier.
I'll wrap up this part here. In the second part I'll get into some sample walk-throughs of some popular questions. Then I'll give a you a detailed plan on how to spend the 45 minutes. I did 9 System Design interviews in the space of 5 weeks so it's fresh in my head.
Now you might think this is a lot of work. It is. It depends on how much you want the job. Want it badly? Throw in the effort then, that's all it takes. Was it worth the effort for me? Yes.
Old TC: 175K USD, New TC: 593k USD
#SystemDesign #Interview #L6 #E6
Giving back - how I cleared L6 System Design - Part 3 
teamblind
comments
Unfortunately, you can’t replace real world experience with just a book.
Me writing here how to do it will not build up your experience, and that is what I’m looking for.
d3doo - and it is ok. That is why there are levels that are different in different companies. I was top level in a small company and got severely down leveled when I joined FAANG and it was fair. I learned, grew and now in a much better place.
And if you are not working and exposed to those problems than you are not qualified to work on them. Let me rephrase - I’m a doctor who really want to do brain surgery but I’m not exposed to it. I will read a book, get hired as a brain surgeon and get great TAC.
Would you want such a surgeon performing your surgery if needed?
I agree with you that for senior positions or higher depth plays a significant role. However, one would get more bang for their buck if they grasped big ideas, and improvised during the interview with the disclaimer to the interviewer: “this is not my expertise, but this is how I would solve it...”. Otherwise, it will be suspected and then a trust issue may arise.
For people who have 1 year or more of time to prepare, I would advise to get involved with a hands on project. Build your own app, server, database or whatever and you will learn infinitely more when it’s coupled with theory. For those who don’t have a lot of time, it’s a waste of time trying to specialize in a particular domain. You should probably go breadth over depth, but that’s exactly why I was saying understanding the big ideas is more effective.
Big ideas won't cut it for L6 in my experience. You need to be able to talk the talk on the high level and then walk the walk on the main pain point(s) of the problem. In walking the walk you need a wide variety of weapons, some acquired through experience and some through knowledge, to tackle it.
The proverbial devil in the details and your intuition skill in getting those right is what differentiates an armchair designer from a real life one. I do not believe an interviewee should be looked down upon if they gathered that intuition and skill through studying and knowledge of others, rather than own experience. I believe they should be equally rewarded, encouraged and cultivated.
You seem smart, but the job at the L6 (L7 at Amazon) isn’t just about being technically smart, it’s about leadership. If you haven’t come from a role where strong leadership was required, I’d try to focus on that a bit. It’s unfortunately hard to do with just books.
The principle level you’re talking about isn’t really for the most part about solving hard problems yourself, rather enabling others to solve those problems. Doing that takes experience. Keep this in mind when you join.
Any suggestions on all papers you read?
I just ran it on my local laptop, no containers. It got messy, but I got what I wanted and then wiped the whole directory. I still remember the fan sounding like a tornado while I was load testing it ;)