Reading about large scale system for an interview is one thing. There are a few good resources like grokking the system interview, hiredtech, designing data intensive applications, etc. What if I have not worked on large scale problems in real world and would like to experiment/learn say building an app with something random as in here. 1M writes/sec Spikes upto 3X during certain intervals A search query NoSQL data store And some fairly intensive computation for every query Can be anything - ecommerce, sports betting, stock trading apps, let's assume the requirements are almost the same across the board. Is there a playground like leetcode? What is a good book/tutorial with an end to end scaling solution in a step by step approach? I'd like to build out the whole thing. Obviously Azure or AWS can help me stretch the scale, is there a better way to test the scaling and identify the bottlenecks in an "experimental playground" without high bucks. I'm ok to pay of the order of a monthly Leetcode subscription for this experiment. Trolls please excuse. TC: Around the Midpoint of 2^17 and 2^19
Yeah I have been thinking about this and here is my plan 1) create a docker image of the code I want to deploy 2) get images of Cassandra 3) scale my app until o start hitting limits of container 4) then try using more container s of either my code or Cassandra to try scale horizontally if something fucks up try to come up with a a new plan
Good start!
Is that the geometric or arithmetic midpoint of 2^17..19?
I wish it was arithmetic ;)
To save everyone's time Geometric = 256k, Arithmetic = 320k.
I don't see how 1M writes home lab will help with the large scale design. You often have to deal with less writes, but in a distributed micro services environment, with the goal of not having your micro-service becoming bottleneck, or page alarm generator. Unless you are writing a distributed database from scratch, then 1M writes is a great benchmark.
Ahh okay. I just wanted to build out something. I cooked up the requirements - usually once I'm done satisfactorily in a system design question in terms of breadth - the follow up is how do we scale up to 1M writes, 50M customers, PB of data ..... Now you see, why I wanted to try building.:) The whole point is I want to learn how to scale out - and willing to do what it takes.
When your system does 1M writes, that's because it was built to scale horizontally. It's not a single host that is doing that much writes. So what you need to do is to get experience building systems that can scale horizontally.
How and where do I start for a real world DIY project please?
That many reads and writes you are not gonna get for monthly leetcode price. My advice is to keep making more free tier accounts for AWS/Azure and building on those.
Thanks, how much do I have to stretch my budget to build at that scale?
Depends on what exactly you’re building and how often it’s up and running. Most of aws is billed on usage. Just build your app, run tour tests and see how you do. Do you really need 1m simultaneous read writes? Or can you test with 10k, then 100k, and extrapolate performance based on that?