L6 SWE System Design on Analytics
Hey my fellow senior engineers, What are the best resources for understanding how to design stream processing and analytic systems? Big data & data warehouse systems. This design works for designing systems like Amazon's product recommendation service or YouTube's video recommendation service. YouTube is not doing me justice. I've watched codekarle go over Apache Kafka + Spark Streaming + Hadoop + Cassandra in some of his design architectures, but he never goes in depth on what exactly happens end-to-end. Other system design videos go over Kafka and Spark separately, but I cannot seem to put everything together. Please help!! Thanks! #systemdesign #softwareengineering
I wouldn’t worry about complex architectures. Watch the videos, but start small. I was once working with a team that was trying to work with 50 gigs of data and were trying to design spark infra around it. I told them, spin up a large ec2 instance write a Python script, load everything into memory. Profit. You’ll spend 1 week and can work this solution for the next year , till you prove your business value.
Get udemy courses for each one. Read databrick and confluent blogs Check the companies behind open source like pinot, flink etc. Infoq has videos on this, lyft had one on yt
Got links? Thank you for your response