Working on Consensus algorithms
I have a question on scaling services in the cloud. Currently, the Senior engineers in my org are involved in designing Multi Region Architecture for our stack. I have read the design docs and couldn't find any reference to distributed consensus algorithms. How often do engineers get to work at this layer - I have read some algos during Grad school (a year ago) and found it very interesting - Raft, Chubby. I want to contribute to the design and learn in the process. Any suggestions how to approach?
Probably unlikely. Most of that stuff has been implemented by really smart people and we just use that to deal with distributed problems. It also depends on how high you are in the ladder. They're probably not going to let a new grad work on stuff like this.
I don't think you'll ever write Paxos or Raft in your career unfortunately. You'll probably use a lock service to coordinate (you mentioned Chubby but that's proprietary Google technology) like etcd or Zookeeper. As much as I'd like to write Paxos from scratch, I don't think it'll ever happen in my career and would have to be something I would do on the side. Please tell me I'm wrong and tell me where I'd get to write cool distributed algorithms.
I was fortunate enough to work on them back in 2011. Hardening these services takes a long time and people don’t rebuild these primitives unless absolutely necessary
Look at what’s being used for the persistence layer and that’s where you’ll find what consensus algorithm is in use.
Paxos.
Yuck! Go for raft.
Raft is the future 🙏