So I was watching the scaling instagram video on youtube and also working through the system design questions. I am not able to confirm how does video metadata work for global audience. Scenario: Video uploaded in US Video viewed in Entire world. Now for better availability and low latency of comments and likes every users request go to closest servers. But a video’s metadata will reside only in 3-5 data centers so how do you stores likes and comments for specific video? If the database is deployed in multiple regions, writes usually go to master isn’t it? If one region has problem for e.g. UK then all videos owned by UK datacentre will be unavailable? Or they continue to work in different region and users can interact with them? Couldn’t find any talks or blogs discussing this issue. Please point me to one if you know. Or Maintain local database in every regions and occasionally sync all databases? Pretty hard in case of comments and replies to comments. Or Every region is deployed as a separate stack for availability and transfers the call to region which owns the video? Does country specific regulations also come into play here where video from china should have all its data in china? #tech #systemdesign #availability #interview
Read more on CDN. Content delivery network. Will get your answer
Cdn is read only tho, how do you scale writes
Cdn can serve the video or cache the video metadata but when you like the video it still gotta go to application
You are describing the CAP theorem. Given network partitioning, you can optimize for either Availability or Consistency. In case of video metadata you would opt for availability. If the numbers (view/ like counts) are stale they are still indicative.
Yes but i am not worried about correct like count. Incorrect is okay but eventually system should sync all comments and likes. that is what i am wondering how does that work, you will keep your like api available in partition and when partition recovers you need to sync
I can think of multiple ways of dealing with this. If you are using something like Cassandra where have a quorum for reading and writing, you tolerate failures by relaying on the quorum. When a node come backs up, if it's out of sync with the other nodes when polled, it can be updated at that point. If you have a dedicated master for writes, you can add a queue in front of it. So if the master node was temporarily unavailable, when it comes back up it can just continue processing the queue to make sure it is caught up.
I can imagine the writes going to the master while each server has a read replica.
Master is prone to partition and like api will go down globally
Writes will go to replicas and will be pushed to master in batches. After the push to master replicated data is re-synced with master.
Is that supported by nosql db like Cassandra natively, or is it custom app logic
Even if it is supported, the wisest thing is to write a custom logic for it. An analogy for this is the client sync up mechanism that drop box uses. Not exactly similar implementation but similar idea
I think you have a couple of different options here. Single leader, multiple leader, leaderless replication. For counting the likes you can use distributed counter as others mentioned. You keep multiple counters per region/zone and they sync with each other. Each counter instance stores also counts of other instances. (something like vector clocks) When they receive a like it increases the value in its own bucket and send the snapshot of the counter to other instances. When you receive a snapshot you compare it with your local instance and pick the one with the max value. To return the total count you sum each value in the counter vector.
TC or …