System Design: How to handle concurrency?

Amazon
bejeezos

Go to company page Amazon

bejeezos
Jun 16, 2021 43 Comments

I have little sde background, so apologies for noob question. I was recently asked in a System Design interview to design a system that will get hundreds of thousands of hits within a short time span (think multiplayer online short video game). The main challenge is to handle concurrency issue. I gave a standard answer, "have multiple application servers and sync server with replicated cache and write locks on db". But interviewer was disappointed and also mentioned it in the feedback. I'm curious to learn how to actually answer questions like this. Could some Blind ninja please spare a few minutes to answer?

Edit: You're supposed to design a game where there is no limit to the number of users (open to make assumptions here). Each player is looking to uncover treasures on the game map. So each player should also get a consistent copy of the game map and current state. This is not a turn-based game.

p.s. I understand that grokking course and that book and those YouTube channels are the best way to learn. But I'm looking for keywords for this answer so I can directly read those topics and be able to get this out of my head.

Tc: 280k

comments

Want to comment? LOG IN or SIGN UP
TOP 43 Comments
  • Google / Eng
    muñañyo

    Go to company page Google Eng

    muñañyo
    Just some ideas:

    - async/await (I.e. event loops) to handle incoming requests
    - message queues to decouple critical actions from non-critical actions
    - use distributed data layers (e.g. redis, cassandra, spanner)
    - consider alternatives to threads such as Java Fiber or Goroutines
    Jun 16, 2021 2
  • The game client maintains a local state of the environment and a persistent connection with a server. Depending on the game all players can fit into a single server, or can be partitioned appropriately.
    Clients update state changes at regular intervals which gets written to an event stream. Many player moves are non conflicting and can happen concurrently. Those that are conflicting, needs to be totally ordered and causal. The client side logic can either be pessimistic or optimistic when dealing with conflicting changes. The state changes are persisted as an event stream, that allows for fault tolerance.
    The server determines the actual state in presence of conflicts and writes the updates to another stream that needs to be fanned out to clients.
    (I have no idea how multiplayer games are actually built)
    Jun 16, 2021 5
    • Bloomberg / Eng
      b!!!!

      Go to company page Bloomberg Eng

      b!!!!
      Agree!
      Jun 18, 2021
    • New
      Nitp

      New

      Nitp
      Let’s say there are 1000 clients connected using web sockets on 10 front end servers. All the clashing events are sent on a distributed message queue (like Kafka) which maintains order. From there it’s picked up by a service and then fanned in the same order to various clients. Clients ensure that they process ordered messages as relayed by the server based on some sequence ids. Will this work ?
      Nov 25, 2021
  • Amazon
    mush0n

    Go to company page Amazon

    mush0n
    I highly recommend also reading "designing data-intensive applications" by Martin Keppelmann
    Jun 16, 2021 5
    • Uber
      AryanIndia

      Go to company page Uber

      AryanIndia
      Two chapters are titled specifically for this problem. The words you’re using are trade offs within that problem space. You should read the book
      Jun 17, 2021
    • Uber
      DollarSign

      Go to company page Uber

      DollarSign
      Read it a couple of times. No where does it say how to get high availability “AND” high consistency. It just hints at picking one over the other in different scenarios and that is basically just the CAP theorem. That missing part is what Google Spanner solves.
      Jun 17, 2021
  • Uber
    DollarSign

    Go to company page Uber

    DollarSign
    Read about Google Spanner and FB Live commenting design.
    Jun 16, 2021 2
  • Rubrik
    aYbhGyd

    Go to company page Rubrik

    aYbhGyd
    How will you ensure members of one game are mapped to same instance?
    What if they disconnect for a while and join again?
    How will you check if someone have left the game?
    Jun 16, 2021 4