Sharding for HOT users

Jun 26, 2018 15 Comments

Suppose we have to design Instagram. We do sharding based on user ID. How do we handle it for HOT users like Brad Pit for 100s of followers?

PS- Practicing System Design for Google

comments

Want to comment? LOG IN or SIGN UP
TOP 15 Comments
  • Yahoo 7331er
    caching
    Jun 26, 2018 6
    • Yahoo 7331er
      now we know why oracle is going down
      Jun 26, 2018
    • Oracle / Eng Madmaxf
      Lol yahooooo . Great going
      Jun 26, 2018
    • Yahoo 7331er
      at least we work on serving billion requests a day whereas u work on tooling.
      Jun 26, 2018
    • Microsoft BarFoo
      People still use Yahoo? Interesting.
      Jun 26, 2018
    • Oracle / Eng Madmaxf
      @yahoo stop liking your own posts . Lollllllll
      Jun 26, 2018
  • Amazon / Eng G00G-BOS
    If Brad Pitt is a relatively infrequent poster, then I’m not sure what you need beyond simple a simple write through cache for holding post metadata and the image itself being cached at various levels (CDN, server mem, load from s3). Maybe you could have a fancier cache to always cache posts of popular users but there’s something to be said about how popular a user really is if he can’t even stay in your lru cache.


    If you need to send notifications to all his followers or something, that’s a bit harder assuming there’s maybe 2 million of them. Normally, You’d queue up a notification saying user x has a new post and some worker picks it up and sends push notifications to all the users followers, but not quite sure how a prod system would work. Maybe yet another layer of indirection where reasonably sized sets of notifications are bundled up and sent out to another cluster.
    Jun 26, 2018 0
  • Uber lLjv00
    Google won't ask about Instagram, instead they'll ask about designing Google Plus Photos where the hottest person on there really is brad Pitt and he literally only has 100s (not millions) of followers on their clone app...

    With that out of the way there are tons of features for this app to design, but it sounds like you are interested in alerting users whenever brad posts a new photo.

    I doubt all of brads followers use the app all the time, only the most active users will really care to see his posts, everyone else who follows him logs in maybe once a day to once a month or even once a year or never. On top of that even those top users will only care about the most recent posts so a post from a day ago becomes irrelevant if he's posted 10 times in the past hour. So one approach could be in the case of a hottie like Mr Pitt to make the app really fast for top users and really fast for most recent photos. And slow for everyone else and older posts.

    Reading from a cache is really fast, but storing data in ram is expensive.

    Sounds like this becomes a problem of decided who's news feeds are cached and who's aren't? Which posts are cached and who's aren't?
    Jun 26, 2018 0
  • OP
    Thanks a lot everyone!
    Jun 26, 2018 0
  • Netflix yesottoman
    replicas
    Jun 26, 2018 0
  • New / Mgmt
    jtjw20

    New Mgmt

    PRE
    Shopify
    jtjw20more
    Projections
    Jun 26, 2018 0

Salary
Comparison

    Real time salary information from verified employees