Does Facebook, twitter etc newsfeed compute them as you refresh is its pre computed?
As in title. Feels like if it’s computed during refresh that can take a long time. Of course fb and twitter have large computation power to do this but for someone without that resource will you still do it in same way? Or compute frequent users newsfeed and cache them etc. If so how often will you run such algorithm? Just curious ty x
Well... We used to have control over this. But the AI has started to take over. It precomputes when it wishes and computes on the fly randomly. We’ve even observed instances where it computes as if you were using your dad’s account. It will reverse the order if it detects you have ever complained about Instagram moving away from chronological order.
It is a common distributed computing problem - aggregate on read (when you request your feed) or aggregate on write (when the event happens). Facebook aggregates on read - look at talk from F8 2008 or 09
That would make sense why I get a completely different feed 5 mins after I reopen the app. Interesting.
Create reverse chronological timelines on write, inject, rank and enrich on read
I can see how that prevents hot spots on read but you pay a huge cost each time a write by someone with a lot of followers happens. Majority of these writes are wasted because your DAU/MAU is low. Why not just use caching to work around hot spots and aggregate on read ?
It's a combination of both. The recommendation model would work against a smaller set of data that is cached based on some criteria. We do that @ Bloomberg news. You could also use the search engine to creatively solve this problem.