Amazon Prime Video X-Ray

Roblox
nonsmoking

Go to company page Roblox

nonsmoking
Feb 5 4 Comments

I’m loving this feature. Makes me wonder how it’s implemented. Did a Google search, found nothing.

Obviously it’s doing some image analysis based on the video frame. Is it though? Is Amazon using info from IMDB? The latter’s also owned by Amazon.

I think these kind of algorithms are fucking cool.

Any thoughts/ideas?

#entertainment #media

comments

Want to comment? LOG IN or SIGN UP
TOP 4 Comments
  • Google
    trickybit

    Go to company page Google

    trickybit
    Runs face recognition on the video frames or uses the actor tags that content creator gave them (or IMDb) to display the IMDb pages for each of them
    Feb 6 1
  • Hulu
    Huluween13

    Go to company page Hulu

    Huluween13
    I don't have inside knowledge, but having worked in this space for a while I am pretty confident I can make an educated guess on how it's automated.

    First component is face recognition based on image analysis. There are two opportunities and data sources for this analysis. One is the source video that's decoded during transcoding, and the other are the JPEG thumbnails extracted during transcoding for purposes of seeking/navigation. Scene change detection can also be performed during transcoding which makes it possible to segment the timeline into scenes.

    The other key component is IMDB. It provides you with both a cast list for the movie as well as photos of most of the cast.

    So once you have a cast list and their photos, performing facial recognition isn't very difficult because you're working with a constrained set of possible faces to match. The facial recognition algorithm doesn't need to be perfect, it just needs to be good enough to pick the likeliest match out of a constrained set of cast photos.

    An alternative solution (or just a method of increasing facial recognition likelihood) would be to run speech-to-text conversion of the audio track (the one featuring the original cast) and then match it up to a script or caption file that associates dialog with character names. That would also cover cases where an actor is only heard but not seen (e.g. voiceover) which otherwise wouldn't work with only facial recognition.

    But it's also entirely possible they just pay people somewhere to watch the movie and index all the characters/actors manually. For companies which already do subtitling/captioning this extra task wouldn't be much of a stretch. 🙂
    Feb 12 0
  • Amazon / Other
    nocallouts

    Go to company page Amazon Other

    nocallouts
    This was manual at one point, but I can’t imagine it still is.
    Feb 12 0