According to Facebook, over half of Instagram’s roughly 1 billion users visit Instagram Explore to discover videos, photos, livestreams, and Stories each month. Predictably, building the underlying recommendation engine — which curates the billions of pieces of content uploaded to Instagram — posed an engineering challenge, not least because it works in real time.
In a blog post published this morning, Facebook for the first time peeled back the curtains on Explore’s inner workings. Its three-part ranking funnel, which the company says was architected with a custom query language and modeling techniques, extracts 65 billion features and makes 90 million model predictions every second. And that’s just the tip of the iceberg.
Before the team behind Explore embarked on building a content recommendation system, they developed tools to conduct large-scale experiments and obtain strong signals on the breadth of users’ interests. The first of these was IGQL, a meta language that provided the level of abstraction needed to assemble candidate algorithms in one place.
IGQL is optimized in C++, which helps minimize latency and compute resources without sacrificing extensibility, Facebook says. It’s both statistically validated and high-level, enabling engineers to write recommendation algorithms in a “Python-like” fashion. And it complements an account embeddings component that helps identify topically similar profiles as part of a retrieval pipeline that focuses on account-level information.
A framework — Ig2vec — treats Instagram accounts a HUC99 user interacts with as word sequences in a sentence, which informs the predictions of a model with respect to which accounts the user might interact with. (Facebook notes that a sequence of accounts interacted with in a session is more likely to be topically coherent compared with random accounts.) Concurrently, Facebook’s AI Similarity Search nearest neighbor retrieval library (FAISS) queries millions of accounts based on a metric used in embedding training.