X's Recommendation Algorithm

Overview of X's Recommendation Algorithm

X's newly open-sourced recommendation algorithm, released on January 20, 2026, powers the "For You" feed by blending content from accounts you follow (in-network) with discovered content from across the platform (out-of-network). Unlike previous versions, this iteration fully eliminates manual feature engineering, relying instead on a transformer model adapted from xAI's Grok-1 to learn relevance directly from user engagement data. This shift to AI-driven predictions aims to create more personalized and engaging feeds, but Elon Musk has publicly called the current system "dumb" and pledged ongoing improvements, with code updates every four weeks.

The algorithm is hosted on GitHub under xai-org/x-algorithm and is licensed under Apache 2.0. It's primarily written in Rust (for performance-critical components) and Python (for ML elements), making it more efficient than the 2023 version from twitter/the-algorithm, which hadn't been updated in years. Key changes include a complete rewrite to incorporate Grok's transformer architecture, focusing on multi-action predictions rather than simplistic relevance scores.

How the Algorithm Works: Step-by-Step Breakdown

The system processes feed requests through a modular pipeline, funneling thousands of potential posts down to a ranked list of the most relevant ones. Here's a structured overview:

1. Input and Query Hydration:

   • Starts with a user's "For You" request.
   • Fetches user-specific data: engagement history (likes, replies, etc.), following list, preferences, and features like muted keywords or blocked accounts.

2. Candidate Sourcing:

   • In-Network (Thunder Component): Pulls recent posts from followed accounts using real-time Kafka streams and in-memory stores. This ensures low-latency access to familiar content, with automatic trimming of expired posts.
   • Out-of-Network (Phoenix Retrieval): Uses a two-tower embedding model to search a global corpus of posts. One tower encodes the user's profile and history; the other encodes posts. Similarity is computed via dot products to retrieve top candidates. This step gathers a large pool (e.g., thousands) of potential posts.

3. Candidate Hydration and Pre-Scoring Filtering:

   • Enriches candidates with metadata: text, media, author details, video length, etc.
   • Applies filters to remove ineligible content, such as duplicates, old posts, self-authored items, blocked/muted authors, or previously seen content. This reduces the pool while respecting user preferences.

4. Scoring and Ranking (Phoenix Scorer):

   • The core of the system: A Grok-based transformer predicts probabilities for 15 user actions per candidate post.

   • Predicted Actions:	Category	Examples	Impact on Score
     Positive	Like, Reply, Repost, Quote, Click, Profile Click, Video View, Photo Expand, Share, Dwell, Follow Author	Increases score (weighted positively)
     Negative	Not Interested, Block Author, Mute Author, Report	Decreases score (weighted negatively)

   • Final score = Sum of (weight × probability) for each action. Weights prioritize meaningful engagement like replies over likes (which are "nearly worthless" in boosting visibility).
   • Additional adjustments: Author diversity scorer penalizes repeated authors to promote variety; out-of-network scorer tweaks scores for broader discovery.
   • Key innovation: Candidates are isolated during attention (no inter-candidate influence), allowing cacheable, consistent scoring.

5. Selection and Post-Selection Filtering:

   • Sorts by final score and selects top results.
   • Final filters handle edge cases: spam detection, deleted content, violence, or thread deduplication.

6. Output:

   • Delivers the ranked feed via a gRPC endpoint.

Key Insights and Improvements

• Engagement Prioritization: Unlike simplistic views that emphasize likes, the algorithm values "conversational engagement" (replies, quotes) far more. Likes contribute minimally, while negative signals like reports heavily demote content. To optimize visibility, creators should aim for concise, reply-provoking posts rather than frequent posting.
• Diversity and Fairness: Built-in mechanisms limit any single author's dominance, addressing common complaints about echo chambers.
• Efficiency and Scalability: Rust handles real-time components for sub-millisecond latency, while Python manages ML. Hash-based embeddings enable fast similarity searches without full vector computations.
• Transparency and Future Changes: Musk's pledge for bi-monthly updates includes explainers for tweaks, potentially addressing EU regulatory pressures on transparency. However, the system still integrates ads into organic rankings, which could blur lines between paid and earned visibility.
• Limitations: The transformer relies on historical data, so new users or trends may see less accurate recommendations initially. No hand-tuned features mean the model could amplify biases in engagement data if not monitored.

This analysis draws directly from the source code and documentation, providing a more precise, code-backed view than high-level overviews that might overlook modular components like Thunder or multi-action weighting. For hands-on exploration, clone the repo and experiment with the pipeline framework.