Design a Social Media Feed

Difficulty: Advanced

Question

Design a social media news feed like Twitter or Instagram. How would you handle fan-out for users with millions of followers?

Answer

The news feed shows a personalized, ranked timeline of posts from people a user follows.

Two main approaches: 1. Fan-out on Write (Push): When a user posts, immediately push the post to all followers' feed caches. Fast reads but expensive writes for celebrities. 2. Fan-out on Read (Pull): When a user opens their feed, fetch posts from all followed users and merge. Cheap writes but slow reads. 3. Hybrid: Push for normal users (< 10K followers), pull for celebrities (> 10K followers).

Feed ranking: Chronological is simplest. ML-based ranking considers recency, engagement, relationship strength, and content type.

Key metrics: Feed generation latency, post delivery latency, engagement rates.

Code examples

Fan-out on Write (Push Model)

// When a user creates a post:
async function createPost(authorId, content) {
  // 1. Save post to database
  const post = await db.posts.create({ authorId, content });

  // 2. Get all followers
  const followers = await db.followers.findAll({
    where: { followingId: authorId },
    select: { followerId: true }
  });

  // 3. Fan-out: push to each follower's feed cache
  const pipeline = redis.pipeline();
  for (const { followerId } of followers) {
    pipeline.lpush(`feed:${followerId}`, JSON.stringify({
      postId: post.id,
      authorId,
      timestamp: post.createdAt
    }));
    pipeline.ltrim(`feed:${followerId}`, 0, 999);  // Keep last 1000
  }
  await pipeline.exec();

  return post;
}

// Reading the feed is fast:
async function getFeed(userId, page = 0, limit = 20) {
  const start = page * limit;
  const feedItems = await redis.lrange(
    `feed:${userId}`, start, start + limit - 1
  );

  // Hydrate with full post data
  const postIds = feedItems.map(item => JSON.parse(item).postId);
  const posts = await db.posts.findMany({
    where: { id: { in: postIds } },
    include: { author: true, likes: true }
  });

  return posts;
}

// Problem: celebrity with 10M followers
// Writing to 10M Redis lists takes too long
// Solution: hybrid approach (see next example)

Fan-out on write pre-computes every user's feed so reads are instant (just read from Redis list). But posting is expensive when the author has millions of followers.

Hybrid Fan-out Approach

// Hybrid: push for normal users, pull for celebrities
const CELEBRITY_THRESHOLD = 10000;

async function createPost(authorId, content) {
  const post = await db.posts.create({ authorId, content });
  const followerCount = await db.followers.count(authorId);

  if (followerCount < CELEBRITY_THRESHOLD) {
    // Push model: fan-out to all followers
    await fanOutToFollowers(authorId, post);
  } else {
    // Pull model: don't fan-out, followers will pull
    // Just mark this celebrity as having new content
    await redis.set(`celebrity:newpost:${authorId}`, post.id, 'EX', 86400);
  }

  return post;
}

// Merge pushed feed with pulled celebrity posts
async function getHybridFeed(userId, page = 0, limit = 20) {
  // 1. Get pre-computed feed (from push model)
  const pushedFeed = await redis.lrange(`feed:${userId}`, 0, 199);

  // 2. Get list of celebrities this user follows
  const celebrities = await getCelebrityFollowings(userId);

  // 3. Fetch recent posts from celebrities (pull model)
  const celebrityPosts = await db.posts.findMany({
    where: {
      authorId: { in: celebrities },
      createdAt: { gte: oneDayAgo() }
    },
    orderBy: { createdAt: 'desc' },
    take: 50
  });

  // 4. Merge and sort by timestamp
  const merged = [...parseFeed(pushedFeed), ...celebrityPosts]
    .sort((a, b) => b.createdAt - a.createdAt)
    .slice(page * limit, (page + 1) * limit);

  return merged;
}

The hybrid approach avoids the celebrity fan-out problem. Normal users' posts are pushed; celebrity posts are pulled on demand. The feed merges both sources and sorts by timestamp.

Feed Ranking Algorithm

// Simple ranking score for feed items
function calculateFeedScore(post, viewer) {
  let score = 0;

  // 1. Recency (decays over time)
  const hoursAgo = (Date.now() - post.createdAt) / 3600000;
  score += Math.max(0, 100 - hoursAgo * 2);  // -2 points per hour

  // 2. Engagement signals
  score += Math.log2(post.likesCount + 1) * 10;
  score += Math.log2(post.commentsCount + 1) * 15;
  score += Math.log2(post.sharesCount + 1) * 20;

  // 3. Relationship strength
  const interactions = getInteractionCount(viewer.id, post.authorId);
  score += Math.min(interactions * 5, 50);  // Cap at 50

  // 4. Content type boost
  if (post.hasMedia) score += 10;
  if (post.isVideo) score += 15;

  // 5. Diversity penalty (avoid too many posts from same author)
  // Applied during feed assembly, not per-post

  return score;
}

// Apply ranking to feed
async function getRankedFeed(userId, limit = 20) {
  // Get candidate posts (last 24h from followed users)
  const candidates = await getCandidatePosts(userId, 200);

  // Score each post
  const scored = candidates.map(post => ({
    ...post,
    score: calculateFeedScore(post, { id: userId })
  }));

  // Sort by score descending
  scored.sort((a, b) => b.score - a.score);

  // Diversify: limit 3 posts per author in top 20
  return diversify(scored, { maxPerAuthor: 3 }).slice(0, limit);
}

Feed ranking balances recency, engagement, and relationship signals. Logarithmic scaling prevents viral posts from dominating. Diversity ensures variety from different authors.

Key points

Concepts covered

Fan-out, News Feed, Timeline, Ranking Algorithm, Caching