Difficulty: Advanced
Design a video streaming service like YouTube or Netflix. How would you handle video upload, transcoding, and adaptive streaming?
A video streaming service has two main flows: upload pipeline (ingest, transcode, store) and playback pipeline (serve, stream, adapt).
Upload Pipeline: 1. Client uploads raw video to object storage (S3) 2. Transcoding service converts to multiple resolutions (240p, 480p, 720p, 1080p) 3. Each resolution is split into small segments (2-10 seconds) 4. Segments are distributed to CDN edge servers
Playback Pipeline: 1. Client requests manifest file (list of available qualities) 2. Player selects quality based on bandwidth 3. Segments are fetched from nearest CDN edge 4. Adaptive bitrate: quality adjusts in real-time based on network conditions
Protocols: HLS (Apple, most common), DASH (open standard). Both use chunked HTTP delivery.
// Upload endpoint: accept video and queue transcoding
app.post('/api/videos/upload', auth, upload.single('video'), async (req, res) => {
// 1. Upload raw video to S3
const s3Key = `raw/${req.user.id}/${Date.now()}_${req.file.originalname}`;
await s3.upload({
Bucket: BUCKET,
Key: s3Key,
Body: req.file.buffer,
ContentType: req.file.mimetype
}).promise();
// 2. Create video record
const video = await db.videos.create({
userId: req.user.id,
title: req.body.title,
rawS3Key: s3Key,
status: 'PROCESSING',
duration: null // Set after transcoding
});
// 3. Queue transcoding job
await transcodingQueue.addJob('TRANSCODE_VIDEO', {
videoId: video.id,
s3Key,
resolutions: [240, 480, 720, 1080]
});
res.json({ videoId: video.id, status: 'PROCESSING' });
});
// Transcoding worker (uses FFmpeg)
async function transcodeVideo(job) {
const { videoId, s3Key, resolutions } = job.data;
// Download raw video
const rawPath = await downloadFromS3(s3Key);
for (const resolution of resolutions) {
// Transcode to each resolution
const outputDir = `/tmp/transcode/${videoId}/${resolution}p`;
// FFmpeg command: transcode + segment into HLS chunks
await exec(`ffmpeg -i ${rawPath} \
-vf scale=-2:${resolution} \
-c:v h264 -b:v ${getBitrate(resolution)} \
-c:a aac -b:a 128k \
-hls_time 6 \
-hls_segment_filename '${outputDir}/segment_%03d.ts' \
${outputDir}/playlist.m3u8`);
// Upload segments to S3
await uploadDirectory(outputDir, `videos/${videoId}/${resolution}p/`);
}
// Generate master playlist
await generateMasterPlaylist(videoId, resolutions);
// Update video status
await db.videos.update(videoId, { status: 'READY' });
}Videos are uploaded to S3, then a background worker transcodes them into multiple resolutions. FFmpeg segments each resolution into 6-second HLS chunks for adaptive streaming.
// Master playlist (master.m3u8)
// Lists all available quality levels
#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=400000,RESOLUTION=426x240
/videos/vid_123/240p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=854x480
/videos/vid_123/480p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2500000,RESOLUTION=1280x720
/videos/vid_123/720p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
/videos/vid_123/1080p/playlist.m3u8
// Quality playlist (720p/playlist.m3u8)
// Lists segments for this quality level
#EXTM3U
#EXT-X-TARGETDURATION:6
#EXTINF:6.0,
segment_000.ts
#EXTINF:6.0,
segment_001.ts
#EXTINF:6.0,
segment_002.ts
#EXTINF:4.5,
segment_003.ts
#EXT-X-ENDLIST
// How adaptive bitrate works:
// 1. Player loads master.m3u8
// 2. Measures available bandwidth
// 3. Selects appropriate quality
// 4. Fetches segments one by one
// 5. If bandwidth drops: switch to lower quality
// 6. If bandwidth improves: switch to higher quality
// Each segment is a standalone file - quality can change per segment
// Serving from CDN
app.get('/videos/:videoId/*', (req, res) => {
// CDN handles this in production
// Set long cache headers for segments (immutable content)
res.set('Cache-Control', 'public, max-age=31536000, immutable');
// Stream from S3 or local cache
const s3Key = `videos/${req.params.videoId}/${req.params[0]}`;
s3.getObject({ Bucket: BUCKET, Key: s3Key }).createReadStream().pipe(res);
});HLS works by splitting videos into small segments at multiple quality levels. The player adaptively switches between qualities based on real-time bandwidth measurements, ensuring smooth playback.
// Video metadata schema
CREATE TABLE videos (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID REFERENCES users(id),
title VARCHAR(255) NOT NULL,
description TEXT,
duration INT, -- seconds
status VARCHAR(20), -- PROCESSING, READY, FAILED
thumbnail_url VARCHAR(500),
view_count BIGINT DEFAULT 0,
like_count INT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
);
// View tracking (for recommendations)
CREATE TABLE video_views (
id BIGSERIAL PRIMARY KEY,
video_id UUID REFERENCES videos(id),
user_id UUID,
watch_duration INT, -- how many seconds watched
total_duration INT,
created_at TIMESTAMP DEFAULT NOW()
);
// View count: use Redis for real-time, batch update to DB
async function trackView(videoId, userId, watchDuration) {
// Increment in Redis (fast)
await redis.hincrby(`video:views:${videoId}`, 'count', 1);
// Log detailed view for analytics (async)
await analyticsQueue.addJob('TRACK_VIEW', {
videoId, userId, watchDuration
});
}
// Batch sync views to PostgreSQL every 5 minutes
setInterval(async () => {
const keys = await redis.keys('video:views:*');
for (const key of keys) {
const videoId = key.split(':')[2];
const count = await redis.hget(key, 'count');
if (count > 0) {
await db.videos.update(videoId, {
viewCount: { increment: parseInt(count) }
});
await redis.hdel(key, 'count');
}
}
}, 300000);View counts use Redis for real-time tracking to avoid write contention on the database. Counts are periodically flushed to PostgreSQL. Detailed view data is logged asynchronously for recommendations.
Video Encoding, Adaptive Bitrate, CDN, Chunked Streaming, HLS/DASH