Design a File Storage System

Difficulty: Advanced

Question

Design a file storage system like Google Drive or Dropbox. How would you handle large file uploads, deduplication, and sharing?

Answer

Core requirements: 1. Upload and download files of any size (up to several GB) 2. Organize files in folders with metadata 3. Share files with permissions (view, edit) 4. Sync across devices 5. Version history

Key design decisions: - Chunked uploads: Split large files into 4-8MB chunks for resumable uploads - Object storage: Use S3 or similar for blob storage, database for metadata - Deduplication: Hash-based dedup saves storage (same file uploaded by different users stored once) - Pre-signed URLs: Direct client-to-S3 uploads bypass your server - CDN: Frequently accessed files cached at edge locations

Code examples

Chunked Upload Flow

// Client: split file into chunks
async function uploadLargeFile(file) {
  const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB chunks
  const totalChunks = Math.ceil(file.size / CHUNK_SIZE);

  // 1. Initialize upload session
  const { uploadId } = await api.post('/uploads/init', {
    filename: file.name,
    fileSize: file.size,
    totalChunks,
    contentType: file.type
  });

  // 2. Upload chunks in parallel (with concurrency limit)
  const uploadChunk = async (index) => {
    const start = index * CHUNK_SIZE;
    const chunk = file.slice(start, start + CHUNK_SIZE);
    const hash = await computeSHA256(chunk);

    await api.put(`/uploads/${uploadId}/chunks/${index}`, chunk, {
      headers: {
        'Content-Type': 'application/octet-stream',
        'X-Chunk-Hash': hash
      }
    });
  };

  // Upload 3 chunks at a time
  await parallelLimit(totalChunks, 3, uploadChunk);

  // 3. Complete upload - server assembles chunks
  const result = await api.post(`/uploads/${uploadId}/complete`);
  return result.file;
}

// Resume interrupted upload
async function resumeUpload(uploadId) {
  const { completedChunks } = await api.get(`/uploads/${uploadId}/status`);
  // Only upload missing chunks
  const remaining = allChunks.filter(i => !completedChunks.includes(i));
  // ... upload remaining chunks
}

Chunked uploads allow resuming interrupted transfers. Each chunk is verified with a hash. The server tracks completed chunks and assembles the final file when all chunks arrive.

Pre-signed URL Upload (Direct to S3)

// Server: generate pre-signed URL for direct upload
app.post('/api/files/upload-url', auth, async (req, res) => {
  const { filename, contentType, fileSize } = req.body;

  // Validate
  if (fileSize > MAX_FILE_SIZE) {
    return res.status(400).json({ error: 'File too large' });
  }

  const key = `users/${req.user.id}/${Date.now()}_${filename}`;

  // Generate pre-signed URL (valid 15 min)
  const uploadUrl = await s3.getSignedUrl('putObject', {
    Bucket: BUCKET_NAME,
    Key: key,
    ContentType: contentType,
    Expires: 900
  });

  // Create file metadata record
  const file = await db.files.create({
    userId: req.user.id,
    name: filename,
    s3Key: key,
    size: fileSize,
    contentType,
    status: 'UPLOADING'
  });

  res.json({ uploadUrl, fileId: file.id });
});

// Client: upload directly to S3
const { uploadUrl, fileId } = await api.post('/api/files/upload-url', {
  filename: 'document.pdf',
  contentType: 'application/pdf',
  fileSize: file.size
});

await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

// Notify server upload is complete
await api.post(`/api/files/${fileId}/confirm`);

Pre-signed URLs let clients upload directly to S3, bypassing your server entirely. This reduces server load and bandwidth costs. The server only handles metadata and URL generation.

File Deduplication & Metadata Schema

-- Metadata database schema
CREATE TABLE files (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  owner_id UUID REFERENCES users(id),
  name VARCHAR(255) NOT NULL,
  parent_folder_id UUID REFERENCES folders(id),
  blob_id UUID REFERENCES blobs(id),  -- points to actual content
  size BIGINT NOT NULL,
  content_type VARCHAR(100),
  version INT DEFAULT 1,
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW()
);

-- Actual file content (deduplication layer)
CREATE TABLE blobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  sha256_hash VARCHAR(64) UNIQUE NOT NULL,  -- dedup key
  s3_key VARCHAR(500) NOT NULL,
  size BIGINT NOT NULL,
  ref_count INT DEFAULT 1  -- how many files reference this blob
);

-- File sharing
CREATE TABLE file_shares (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  file_id UUID REFERENCES files(id),
  shared_with_id UUID REFERENCES users(id),
  permission VARCHAR(20) DEFAULT 'VIEW',  -- VIEW, EDIT
  share_link VARCHAR(100) UNIQUE,
  expires_at TIMESTAMP
);

-- Deduplication: check if blob already exists
// Before storing a new file:
const hash = computeSHA256(fileBuffer);
let blob = await db.blobs.findByHash(hash);

if (blob) {
  // File content already exists - just increment ref count
  await db.blobs.incrementRefCount(blob.id);
} else {
  // New content - upload to S3
  const s3Key = await uploadToS3(fileBuffer);
  blob = await db.blobs.create({ sha256Hash: hash, s3Key, size: fileBuffer.length });
}

// Create file metadata pointing to the blob
await db.files.create({ ownerId: userId, blobId: blob.id, name: filename });

Content-addressable storage uses SHA-256 hashes for deduplication. Multiple file records can point to the same blob. When the last reference is deleted, the blob can be garbage collected from S3.

Key points

Concepts covered

Object Storage, Chunking, Deduplication, CDN, Pre-signed URLs