Streams & Buffers

Difficulty: Advanced

Question

Explain Node.js streams. Why are they important for handling large files?

Answer

Streams process data piece by piece (chunks) instead of loading everything into memory. This is essential for handling large files, network data, or real-time processing.

Stream types: 1. Readable: source of data (fs.createReadStream, http request) 2. Writable: destination (fs.createWriteStream, http response) 3. Duplex: both readable and writable (TCP socket) 4. Transform: modify data as it passes through (zlib.createGzip)

Key advantage: A 10GB file can be processed with ~64KB of memory using streams.

Code examples

Reading Large Files: Buffer vs Stream

const fs = require('fs');

// BAD: loads entire file into memory
// For a 2GB file, this uses 2GB+ RAM
fs.readFile('huge.csv', 'utf8', (err, data) => {
  const lines = data.split('\n');
  console.log(`Lines: ${lines.length}`);
});

// GOOD: stream processes line by line
// Uses only a few KB of memory
const readline = require('readline');
const rl = readline.createInterface({
  input: fs.createReadStream('huge.csv'),
});

let count = 0;
rl.on('line', (line) => {
  count++;
});
rl.on('close', () => {
  console.log(`Lines: ${count}`);
});

fs.readFile loads the entire file. Streams process it chunk by chunk, keeping memory usage constant regardless of file size.

Pipe Chain: Read → Transform → Write

const fs = require('fs');
const zlib = require('zlib');

// Compress a file using pipe
fs.createReadStream('input.txt')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('input.txt.gz'))
  .on('finish', () => console.log('Compressed!'));

// HTTP response streaming
const http = require('http');
http.createServer((req, res) => {
  const stream = fs.createReadStream('bigfile.mp4');
  // Pipe directly to response - no memory buildup
  stream.pipe(res);
}).listen(3000);

pipe() connects streams together and handles backpressure automatically. Data flows through the chain without buffering the entire content.

Custom Transform Stream

const { Transform } = require('stream');

// Transform that converts text to uppercase
const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    const upper = chunk.toString().toUpperCase();
    callback(null, upper);
  },
});

// Usage
process.stdin
  .pipe(upperCase)
  .pipe(process.stdout);

// Or with file
fs.createReadStream('input.txt')
  .pipe(upperCase)
  .pipe(fs.createWriteStream('output.txt'));

Transform streams let you process and modify data in transit. The callback(null, data) pattern passes transformed data to the next stream.

Key points

Backpressure: when the writable stream can't keep up, the readable stream automatically pauses
pipe() handles backpressure automatically - use it instead of manual event handling
In Express/Koa, req and res are streams - you can pipe directly to/from them

Concepts covered

Readable Stream, Writable Stream, Transform Stream, Pipe, Buffer, Backpressure