Writable Streams

Difficulty: Intermediate

Writable streams are the counterpart to readable streams: they represent a destination to which data can be written. Common writable streams include fs.createWriteStream for writing to files, HTTP response objects (res), process.stdout and process.stderr, and TCP sockets. Understanding writable streams and backpressure is essential for building efficient data processing pipelines.

fs.createWriteStream(path, options) creates a writable stream to a file. You write data using the stream.write(chunk) method, which returns a boolean indicating whether the internal buffer is full. When the buffer is full (write returns false), you should stop writing and wait for the 'drain' event before continuing. When you are done writing, call stream.end() to flush remaining data and close the stream. The 'finish' event fires after end() is called and all data has been flushed.

Backpressure is a flow control mechanism that prevents a fast producer from overwhelming a slow consumer. When you write faster than the stream can flush data to its destination (disk, network, etc.), the internal buffer fills up. The write() method returns false when this happens, signaling you to pause. When the buffer is drained and ready for more data, the 'drain' event fires. Ignoring backpressure (continuing to write when write() returns false) causes the buffer to grow without bound, potentially consuming all available memory and crashing the process.

Handling backpressure manually requires checking the return value of write() and pausing on false. A correct implementation looks like: if (!stream.write(data)) { await new Promise(resolve => stream.once('drain', resolve)); }. This pattern is tedious but important. The pipe() and pipeline() methods handle backpressure automatically, which is why piping is always preferred over manual read/write loops.

process.stdout and process.stderr are writable streams that output to the terminal. console.log() is essentially process.stdout.write(data + '\n'). In some environments (like piping output to a file), stdout can exert backpressure. HTTP response objects (res) in Express/Node HTTP servers are also writable streams, which means you can pipe readable streams directly to them for efficient file serving or data streaming.

Code examples

Writing to Files with createWriteStream

const fs = require('fs');

const stream = fs.createWriteStream('output.txt', { encoding: 'utf8' });

// Write some data
stream.write('Line 1: Hello\n');
stream.write('Line 2: World\n');
stream.write('Line 3: Done\n');

// End the stream (flushes buffer and closes file)
stream.end('Final line\n');

stream.on('finish', () => {
  console.log('All data written');

  // Verify
  const content = fs.readFileSync('output.txt', 'utf8');
  console.log('File contents:');
  console.log(content);

  fs.unlinkSync('output.txt');
});

stream.on('error', (err) => {
  console.error('Write error:', err.message);
});

write() adds data to the internal buffer. end() optionally writes final data, then flushes and closes. The 'finish' event confirms everything was written to disk. Always listen for 'error' in case of write failures.

Handling Backpressure Correctly

const fs = require('fs');

async function writeWithBackpressure() {
  const stream = fs.createWriteStream('big-file.txt');
  let linesWritten = 0;

  for (let i = 0; i < 100; i++) {
    const data = `Line ${i}: ${'x'.repeat(100)}\n`;
    const canContinue = stream.write(data);
    linesWritten++;

    if (!canContinue) {
      // Buffer is full, wait for drain
      await new Promise(resolve => stream.once('drain', resolve));
    }
  }

  stream.end();

  await new Promise(resolve => stream.on('finish', resolve));
  console.log(`Wrote ${linesWritten} lines with backpressure handling`);

  const stats = fs.statSync('big-file.txt');
  console.log(`File size: ${stats.size} bytes`);
  fs.unlinkSync('big-file.txt');
}

writeWithBackpressure();

Checking write()'s return value and awaiting 'drain' prevents memory exhaustion. Without this, writing millions of lines could buffer everything in memory before any data reaches disk. This pattern is essential for writing large amounts of data.

process.stdout as a Writable Stream

// process.stdout is a writable stream
// console.log() is shorthand for process.stdout.write() + '\n'

process.stdout.write('Hello ');
process.stdout.write('World');
process.stdout.write('\n'); // Must add newline manually

// Writing formatted output
const items = ['Node.js', 'Express', 'Streams'];
items.forEach((item, i) => {
  process.stdout.write(`${i + 1}. ${item}\n`);
});

// Difference from console.log:
// console.log(obj) -> util.inspect(obj) + '\n' -> stdout
// process.stdout.write() -> raw output, no formatting

const data = { name: 'test', count: 42 };
process.stdout.write('Raw: ' + JSON.stringify(data) + '\n');
console.log('Formatted:', data);

process.stdout.write gives you raw control over output without automatic newlines or object formatting. console.log adds a newline and uses util.inspect for objects. For progress bars, spinners, and custom formatters, use stdout.write directly.

Key points

Always handle the return value of write(). When it returns false, the internal buffer is full and you should wait for the 'drain' event before writing more.
stream.end() can optionally accept a final chunk of data to write before closing. After end() is called, no more writes are allowed.
The 'close' event fires after the stream and its underlying resources (file descriptor) are fully released. 'finish' fires when data is flushed but resources may still be held.

Concepts covered

createWriteStream, write, end, drain Event, Backpressure