Inter-process communication: pipes, shared memory and message queues

Difficulty: Intermediate

Question

What are the different IPC mechanisms? Compare pipes, message queues and shared memory, and say when you would use each.

Answer

Processes are isolated by design, which is great for safety but awkward when they must cooperate. A browser needs to talk to its renderer, a shell pipeline needs to connect ls to grep, and a database needs to talk to clients. Inter-process communication is the set of OS-provided channels that let isolated processes exchange data or signals without breaking the isolation wall. The two fundamental models are message passing, where the kernel copies data between processes, and shared memory, where processes map a common region and read and write it directly.

A pipe is the simplest mechanism. An anonymous pipe is a unidirectional byte stream created with pipe(), giving two file descriptors, one for reading and one for writing. It works between related processes, because the child inherits the descriptors through fork. It is what the shell uses for ls | grep foo. Data is a stream with no message boundaries, and the kernel buffers it, blocking the writer if the buffer is full and the reader if it is empty. A named pipe or FIFO is created with mkfifo and appears as a file in the filesystem, so unrelated processes can use it. Pipes are unidirectional, so for two-way communication you need two of them or a socket pair.

Message queues preserve message boundaries. A process sends discrete messages (with a type or priority) into a kernel-managed queue, and another process receives them, possibly selectively. POSIX (mq_open, mq_send) and System V (msgget, msgsnd) variants exist. They decouple sender and receiver in time, since the queue persists until read, and they are neat for structured commands. The downsides are size limits and the fact that every message is copied twice, from sender to kernel and from kernel to receiver.

Shared memory is the fastest IPC. The kernel maps the same physical pages into the address spaces of multiple processes (shm_open plus mmap, or shmget and shmat). After setup, no system calls or copies are needed for communication; processes just access memory. The catch is that the OS provides no synchronization, so you must protect the data yourself with semaphores or mutexes placed in shared memory; otherwise you get race conditions. It is the best choice for large data volumes and high-frequency exchange, such as video frames or database buffer caches.

Other mechanisms complete the picture. Sockets, both Unix domain and network, support two-way communication and work across machines. Signals are lightweight asynchronous notifications with almost no data. Memory-mapped files provide shared state backed by disk. Semaphores are strictly for synchronization rather than data transfer.

Here is the decision guide I would give: use pipes for simple streaming between parent and child or in a pipeline; message queues for discrete, structured messages and loose coupling; shared memory for bulk data where speed matters and you can manage synchronization; sockets when processes may run on different hosts or you need a full duplex, general-purpose channel.

A subtle interview point: shared memory is fast because it avoids copying and kernel involvement after setup, but the complexity moves to the programmer. Message passing is slower but safer and easier to reason about, and it is naturally extensible to distributed systems.

Code examples

Parent to child over an anonymous pipe

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/wait.h>

int main(void) {
    int fd[2];
    char buf[64];
    pipe(fd);                          /* fd[0] = read end, fd[1] = write end */

    if (fork() == 0) {                 /* child: reader */
        close(fd[1]);
        int n = read(fd[0], buf, sizeof buf - 1);
        buf[n] = '\0';
        printf("child got: %s\n", buf);
        close(fd[0]);
        return 0;
    }
    close(fd[0]);                      /* parent: writer */
    const char *msg = "hello via pipe";
    write(fd[1], msg, strlen(msg));
    close(fd[1]);
    wait(NULL);
    return 0;
}

Each side closes the end it does not use. Closing the write end also lets the reader see end-of-file.

Key points

Concepts covered

IPC, pipe, shared memory, message queue, sockets