NamedPipe

Many people come here searching for what a named pipe is. Since I’ve got some knowledge on this topic (and it’s the blog’s namesake), I thought I’d try to help them out finally. The article below requires some basic Unix knowledge.

To start off, a pipe (symbolized by |) is way to transfer data between processes without using any I/O. It truly acts like a “virtual pipe” for the data coming out of one process to flow into the input of another process. Most shell users have used these with something like:

ls -l | head -5

This will show the top 5 lines from a directory listing. It pipes the directory listing from the ls -l command into the head command – and it avoids using any I/O (with the exception of reading the directory listing).

A named pipe (sometimes called a FIFO – for “first in, first out”), is a Unix pipe that acts like a file. It has a name so it can be referred to by processes (e.g. open, read from, written to). But it’s still a pipe in that the data flows through the named pipe – data written to the pipe is read by processes that are reading from the pipe. The nice thing about named pipes is that multiple processes can read and write to them. And processes don’t start sending data through the pipe until there is something there to consume it.

How to make a named pipe

FreeBSD – mkfifo [-m] pipename
Solaris – mkfifo [-m] pipename
Linux – mknod pipename p or mkfifo [-m] pipename

An example

A great way to use named pipes is to avoid writing intermediate files for compressing while backing up databases. Usually people will dump their database to the filesystem and then compress it from there. But with named pipes you can dump your database to a named pipe and then compress the namedpipe. (I realize the example below is a little trivial since mysql dumps to STDOUT, so a regular pipe would work fine.)


mkfifo dbdump

mysqldump -uroot -ppassword database >dbdump &

cat dbdump | gzip > database.gz


The above creates a named pipe called dbdump. Then starts the mysqldump command to send the database dump to the named pipe. This step is set to the background so you can run the compress command next. Then you send the output of the named pipe into the gzip command and direct it to a compressed file. There you go, no I/O until you have the compressed data.

References

Advanced Unix Programming, Marc J. Rochkind. Prentice Hall P T R, 1985.
Introduction to Named Pipes – Linux Journal
The lost art of named pipes – TechTarget