use and abuse of pipes with audio data

fifo file fun!

What is a fifo ? Well, it's a special sort of file. The name stands for "file in, file out", indicating that the file goes in and out. It is sometimes also called a named pipe.

Why is that?

Well, suppose I type

mpg123 -s *.mp3 | rawplay

Looking at it a bit closer, I could say I am piping the output of mpg123 to a temporary file which I do not see (symbolised by the | sign) and then running that temporary file past rawplay. I could say the temporary file is hidden from sight. I could say all that, but I'd be wrong actually, because it is not a temporary file - it is a buffer - and data continuously streams via this buffer.

What's the difference between a file and a buffer?

Well, let me uglify the one-liner by bringing a temporary file into sight. Let me name the temporary file explicitly. I could do:

mpg123 -s *.mp3 > tempfile
cat tempfile | rawplay
rm tempfile

and get the same result as our one-liner. But this is now a multistep process, and would probably involve an awful lot of diskswapping as tempfile is created in its full glory, and only then streamed into rawplay. Not at all nice and continuous compared to the one-liner buffer version.

There is another way of making it a multistep process though. I can make an equivalent of the one-liner buffer version, without the diskswapping overhead, but still using a temporary handle, like this:

mkfifo myfifo
mpg123 -s *.mp3 > myfifo
cat myfifo | rawplay
rm myfifo

Oh yes. I sneaked the fifo thing in here. It is called a "named pipe" because it is a pipe type of operation being done and the pipe has been explicitly named. In this case it has been called myfifo. myfifo does the file in/file out bit, and it differs from a normal file (such as tempfile earlier) because it has a file size of 0. Not surprisingly, perhaps, because it is a buffer, sending data out as it gets it.

That's nifty, But what's it good for? Why should I even bother with a multistep fifo process using this weirdo stuff when I have the one-liner?

Well, now I have an explicit handle on the step in the middle, a handle that takes up no diskspace, and it lets me do things on-the-fly to the stream.

Fifo? A handle that lets me do things on the fly?

Yes. Some more examples are probably a good idea now:

I will use the netcat utility to illustrate things. netcat (or nc) is a must-have tool that allows the machine to send and listen to stuff on ports. (We covered an introduction to netcat previously - but if you don't have it "apt-get install netcat" will install it).

Netcat is like a telnet on steroids. Eg: suppose I have a smtp server, which I will unimaginatively call smtpserver, allowing access on the standard port 25. Then:

nc smtpserver 25

will be pretty much the same as

telnet smtpserver 25

See the man page for netcat for more details.

Now, suppose I have this tinpot laptop with horrible shrill speakers, and a multimedia desktop machine with StupendaSound speakers, (you know the kind I mean, the ones with an eyepopping, earbleeding bass). They are connected via LAN. I don't want to hear the stream on my tinny sounding laptop - I want to hear it in its full ghettoblasting glory on the multimedia desktop.

So on tinpot I type:

tinpot:~$ mkfifo myfifo 
tinpot:~$ mpg123 -s *.mp3 > myfifo

Ie: I make the fifo, and stream the raw audio to it.

On ghettoblaster I type:

ghettoblaster:~$ nc -l -p 2345 | rawplay

Here, netcat is listening on port 2345, and piping anything it hears to rawplay. There is no sound being heard yet.

Then back on tinpot I type:

tinpot:~$ cat myfifo  | nc ghettoblaster 2345

Ie, I am taking the myfifo buffer on tinpot and forwarding it to netcat, which sends it out to ghettoblaster's port 2345.

The moment I type in the last line, the sound starts up (think of it as a valve in a plumbing pipe system - until the myfifo valve is released, nothing flows). Of course, this would be a terrible way to do distribute the stream over the internet, since it is uncompressed. But on a LAN with loads of bandwidth this is just dandy.

Why use fifo tricks at all? After all, I could use just these two lines, and do away with myfifo entirely:

$ghettoblaster:~$ nc -l -p 2345 | rawplay
$tinpot:~$ mpg123 -s *.mp3 | nc ghettoblaster 2345 

True, and that works splendidly for two machines if I want that only one of them play the sound. But suppose I have a third machine in the lounge and I want to play the stream there as well, at the same time? Try as I may, there is no way of getting sound out on more than one machine on-the-fly without installing a full distributed sound application - unless I use fifo, the magic handle in the middle. You see, once you have a handle, you can use all the usual trickery with pipes, redirection, and, most importantly in our case, tee.

tee for two (or more)

Let's get really adventurous, and pipe the raw sound data not just to two, or even three machines. No indeed. Let's send it to 6 machines all over the house. It makes the structure of things a bit clearer actually. The set up then goes like this:

On the machine that is the source of the sound, (tinpot) I do:

tinpot:~$ mkfifo myfifo1 myfifo2 myfifo3 .... myfifo6
tinpot:~$ mpg123 -s *.mp3 > myfifo1

machine1:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo1 | tee myfifo2 | nc machine1 2345

machine2:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo2 | tee myfifo3 | nc machine2 2345

...

machine6:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo6 | nc machine6 2345

The moment I release the valve (ie type "cat myfifo8 | nc machine6 2345" on the last machine), all 6 machines start playing. (Remember, tee pipes stuff to STDOUT as well as to a file (in this case a fifo). It functions like an audio splitter in this case).

So there I am, with music blaring more or less in sync throughout the house, and a big geeky grin on my face. For I am now a Master of fifo, tee, and netcat! With my newfound nerd bravado, I venture forth to the next adventure in GNU/Linux geekdom:

natty new nettee net utility - tee for fifo

So what if the gobbledygook title of this subsection doesn't even sound like English? Think of it as me being incoherent with excitement over David Mathog's rather spiffy nettee.

nettee is a clever new utility that sort of combines netcat with tee, and so does away with fifos. Grab it and compile it off the site (there is no debian package for it to date (24 May 2005) that I am aware of, but that may have changed by the time you read this - it seems to only have been around for about a month so far).

Install nettee on all the machines, and run it on each like this:

machine1:~$ ./nettee | rawplay
machine2:~$ ./nettee | rawplay
...
machine6:~$ ./nettee | rawplay

so now each of these is listening on the nettee port (9997 by default).

Then on the source machine you type in:

tinpot:~$ mpg123 -s *.mp3 | ./nettee -in - -next machine1,machine2,machine3, ... ,machine6

The walls are a-quaking now as I blast out "WE WILL ROCK YOU!!!" at full annoy-my-neighbours level. Not just on one machine, but simultaneously now on all the machines in the house. The plaster is falling down, and the irritating thumps of my neighbour's broom are drowned out by the boneshaking ear-crushing sound. I thump my desk frenziedly along with the music in the primal excitement of the moment, feeling deliriously self-satisfied and smug, I bellow out aloud: "Yep, nettee really, literally, ROCKS!!!"

Epilogue

And as this article at last comes back from its voyage of discovery on the endless ocean of GNU/Linux, coming back to rest on terra firma once again, I say unto you have have so bravely gone with me: go forth, young geek, and discover new ways to abuse these tools, and spread the good word! Yea, for verily you are one of the Chosen ones!

PJ