This site is now 100% read-only, and retired.

A brief introduction to publish-subscribe queuing with redis

Posted by Steve on Wed 17 Dec 2014 at 16:46

Tags: ,

In this brief article we'll demonstrate using Redis for a publish/subscribe system. There are dedicated publish-subscribe platforms out there, but Redis is reasonably performant and essentially configuration-free.

We previously looked at using the beanstalkd queue, and also considered using the Redis data-store in the past. Today we'll look at the publish-subscribe facilities which are present in Redis.

When we looked at using the beanstlkd-queue previously we briefly discussed:

  • Adding messages/objects/things to the queue.
  • At a later point taking items from that queue.
    • Items were pulled out in the same order as they were added.
    • Given that the queue could be accessed over TCP/IP, itemss to be both inserted and removed from multiple hosts.

Queues are perfect for distributing work, because you could easily configure an arbitrary number of hosts which would pull jobs from the queue, and process them in turn.

The downside is that all hosts are potentially receiving all messages, and this is where the idea of "topics" comes into play. In a publish and subscribe system messages aren't added to a global queue - instead they're added to particular channels, or topics. This means workers can dedicate themselves to listening for particular things - rather than having to receive all messages and discard those that don't apply. This cuts down on complexity and bandwidth use.

For example you might have a set of hosts with IPv6 connectivity, and they would take jobs which the hosts running only IPv4 couldn't handle. They could do that by listening for jobs with an IPv6 topic.

The key is that :

  • We have an emitter which publishes messages/items/objects/things.
    • Each message has an associated topic.
  • We have consumers which only receive messages they explicitly listen for.

As an example this site sends out emails when users join, when they forget their passwords, etc. Rather than sending the email immediately they are sent out asynchronously.

The following example, written in Perl, uses the redis client-library to connect to a (redis) server and broadcast a message letting different systems know that a new user has registered:

#!/usr/bin/perl

use strict;
use warnings;
use Redis;

# connect to redis
my $r = Redis->new();

# Broadcast the new user
$r->publish( "user.created",
             '{"email":"steve@example.com","username":"steve"}' );

This example assumes you have Redis running on the local system, and have the appropriate Perl library installed ("apt-get install libredis-perl").

The message which has been published contains the new username, and the associated email address of that user, under the topic user.created. Any other part of our system which cares about new users can listen for messages with that topic, and carry out actions.

Here is a corresponding listener which awaits user-creation events, and sends out the email to new users:

#!/usr/bin/perl

use strict;
use warnings;

use JSON;
use Redis;

sub new_user_created
{
    my ($message, $topic, $subscribed_topic) = (@_ );

    # Decode the message which is JSON
    my $obj = decode_json( $message );

    # Now send out the email ..
    print "Sending mail to " . $obj->{'email'} . "\n";
    print "New account is  " . $obj->{'username'} . "\n";

    # ...
}


# connect to the queue.
my $r = Redis->new();

# listen for messages
$r->subscribe( "user.created", \&new_user_created );

# loop forever
$r->wait_for_messages(5) while( 1 );

The key thing here is that the code which publishes the user-creation event neither knows, nor cares, what is listening for that. Providing it sends out the appropriate data along with the event, such as the new users email address, then it can be used elsewhere without any tight coupling.

With this in mind we can imagine a series of events such as "user.created", "user.destroyed", and "user.forgot_password".

Rather than having seperate processes listening for each event we'll just listen for the wildcard events "user.*" with the call to psubscribe:

#!/usr/bin/perl

use strict;
use warnings;

use Redis;

sub event_received
{
    my ($message, $topic, $subscribed_topic) = (@_ );

    print "The message was: $message\n";
    print "The topic was: $topic\n";
    print "We subscribed to the topic: $subscribed_topic\n";
}


# connect to the queue.
my $r = Redis->new();

# listen for messages - using a pattern
$r->psubscribe( "user.*", \&event_received );

# loop forever
$r->wait_for_messages(5) while( 1 );

This will output things like:

The message was: {"email":"steve@example.com","username":"steve"}
The topic was: user.created
We subscribed to the topic: user.*

The message was: {"username":"steve"}
The topic was: user.destroyed
We subscribed to the topic: user.*

As you can see we received both the topic under which the message was published, as well as the pattern we filtered against ("user.*" in our case) along with the actual message.

Using redis like this gives pretty good performance, but if you're looking at using messages like this in high-availability situations you'll probably prefer to use something that absolutely definitely won't drop messages, and which can work in a variety of contexts.

(Obviously for any kind of publish-subscribe system messages are not stored on disk, persisted, or saved. If you're not listening at the moment an event is transmitted to all listeners you don't get to see it.)

Popular queuing systems include RabbitMQ and ZeroMQ. If you want to be both low-level and simple then I'd recommend the nanomsg library which makes it very very simple to do interesting things with simple publish-subscribe queue, and supports many more modes of operation.

Although our code was Perl-specific there are many bindings and libraries available for Redis, and the coding should be roughly the same.

Hopefully despite the broad overview this introduction should be as usefual as our prior coverage of using the beanstalkd queue.

 

 


Re: A brief introduction to publish-subscribe queuing with redis
Posted by Anonymous (80.147.xx.xx) on Thu 18 Dec 2014 at 08:55
While redis pub-sub gives no guarantees about delivery, in my experience it does a pretty good job. In an app I've been sending messages via this method to three different data centers from a fourth for a few months now and according to my monitoring there's hardly any drops. So it's not as bad as it sounds, but drops can and will happen, but I'd gauge them at 1% and lower.

[ Parent ]