Building a simple dashboard with redis and the node.js server.

Posted by Steve on Fri 18 Jan 2013 at 08:49

Tags: , ,

Recently we introduced redis, and the built-in data-types it has. Taking that introduction a step-further we're going to present a simple "dashboard" written using a combination of Redis and node.js.

The basic goal of a dashboard is to show you interesting activity in real-time. That interesting activity might be simple text strings describing events, load graphs, or even sales-counts. If you're running a cluster of some sort a dashboard is very useful, because it allows you to see events that have happened globally - rather than logging on to each machine in turn to pull out metrics and ensure all is well.

Once you realize that you want to combine events from multiple machines it becomes obvious that you need to listen on the network, somehow, to accept the submission of "interesting events". In the past I've used various things to run simple servers, from Apache + CGI scripts, to sinatra, and most recently node.js.

If you've not yet heard of node.js it can be explained as event-driven server-side javascript. It is designed to allow you to write highly-scalable software, because everything is based on events and callbacks.

Unfortunately the biggest downside to using node.js is that it is still a very young project, with semi-regular updates. There is a Debian GNU/Linux package for node, but it is available only in the unstable distribution - and as of January 2013 even that is out of date (#692312).

For my own personal systems I track recent releases and install directly into /opt. Every time a new release is issued I update the system, and I've not yet found this too painful a prospect.

The most recent release right now is v0.8.17, although that might change if you're reading this piece in the future, and it may be installed by downloading the source and building like so:

# cd /opt
# wget http://nodejs.org/dist/v0.8.17/node-v0.8.17.tar.gz
# tar zxf node-v0.8.17.tar.gz
# cd node-v0.8.17
# ./configure --prefix=/opt/node-v0.8.17 && make install

The end result of this build will be a binary installation located in /opt/node-v0.8.17, but to avoid having to deal with version numbers it makes sense to create a symlink:

# cd /opt
# ln -s node-v0.8.17 node

Once you've created this symlink you may then refer to the node.js binary, regardless of the version, with the path /opt/node/bin/node - and this means you never need to update your PATH, or your scripts, more than once. (This approach also allows you to roll back to previous versions if you ever need to.)

We're going to be using node.js to listen upon the network, and receive the "interesting event" messages. The obvious question is then "How do we store these messages?" To that the answer is redis. The reason for choosing redis is that it is fast, both for storing the messages and retrieving them.

When you consider what you're going to do with a status page there are two obvious cases you need to handle:

  • Adding a new event to the list of events.
  • Retrieving the list of past events.

As we've seen with our Redis introduction adding items to a list is trivial, and so are both receiving the list, and truncating the list. We're going to need to a library which allows us to interface with Redis from node.js, and happily interfacing with Redis is possible using this library.

The next decision is how to receive the messages? I chose to use UDP because although delivery isn't guaranteed it is "good enough" in practice, especially within a private network. There is low overhead to sending UDP messages, and such things can be done from many many languages and environments - this means that our event-emitters don't also need to be written in node.js, and can be perl, ruby, or anything else that is capable of sending a UDP packet.

The core of the service is simple:

  • Open a UDP socket for listening.
  • Every time a message is received append that to the list of "current events".
    • Truncate the list to 1000 entries, or some similar figure, to avoid growing indefinitely.
    • We assume the viewing of the interesting events is going to focus on the most recent anyway.

Writing such a service in node.js is almost trivial, and the resulting code is easily understandable if you're familiar with javascript.

To run the code you'll need the script referenced above, as well as the client library for talking to Redis. I've bundled both together into a repository you can checkout from github:

# git clone https://github.com/skx/dashboard.git
Cloning into dashboard...
remote: Counting objects: 70, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 70 (delta 4), reused 69 (delta 3)
Unpacking objects: 100% (70/70), done.
#

Once you've checked out the code launching it is as simple as:

# /opt/node/bin/node dashboard.js
server listening 0.0.0.0:4433

The banner will show you that the server is listening upon :4433, and it will insert any events it receives into the Redis database running upon the localhost. Submitting a new event is a good test of the code. So save this as "submit.pl":

#!/usr/bin/perl

use strict;
use warnings;

use IO::Socket;

my $msg = shift( @ARGV );
exit(0) unless( defined( $msg ) && length($msg) );

my $sock = IO::Socket::INET->new( Proto    => 'udp',
                                  PeerPort => 4433,
                                  PeerAddr => "localhost",
                                ) ;

$sock->send( $msg );
exit(0);

With this code you can now submit some sample events:

# perl submit.pl "This is a test"
# perl submit.pl "So is this"

The end result is that you'll have two entries submitted to the Redis list "dashboard", which you can verify with the redis-cli tool:

#redis-cli lrange dashboard 0 -1
1. 127.0.0.1#Thu Jan 17 2013 18:46:39 GMT+0000 (GMT)#So is this
2. 127.0.0.1#Thu Jan 17 2013 18:46:35 GMT+0000 (GMT)#This is a test

As you can see the messages have been inserted along with the source-IP, and the date/time-stamp. Both of these have been deliminated with the hash ("#") character.

Now that the core of the service has been written we're in good shape:

  • We can listen for "events", and insert them into a redis store.
  • We can submit events.

From here we're nearly done, all we need is a web-page that will show the most recent 100 events, or so. This can be carried out via a simple Perl CGI-script with ease. Something like this will suffice:

#!/usr/bin/perl -w

use strict;
use warnings;
use Redis;

print <<EOH;
Content-type: text/html


<html>
 <head>
  <title>Recent Events</title>
 </head>
 <body>
EOH
my $redis  = new Redis();
my @values = $redis->lrange( "dashboard", 0, 100);

#
#  Process each event.
#
foreach my $event (@values )
{
    my $ip;
    my $time;
    my $msg;

    if ( $entry =~ /^([^#]+)#([^#]+)#(.*)$/ )
    {
        $ip = $1;
        $time = $2;
        $msg = $3;
    }

    print "$ip - $time - $msg\n";
}
print <<EOF;
  </pre>
 </body>
</html>
EOF
;

This CGI script talks to redis directly to receive the most recent 100 events, and then dumps them out in a simple fashion. A mort artistic person than myself might use a pretty table, and nice CSS to make it display well. For bonus points you could invoke the CGI script via AJAX to give yourself a constantly updating status page.

 

 


Posted by Anonymous (108.93.xx.xx) on Thu 24 Jan 2013 at 17:08
IMO, some screen shots of the dashboard would do wonders for your article.

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Thu 24 Jan 2013 at 20:49
[ View Steve's Scratchpad | View Weblogs ]

The presentation is left a little vague because that is left to the user - right now I just use a HTML-table to output the prior events, as demonstrated in the application in the github repository.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (99.100.xx.xx) on Fri 25 Jan 2013 at 06:52
The sudden switch to perl seemed a little bizarre. I mean, if you're going to write most of it in perl, why not write the rest of it in perl too. A little perl web server that serves the dashboard is only a few lines of code.

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Fri 25 Jan 2013 at 09:30
[ View Steve's Scratchpad | View Weblogs ]

It's a weee example, and in perl because it is readable, self-contained, and obvious. (Not to mention I like perl!)

The interesting bit to introduce here was using redis for real, and getting started with node. The actual presentation is the obvious thing that needs to be written more properly, using ajax to reload, etc.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (90.193.xx.xx) on Fri 25 Jan 2013 at 16:54

I've replaced the viewer with a Sinatra application that will spit out JSON, and a jQuery-powered index page to consume the same.

There is a screenshot in the repository, which doesn't demonstrate how the dashboard updates in real-time. Still should be easier for others to visualize.

[ Parent | Reply to this comment ]

Posted by Eirik (80.212.xx.xx) on Sat 9 Mar 2013 at 04:55

Rather than just using a symlink, I recommend using xstow:

# install xstow
sudo aptitude install xstow
#This to aovid having to make install as root
#node doesn't accept/respect prefix=/x/z to make install, but rather
#needs DESTDIR to be set -- with this, you avoid copying files to
#eg. /usr/local by accident
sudo mkdir -p /usr/local/xstow/node-v0.8.22
sudo chown <yourid> /usr/local/xstow/node-v0.8.22
# Get and build node: - You can do this anywhere, eg in $HOME
wget <node-vx.y.z.tar.gz>
tar xzf node-v0.8.22.tar.gz
cd node-v0.8.22
./configure prefix=/usr/local #Should be default
make
make install DESTDIR=/usr/local/xstow/node-vx.y.z
#node doesn't quite understand what we want -- so clean up the
#reduntant usr/local-tree:
pushd /usr/local/xstow/node-v0.8.22
mv usr/local/* .
rmdir usr/local usr
#Optional - but we probably don't want the various node-files to
#be owned by a regular user:
sudo chown root:root -R node-v0.8.22
#Create symlinks under /usr/local - will also work with libs, manpages etc.
sudo xstow node-v0.8.22
# test eg:
man node
node -v
# clean up:
popd
cd ..
rm -r node-*

You should now already have nodejs in your path, linking against any libraries (not used AFAIK for nodejs, but many other packages) and man/info-pages should just work.

Most packages are less work (just ./configure prefix=/usr/local && make && sudo make install prefix=/usr/local/xstow/package-version && cd /usr/local/xstow && sudo xstow package-version) -- the benefits over manually symlinking is that manpaths, path and library paths should work out of the box.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 496 votes ~ 5 comments )