The Angel Network Monitor

30 Dec 2005, posted on angel

Introduction

Angel is a simple yet useful tool to monitor the services on your network. Technically speaking, it’s a Perl program that runs every ‘n’ minutes (usually fired from your cron) and calls different perl subprograms (referred as “plugins” from now on) to do the actual testing. It will then generate an HTML table containing the status of your network.

Download the current version (0.7.3)

Sample Output

Some people like to “see what they’re going into” :) before they download something. Click Here to look at a real life output from Angel (Note: LEDSign was not enabled at the time of this output).

Main Features

Administration is centralized. Adding hosts and/or services involves editing only one file, usually.
The program uses the ‘plug-in’ concept, meaning it’s easy to customize and enhance.
HTML output. You can monitor your hosts/network from anywhere
Support for the excellent LEDSign Java applet.
Javascript support. The browser will popup a child window with alerts, if you choose this option
SSH Support.
It’s Free :)

Inspiration for this program (and for the name) came from the Excellent “Big Brother” package. Big Brother is “watching you”, just like your “Guardian Angel”. :)

Program Basics

The core of the program is the “angel” perl script. It’s meant to be run from your cron every “N” minutes. Once run, the program will read the conf/hosts.conf and conf/angel.conf files, test the specified conditions and generate the HTML output containing the status of your hosts, detailed error messages (returned from your plug-ins, etc). Introduction

Installation Procedures

Finding a place for the files

The first step (as always) is to find a suitable place for Angel. It’s mostly a matter of personal taste, and Angel will make its best to find its “home” directory and proceed accordingly. I like putting custom things under the “/usr/local directory”. If you do not object to that, you could do:

mkdir /usr/local/angel
cd /usr/local/angel
tar zxvf angel.tar.gz

I’m also supposing your file is called “angel.tar.gz”. This will almost always not be true. I have the habit of appending the version number to the tar file.

Configuring your web server

Did I tell you that you need a Web Server to run Angel? Well, that’s almost the whole truth. Angel generates HTML code. If you don’t have a Web Server on the computer Angel is running, you won’t be able to see the output from the outside world (from other computers, I mean). Of course, if your workstation runs Linux, you won’t need an HTTP Server (Just fire your browser and point to the place where Angel writes its files).

However, it would be healthy to have a running server, since it allows you to watch your servers from anywhere. For now, I suppose you’re running Apache.

What you have to do is configure your Apache server so you can “see” the files generated by Angel more easily:

Edit your “httpd.conf” file (location varies depending on your linux distribution. Popular locations for this file are “/etc/apache” or “/etc/httpd”).
Add the line: “Alias /angel /usr/local/angel/html” to this file
Restart your webserver.

From now on, you should be able to see the Angel output pointing your browser to: “http://yourwebserver/angel/”

Configuring Angel

Before you run the Angel, it would be a good idea to take a quick look at the “conf/angel.conf” file. Some default variables can be changed to modify the behaviour of the program. The most important ones are:

$main::Indexhtmlheader
$main::Indexhtmlfooter

These variables control the main HTML file generation. Indexhtmlheader is sent to the html file before anything else. Indexhtmlfooter is the last thing sent to the file. If you modify these variables, please keep the credits (including my name) somewhere.

$main::Indexcolumnwidth

Angel uses the HTML TABLE construct to format its output. This variable controls the default table width (in pixels) for each element. If you change this value, please remember that your red, yellow, green and black gifs should fit inside the table cell! If you have a large number of services, it could be a good idea to reduce this value. Please remember that the total width of your table is (Number of services + 1) multiplied by the Indexcolumnwidth. Currently, this value is also used for the host part of the table (yes, it’s not ideal, I know). Setting this variable to “-1” will cause Angel not to generate width info at all in the html output.

$main::Indexhtmlborder

How thick (in pixels) should be the table borders. Some people love table borders, some hate. The default is 0, or, no border.

$main::Errorhostwidth
$main::Errorlabelwidth
$main::Errorcolumnwidth

Angel will generate an html file containing all the errors found on the last cycle. The table columns contain: 1) The host name, 2) The label (service) where the error was found and 3) The error description. These variables control the width of each of these columns.

$main::Use_ledsign
$main::Ledsignhtmlheader

If you set “$main::Useledsign”, Angel will include support for the excellent Ledsign Java Applet from Darrick Brown. $main::Ledsignhtml_header contains the applet call. Do not change this unless you know exactly what you’re doing. Please, consult the included Ledsign package for more details.

$main::Remote_cmd

This tells the plugins which program should be used to do remote execution. Linux normally uses rsh. HPUX uses remsh and SCO rcmd. It’s a good idea to enable SSH on your site and use SSH, since the “r” commands are insecure by nature.

Editing Your Configuration File

Everything Angel does is based on the contents of the “conf/hosts.conf” file. Following the good Unix tradition, this is a pure ASCII file and should be edited using your preferred editor. Every host and service you want to monitor must be listed here.

For each key pair host/service, there must be one line in the configuration file. If you want to check, say SMTP and HTTP from the same host, you must have two different lines in the configuration file.

The basic format of this file is:

hostname:plugin:plugin_options:label:angel_options

Where:

hostname: Name of the host to be checked. This host must be locatable from the computer where angel is running.
plugin: The plugin name. Each plugin does a different thing and require different “pluginoptions” (see next definition). Currently (v.0.5) the only defined plugins are Checktcp, Checkdisk and Checkload.
plugin_options: These options are passed directly to the plugin. You must check the plugin documentation, since each plugin requires specific options.
label: This one deserves some explanation. Angel will always create a table containing hostnames in the rows and labels in the columns. But what is a “label”? Usually a label is a service. That’s it. I called it “label” just because it can be any string you wish. Internally, the program will group using the pair hostname/label. For instance, you may choose to label your http column as “Web” to make it easier for non-computer people to understand the program output. Which service is effectively checked depends on what you pass to the plugin using the “plugin_options” field.
angel_options: These options control some aspects of the core Angel script. They are not related to the plugins and apply equally to all plugins. Currently, the valid options are “alertyellow”,”alertred”,”alertblack”. These options will cause Angel to generate javascript code which in turn will open an auxiliary window in your browser, warning you when a yellow, red or black condition happen, respectively. Options are separated with a “!” (bang) sign.

It’s a good idea to check the “hosts.conf.example” file that comes with this package to make sure you understand the syntax. Also, check the Plugin specific documentation below to customize Angel to your needs.

Plugin Specific Documentation

This section discusses the basic options for each plugin. This is where you should expect to find the “plugin_options” for each plugin. Please, click on the desired plugin to obtain more information.

The following plugins come bundled with the angel package:

Check_tcp - Checks for a TCP connection on a given port
Check_disk - Checks for low diskspace conditions
Check_load - Checks for high system load conditions
Check_ping - Checks roundtrip times

If you decide to write your own perl plugin:

Please take a look at the existing plugins to understand how things work. Mail me if you have any questions about how the plugins interact with the main program.
After your plugin is complete, please send me a copy so I can include it on the next Angel release (with the due credits, of course).
If you feel like it (it would be veeeery cool), download one of the HTMLs containing the documentation for my plugins and modify it to contain your documentation. I will try to maintain the same HTML format for all plugins.
If you complete any of the previous steps, my sincere thanks. :)

Things to do

As you know, a software project, no matter how small, generates a lot of work. In fact, only this documentation took almost the same time as the code. Count the debugging sessions (and the fact that I’m doing this in my spare time) and you’ll understand why bugs are very likely to exist.

If you feel like helping, know that I would love to receive emails with comments, corrections and ideas for Angel. Please email me at paganini@paganini.net.

Here’s a (partial) list of things I know should be changed or improved:

Perl wizards would throw up if they look at the code. I’m going to MODularize this whole thing some day. Please note that this version loads the “plugins” on demand. That’s why I just didn’t use modules.
I need to find a better way to store the program defaults. Today, they’re hardwired in a separate while, which is “require()d” by perl at runtime.
The configuration file sucks. It’s hard to read, but easy to parse. I need to do some research on CPAN and find a better way to do this.
The program uses the same table width for everything, including the labels. That’s clearly stupid.
Sometimes, Netscape refuses to show things on the small (javascript generated) window. I don’t know why this happens (looks like a Netscape problem). In these cases, please restart netscape.
I need an SNMP plugin. That would rule…
I still need to think if this task is more suitable for a C program. Perl eats just too much memory. That’s why I didn’t add a “Daemon mode” to Angel.
The program accepts no command line parameters. That would be handy. I also need to add some “debug” hooks to the program.
My english sucks… (I’m not a native english speaker, so please, be nice). If you find something wrong with my english, please email me. :)

News and Updated Info

Version 0.7.3 (26/Nov/2002)

This version was brought to you thanks to the coolness of Matt A. Callihan. The fixes are:

Changed hrheader sorting to place PING in the first column.
Fixed Check_Ping.pl to work with iputils-ss020124 and beyond.
There’s now a “Setup” script that should help people with the installation.

Version 0.7 (May/17/1998)

Lots of bugs corrected. The main code is now much cleaner. We got rid of tons of unused variables, etc.
The program now works fine without “angel options” in the configuration file (i.e. you may choose not to have alert**, etc).
The configuration data has been moved to another file, called conf/angel.conf. It no longer lies inside the main program.
Support for other “remote execution” commands has been added. Just set the variable $main::Remote_cmd in conf/angel.conf.
The programs now use the “English” module.
A new plugin was born: Check_ping. It’s useful to check how loaded are your lines or if you have lag…
Check_disk now supports AIX, SunOS, IRIX, Solaris and others.

Version 0.6a (Apr/04/1998)

This is just a small modification to the home page itself, nothing in the program. Since the tar.gz carries a perfect image of the page, I decided to release a minor version.
The home page now has output samples and the ability to warn you when something changes. Courtesy of NetMind.

Version 0.6 (Mar/18/1998)

I finally had the time to edit the code and kill the infamous alarm() bug. It caused a SYSSEGV on certain conditions, inside the Check_tcp plugin.
Check_tcp now uses the IO::Socket module from Graham Barr.
The “docs” subdirectory is now under the “html” tree. It’s now possible to access the documentation using http://your.angel.url/docs

Special Thanks

My special thanks and credits to:

Norbert Gruener (nog@MPA-Garching.MPG.DE) - for his ideas, improvements and bug corrections. Thanks to Norbert, you’re going to see a whole different version of Angel soon.
Philippe Charnier (charnier@xp11.frmug.org) for many patches to the Check_disk plugin.
Adam Soltan (soltan@speech.kth.se) for patches to the Check_disk plugin.
Michael Schilli (schilli@tep.e-technik.tu-muenchen.de ) - for the great Proc::Simple perl module.
MacLawran Group for “Big Brother”. Big Brother gave me the idea for Angel (and for the “Angel” name too. :))
Graham Barr for IO::Socket. It’s much better than the regular socket module.