Wednesday, December 24, 2014

Service Checks and Mail Handlers with Sensu Monitoring

If you've kept up with this blog, by now you should know how to set up a logstash cluster and sentry monitoring. Today, we're exploring an alternative to Sentry. Both solutions have their advantages and disadvantages, but they have a good deal in common. Both have free versions, leverage ruby for their plugin ecosystem, and share many of the same dependencies. You'll have a smoother ride if you're already familiar with RabbitMQ, Redis, Graphite, InfluxDB, Logstash, and other open source tools.

We're going to start simple, and implement a system that notifies you when processes go down. We won't be setting up anything related to tracking metrics, just yet.

Prerequisites: You'll need at least one server for this guide. It's up to you if you want to install both Sensu client and server on the same machine or if you want them on separate instances. Then, please navigate over to this quickstart guide to set up Sensu.

Implementing Checks

Preparing the Client

Let's start by installing the necessary Sensu plugins.

# install sensu-plugin
sudo gem install sensu-plugin --no-ri --no-rdoc

# install git if you don't already have it
sudo apt-get install git

# pull down the repo
cd /opt
sudo git clone git://github.com/sensu/sensu-community-plugins.git
cd sensu-community-plugins/plugins

# copy the plugin over
sudo cp system/check-process.sh /etc/sensu/plugins

Next, we're going to update the subscriptions line of our Sensu client config by running sudo vi /etc/sensu/conf.d/client.json.


{
  "client": {
    "name": "server_name",
    "address": “sensu.server.ip.address”,
    "subscriptions": [ "nginx" ]
  }
}

You may now restart the client with sudo service sensu-client restart. Alright, that's it for the client. Let's move over to our Sensu server.

Preparing the Server

Create the Sensu Nginx config by running sudo nano /etc/sensu/conf.d/nginx.json and add the following contents:

{
  "checks": {
    "nginx_check": {
      "command": "/etc/sensu/plugins/check-process.sh -p nginx",
      "interval": 30,
      "subscribers": ["nginx"],
      "handlers": ["default", "email"]
    }
  }
}

As you can see, we've set the interval to 30 rather than letting it default to 60. This instructs the check to run every 30 seconds. Also, by using the check-process.sh plugin, we can abstract process checking into a command that accepts any string to pipe through ps aux. You also have the option to check by pid file with the -f flag. This comes in handy when names of different services collide (e.g. logstash and logstash-forwarder). You'll find your pid files in /var/run/. Lastly, we're preparing "email" as one of the handlers, even though we haven't configured it yet. We'll do that on the in the next section.

Setting Up Mail Handlers

Configuring the Server

SSH into your Sensu server and run the following commands:

# install an emailer
sudo apt-get install mailutils

# create the email handler configuration
sudo vi /etc/sensu/conf.d/handler_email.json
Add the following contents to the handler_email.json file you just created.
{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "mail -s 'sensu event' email@address.com"
    }
  }
}

Now restart the Sensu server with service sensu-server restart and run the following commands to make sure all of the sensu services run automatically:

sudo update-rc.d sensu-server defaults
sudo update-rc.d sensu-client defaults
sudo update-rc.d sensu-api defaults
sudo update-rc.d uchiwa defaults

Testing From the Client

Navigate to your Sensu client and stop nginx by running sudo service nginx stop. You should see the warning the in the dashboard and receive an message at the email you provided.

This is a screenshot of three servers I've set up. Intrepid is the Sensu server and the other two are the clients where I've shut off Nginx.

Now, all you have to do to get the notifications to stop is to start up your nginx servers by running sudo service nginx start.

As a final note, when utilizing any of the sensu-community-plugins, make sure to read the comments in the code. Some of them require specific parameters and if those aren't indicated in the usage section, you'll have to deduce them from the options.

No comments:

Post a Comment