Sunday, November 16, 2014

Setting Up Sentry Monitoring on an Ubuntu Server

What is the most popular monitoring solution on the market, today? Here's a hint: It's the first monitoring solution to ever exist and hasn't been updated in centuries. That's right. It's your clients. So unless you like finding out why your software is broken through angry emails, bite the bullet and implement an automated solution.

Setting up Sentry for the first time can be pretty daunting. The steps I've highlighted below take into account the various snags you'll hit along the way, using the Quickstart guide.

Prerequisites: Make sure to check out the Sentry Quickstart, if you'd like to see some official documentation. This guide was written for Linux Ubuntu 14.04 users. You can also carry out these steps if you're on a machine that can ssh into an Ubuntu instance or you have an Ubuntu instance running in VirtualBox.

Setting Up the Ubuntu Instance

The following guide was written with a fresh Ubuntu 14.04 instance.

# make sure you're signed on as the root user
su - root

# create the sentry user
adduser sentry

# add sentry to sudoers
visudo

Add a line for sentry in the sudoers file

# User privilege specification
root    ALL=(ALL:ALL) ALL
sentry    ALL=(ALL:ALL) ALL

Continue running the following commands in the shell.

# update packages
apt-get update; apt-get dist-upgrade; apt-get autoremove

# install all of the dependencies
apt-get install libpq-dev python-setuptools make python-virtualenv

reboot

Sentry Set Up

su sentry
sudo mkdir -p /var/www/sentry
sudo chown sentry /var/www/sentry/
sudo virtualenv /var/www/sentry
source /var/www/sentry/bin/activate

sudo easy_install -UZ sentry[postgres]

Warning: If you get the error, Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1, Just run sudo apt-get install libxml2-dev libxslt1-dev libevent-dev libffi-dev python-dev

Continue running the following commands in the shell:

sentry init # will create config at ~/.sentry/sentry.conf.py

# Edit this file
sudo nano ~/.sentry/sentry.conf.py

Add the following contents:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'sentry',
        'USER': 'postgres',
        'PASSWORD': '',
        'HOST': '',
        'PORT': '',
        'OPTIONS': {
            'autocommit': True,
        }
    }
}

# set the following line if you have a subdomain set up
# No trailing slash!
SENTRY_URL_PREFIX = 'http://sentry.example.com'

SENTRY_WEB_HOST = '0.0.0.0'
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    'workers': 3,  # the number of gunicorn workers
    'secure_scheme_headers': {'X-FORWARDED-PROTO': 'https'},
}

EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_PASSWORD = 'mypassword'
EMAIL_HOST_USER = 'notify@mydomain.com'
EMAIL_PORT = 587
DEFAULT_FROM_EMAIL = 'notify@mydomain.com'

# Add this to the very bottom of the file
# if you have a subdomain set up
ALLOWED_HOSTS=["sentry.example.com"]

Redis

# set up redis
cd /opt
sudo wget http://download.redis.io/redis-stable.tar.gz
sudo tar xvzf redis-stable.tar.gz
cd redis-stable
sudo make
sudo apt-get install tcl8.5
sudo make test # the last step will especially take a while
sudo make install

# running redis as a daemon
cd utils
sudo ./install_server.sh

Note: If you have any issues with the requests library, please see this thread. Thanks to tuxcayc in the comments.

A series of prompts will follow. Select all of the defaults by hitting multiple times.

# start the redis server
sudo service redis_6379 start

# set redis to run on init
sudo update-rc.d redis_6379 defaults

# testing to make sure redis is running
redis-cli ping
#PONG

sudo netstat -tanpu|grep redis
# tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      8666/redis-server *
# tcp6       0      0 :::6379                 :::*                    LISTEN      8666/redis-server *

Now make sure to return to ~/.sentry/sentry.conf.py and uncomment some lines in the Queue and Update Buffers sections.

###########
## Queue ##
###########

# See http://sentry.readthedocs.org/en/latest/queue/index.html for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

# You can enable queueing of jobs by turning off the always eager setting:
CELERY_ALWAYS_EAGER = False
BROKER_URL = 'redis://localhost:6379'

####################
## Update Buffers ##
####################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

# You'll need to install the required dependencies for Redis buffers:
#   pip install redis hiredis nydus
#
SENTRY_BUFFER = 'sentry.buffer.redis.RedisBuffer'
SENTRY_REDIS_OPTIONS = {
    'hosts': {
        0: {
            'host': '127.0.0.1',
            'port': 6379,
        }
    }
}

Then, run the following command:

# install dependencies
sudo apt-get install python-pip
sudo pip install redis hiredis nydus

Postgres

# install postgres
sudo apt-get install postgresql postgresql-contrib

# install postgres adminpack
sudo -u postgres psql
CREATE EXTENSION "adminpack";
\q

# change postgres password & create database
sudo passwd postgres
sudo su - postgres
psql -d template1 -c "ALTER USER postgres WITH PASSWORD 'yourpassword';"
createdb your_sentry_db_name
createuser your_sentry_user --pwprompt
psql -d template1 -U postgres
GRANT ALL PRIVILEGES ON DATABASE your_sentry_db_name to your_sentry_user;
\q
exit

Now make sure to return to ~/.sentry/sentry.conf.py and update the following:

Revisiting the Sentry Config

# Update
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'your_sentry_db_name',
        'USER': 'your_sentry_user',
        'PASSWORD': 'your_password',
        'HOST': 'localhost',
    }
}

Continue running the following commands:

sudo pip install psycopg2

# set up database
sentry upgrade

# create a new user
sentry createsuperuser
# run the automated repair script
sentry repair

# if you created the configfile somewhere other than ~/.sentry/sentry.conf.py...
# ...add --config=path/to/sentry.conf.py flag to two above commands

# start the server
sentry start

# start the workers in a new panel
sentry celery worker -B

Nginx

You'll want to set up nginx to serve the sentry client, as well. Check out this guide and uncomment the proxy pass server block to point it to the sentry instance running on port 9000. You'll also want to replace all instances of "domain.com" with the domain you're using to serve sentry. Our examples point to sentry.example.com. If you want to use a subdomain like this, make sure you set up a DNS record, according line. Otherwise, it can just be a www.example.com or even running on its own port.

If you want to proxy a domain to the app, add the following to /etc/nginx/sites-available/default. Credit goes to github user jqlblue.

It's a good idea at this point to set up supervisor or any alternative to keep sentry running at all times. Also, remember to create an upstart script to start everything up on boot. When everything is working properly, head over to this guide to set up your first team / project.

14 comments:

  1. Great guide, everything worked fine.

    I had just one issue when installing the redis python package which was caused by `requests`, the issue and workaround is described here: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1306991/comments/9

    (Ubuntu 14.04 x86_64)

    ReplyDelete
    Replies
    1. Thanks tuxcayc. I appreciate the feedback and will update the article.

      Delete
  2. Warning: If you get the error, Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1, Just run sudo apt-get install libxml2-dev libxslt1-dev libevent-dev python-dev libffi-dev

    added libffi-dev

    Thanks for the post

    ReplyDelete
    Replies
    1. Thanks. I updated the article with your addition.

      Delete
  3. Is there any way I can upgrade to a latest version of Sentry?

    ReplyDelete
    Replies
    1. Hey Karthik. I feel like you should be able to. Have you had any problems doing so?

      Delete
  4. For the nginx config, I'm getting "Starting nginx: nginx: [emerg] pcre_compile() failed: unrecognized character after (?P in "/api/(?P\d+/)?store/" at "\d+/)?store/" in /etc/nginx/sites-enabled/sentry:27"

    ReplyDelete
    Replies
    1. This one works https://gist.github.com/jqlblue/8019629

      Delete
    2. Thank you. I updated the article.

      Delete
  5. 1. On running `sudo easy_install -UZ sentry[postgres]`, I'm getting `mock requires setuptools>=17.1. Aborting installation, error: Setup script exited with 1`. I have to do `wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python`.

    2. On running `sudo pip install redis hiredis nydus`, I'm getting `ImportError: cannot import name 'IncompleteRead'`. I have to remove python-pip & install with easy_install following http://stackoverflow.com/questions/27341064/how-do-i-fix-importerror-cannot-import-name-incompleteread

    ReplyDelete
    Replies
    1. Thanks natsu902. I'll make sure to update the article with your feedback.

      Delete
  6. Thanks a lot!
    The 'failed with exit'-error might also come when your machine has too less memory. Simple fix by adding some swap: http://stackoverflow.com/a/26762938/1149339

    ReplyDelete
  7. Documentation for sentry has moved. The 'on premises' docs are here:

    https://docs.getsentry.com/on-premise/

    ReplyDelete
    Replies
    1. Thanks for pointing this out, GJKB. Updated.

      Delete