Wednesday, December 24, 2014

Graphite Installation with UWSGI & Nginx

In this guide, we're exploring Graphite, a metrics core that allows you to collect numeric time-series data and generate real-time visualizations. Graphite describes itself as a metrics core because it is one component of a larger system of monitoring/analysis tools. As you may have guessed from the name, Graphite is great at generating graphs. This goes back to time series data, or data where you are tracking values over time. Some examples of time series data include CPU usage, bandwidth, and disk usage, given you are graphing them over a length of time.

Moving on, Graphite is commonly used in conjunction with notification systems like Sentry or Sensu, rich graphing dashboards like Grafana, and data aggregators like StatsD. We won't be going into all of those integrations, just yet. We'll start by setting up with Graphite, first, and then explain how it fits into the overall architecture in a follow-up article.

Prerequisites: Head over to graphite.readthedocs.org/en/0.9.12 if you'd like to learn more. You'll also want to set up an Ubuntu server instance prior to carrying out these steps and set that box up with Nginx.

Preparing the Machine

# update packages
sudo apt-get update

# install python and graphite dependencies
sudo apt-get install -y python python-dev python-virtualenv libevent-dev python-pip python-cairo python-django-tagging python-twisted python-memcache python-pysqlite2

# install web server
sudo apt-get install -y nginx uwsgi uwsgi-plugin-python

# install postgresql and dependencies
sudo apt-get install -y postgresql libpq-dev python-psycopg2

Configuring Nginx

A lot of tutorials you find online about setting up Graphite with some sort of monitoring solution usually leverage Apache as a web server. Since a lot of my tutorials use Nginx over Apache, we're going to use our existing Nginx server over a separate Apache instance. Return to the prerequisites section if you haven't set up Nginx, yet.

# edit the file
sudo vi /etc/nginx/sites-available/default

# add the following server block
server {
  listen                8080 ;

  access_log            /var/log/nginx/example.access.log;
  error_log            /var/log/nginx/example.error.log;

  location / {
    include uwsgi_params;
    uwsgi_pass 127.0.0.1:3031;
  }
}

Configuring UWSGI

uWSGI is a server that implements the WSGI/uwsgi, or Web Server Gateway Interface, protocol. WSGI aims to specify a universal interface between web servers and Python applications.

# create a new file
sudo vi /etc/uwsgi/apps-available/graphite.ini

# add the following contents
[uwsgi]
processes = 2
socket = 127.0.0.1:3031
gid = www-data
uid = www-data
virtualenv = /opt/graphite
chdir = /opt/graphite/conf
module = wsgi:application
sudo ln -s /etc/uwsgi/apps-available/graphite.ini /etc/uwsgi/apps-enabled/

Configuring Postgresql

# database
sudo -u postgres psql

CREATE USER graphite WITH PASSWORD 'password';
CREATE DATABASE graphite WITH OWNER graphite;
\q

Creating a Graphite User

# create a new user
sudo adduser graphite

# add user to sudoers
su - root
visudo

# add an entry for the graphite user
# User privilege specification
root    ALL=(ALL:ALL) ALL
graphite    ALL=(ALL:ALL) ALL

# log in as graphite user
su graphite

Configuring Graphite

When you install Carbon for the first time, you'll get a prompt to remove the database files on purge. Choose "No". Anytime you need to start fresh, you can just manually remove the files in the /var/lib/graphite/whisper directory. If you accidentally answer "Yes", just run sudo apt-get purge graphite-carbon, and then run the installation all over again.

cd /opt; sudo mkdir graphite; sudo chown graphite graphite; sudo chgrp graphite graphite
virtualenv graphite; source graphite/bin/activate

# install graphite and dependencies
pip install carbon graphite-web whisper 'Twisted<12.0'  django==1.5 simplejson psycopg2 django-tagging

cd /opt/graphite/conf/
sudo mkdir examples
sudo mv *.example examples

sudo cp examples/storage-schemas.conf.example storage-schemas.conf 
sudo cp examples/storage-aggregation.conf.example storage-aggregation.conf 
sudo cp examples/carbon.conf.example carbon.conf 
sudo cp examples/graphite.wsgi.example wsgi.py

Graphite Local Settings

# copy the local_settings.py file and edit
sudo cp /opt/graphite/webapp/graphite/{local_settings.py.example,local_settings.py}
sudo vi /opt/graphite/webapp/graphite/local_settings.py

# uncomment the following lines
SECRET_KEY = 'replace_with_string_from_any_secret_key_generator'
TIME_ZONE = 'America/New_York'
USE_REMOTE_USER_AUTHENTICATION = True
DEBUG = True

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'graphite',
        'PASSWORD': 'yourgraphiteuserpassword',
        'HOST': '127.0.0.1',
        'PORT': ''
    }
}

As you can see above, we've set our SECRET_KEY, TIME_ZONE, USE_REMOTE_AUTHENTICATION, and DEBUG variables. Google "secret key generator" if you want something a bit more secure. We populate the DATABASES object with . Uncomment the DEBUG line if you have trouble serving the graphite web client later on.

# type ‘yes’ when prompted to create a user
sudo /opt/graphite/bin/python /opt/graphite/webapp/graphite/manage.py syncdb

sudo chown -R www-data:www-data /opt/graphite/webapp/ /opt/graphite/storage/

Cairo

We now have to link virtualenv to the dist cairo because pip installing on 2.7 is finicky

# link
sudo vi /opt/graphite/lib/python2.7/site-packages/dist-packages.pth

# add this line to the file
/usr/lib/python2.7/dist-packages

Restart All Services

sudo /opt/graphite/bin/python /opt/graphite/bin/carbon-cache.py start
sudo service nginx restart
sudo service uwsgi restart

And finally, navigate to http://<ip.address.of.vm>:8080, and you should be greeted by the graphite dashboard:

If you run into any issues, make sure you are tailing your uwsgi log while you restart uwsgi with sudo tail -f /var/log/uwsgi/app/graphite.log. You'll be able to tell through that if you're missing dependencies or uwsgi can't find your python application. The following is a list of some common errors:

Error Solution
ImportError: No module named psycopg2.extensions You need to run pip install psycopg2(in your virtualenv)
no python application found, check your startup logs for errors Double check your /etc/uwsgi/apps-available/graphite.ini and make sure it matches the example under the Configuring UWSGI section

If you get any rendering issues in your browser or in the console (when navigating to http://<URL>/render, for instance, because you set DEBUG to true in your /opt/graphite/webapp/graphite/local_settings.py file, you should see why in your browser. "No module named cairo" is a popular issue here. To solve that, make sure you've installed cairo. See Cairo section, above.

27 comments:

  1. Hey!
    Thanks for this tutorial -- really appreciate it!

    However; I've got one problem with getting it to work properly. I've literally copy and pasted every command and changed the necessary lines as you've said, but I get this when I try visit http://ip:8080
    https://gist.github.com/xcpop/f1e658843bb7e55dad7f

    I have tried to Google these issues but haven't found anything that would work :/
    Would love some help, thanks :]

    ReplyDelete
    Replies
    1. Thanks for the positive feedback! Have you tried restarting postgresql and monitoring the logs? Also, are you performing these steps on a virtual machine, fresh server, or an existing server with other services running on it?

      Delete
    2. Hi Wildling.
      Did you find a solution for this?

      Delete
  2. Thanks for replying! :]
    I've tried restarting literally every service related to this -- even the server itself, I still have the same problem unfortunately.
    I'm also performing this on a DigitalOcean VPS,
    Description: Ubuntu 14.04.1 LTS
    Release: 14.04
    Codename: trusty
    1GB Ram. It was a complete fresh server when I started with this tutorial.

    I've yet to try this on a dedicated machine but I plan to, once I've got everything working.

    ReplyDelete
    Replies
    1. I'll go through the steps again, tonight and see about reproducing the issue for you. We'll get this sorted out, ASAP.

      Until then, since you're familiar with the manual execution of these steps, it's worth looking into docker, which automates these steps similar to a bash script and provisions isolated containers for you. Much better than having to repeat these steps over and over per server. I've been meaning to write guides about docker, but until then, definitely read up on it. Check out this: https://docs.docker.com/userguide/#getting-started-with-docker-hub, watch videos, and look at example repos like https://github.com/nickstenning/docker-graphite and you'll see a lot of familiar steps in the Dockerfile.

      Delete
    2. Hey Wildling, I was able to reproduce the issue by downgrading my version of django. See more details here: https://github.com/celery/celery/issues/621. Can you verify you're using django 1.5? Hope this helps.

      Delete
    3. I will definitely look into Docker! Seems really neat :]

      Also, my Django version was actually 1.6.1 before, even though I followed the tutorial line by line.. But I just downgraded to 1.5.0 and I'm sadly still having these issues :/

      Delete
    4. So weird that your version of Django was different. Maybe your version of psycopg2 is incompatible, even though I didn't explicitly indicate you need a specific version. Really sucks that it's still giving you issues. While you're looking at docker, I'll try to reproduce the issue a different way. I still have some ideas. I'll get back to you, soon.

      Also, give this a shot - https://gist.github.com/1304815/85928100c92bb7948db964f046bd8b5b16a0a65a

      Delete
    5. Hey Wildling. Didn't forget about you.

      Please see this video: https://drive.google.com/file/d/0B5fLVqQY7CdRODZlVHJtUFhhR00/view?usp=sharing

      Delete
  3. Really nice tutorial! But I have an issue. When I run the nginx, uwsgi and carbon, and I connect to myip:my port it says "Python application not found". If I look in the log file of uwsgi it says: ImportError: No module named wigs
    . How can I solve that?

    Thank you!!

    ReplyDelete
    Replies
    1. Thanks. Real quick, does it say no module named wigs or no module name wsgi?

      Delete
    2. Also, try this:
      1. Log in as graphite user with `su graphite`
      2. Activate your virtualenv with `source /opt/graphite/bin/activate`
      3. Then run `pip freeze` and show me the output

      Delete
  4. I also did some changes in the code and now it says: ImportError: No module named django. Django is installed on my machine and also in the virtualenv. If i check it with pip freeze the result is:

    Django==1.4.5
    PAM==0.4.2
    Twisted==12.0.0
    Twisted-Conch==12.0.0
    Twisted-Core==12.0.0
    Twisted-Lore==12.0.0
    Twisted-Mail==12.0.0
    Twisted-Names==12.0.0
    Twisted-News==12.0.0
    Twisted-Runner==12.0.0
    Twisted-Web==12.0.0
    Twisted-Words==12.0.0
    argparse==1.2.1
    chardet==2.0.1
    distribute==0.6.24
    django-tagging==0.3.1
    psycopg2==2.4.5
    pyasn1==0.1.3
    pycrypto==2.6
    pyserial==2.5
    pysqlite==2.6.3
    python-apt==0.8.8.2
    python-debian==0.1.21
    python-memcached==1.48
    virtualenv==1.7.1.2
    wsgiref==0.1.2
    zope.interface==3.6.1

    I can't understand why python can't find django module.

    ReplyDelete
    Replies
    1. Hey Mariano. I see that our django versions are different even though there's the pip install carbon graphite-web whisper 'Twisted<12.0' django==1.5 simplejson psycopg2 django-tagging step where you have to indicate 1.5.

      Please see this video: https://drive.google.com/file/d/0B5fLVqQY7CdRODZlVHJtUFhhR00/view?usp=sharing

      Also, here is the output of my pip freeze. I hope this helps!
      Cheetah==2.4.4
      Django==1.5
      Landscape-Client==14.01
      PAM==0.4.2
      PyYAML==3.10
      SecretStorage==2.0.0
      Twisted==11.1.0
      Twisted-Conch==13.2.0
      Twisted-Core==13.2.0
      Twisted-Lore==13.2.0
      Twisted-Mail==13.2.0
      Twisted-Names==13.2.0
      Twisted-News==13.2.0
      Twisted-Runner==13.2.0
      Twisted-Web==13.2.0
      Twisted-Words==13.2.0
      apt-xapian-index==0.45
      argparse==1.2.1
      chardet==2.0.1
      cloud-init==0.7.5
      colorama==0.2.5
      configobj==4.7.2
      django-tagging==0.3.4
      html5lib==0.999
      httplib2==0.8
      jsonpatch==1.3
      jsonpointer==1.0
      keyring==3.5
      launchpadlib==1.10.2
      lazr.restfulclient==0.13.3
      lazr.uri==1.0.3
      oauth==1.0.1
      prettytable==0.7.2
      psycopg2==2.6
      pyOpenSSL==0.13
      pyasn1==0.1.7
      pycrypto==2.6.1
      pycurl==7.19.3
      pygobject==3.12.0
      pyserial==2.6
      pysqlite==2.6.3
      python-apt==0.9.3.5
      python-debian==0.1.21-nmu2ubuntu2
      python-memcached==1.53
      requests==2.2.1
      simplejson==3.6.5
      six==1.5.2
      ssh-import-id==3.21
      txAMQP==0.6.2
      urllib3==1.7.1
      virtualenv==1.11.4
      wadllib==1.3.2
      whisper==0.9.13
      wsgiref==0.1.2
      zope.interface==4.1.2

      Delete
    2. I solved it. I just had one error in the graphite.ini file and for this reason UWSGI could not be able to find the virtualenv (and all its dependencies). Thank you!!

      Delete
    3. Glad that worked out. I do have an article for setting up a Grafana Dashboard. That might be useful, also.

      Delete
  5. Hey Rob,

    Brilliant article. :) I have an issue. I recently updated the hostname of my EC2 instance and since then, the Grafana is not receiving any data. And even Graphite. Do you think this is an issue with Sensu sending data to Graphite? or something else?

    ReplyDelete
    Replies
    1. Thanks, Karthik. How did you change your hostname, exactly? And were other services affected, also? Check out this link.

      Delete
  6. $ sudo /opt/graphite/bin/python /opt/graphite/webapp/graphite/manage.py syncdb

    gave me this:

    ImportError: No module named fields

    any ideas?

    ReplyDelete
    Replies
    1. Hm. Didn't come across this issue, myself. Let me go through the steps again and see if I can reproduce it.

      Delete
    2. I get the same thing. Any help appreciated. :)

      Delete
    3. I got past this by installing "django-tagging<0.4" in the pip install section.

      Delete
  7. HI Rob,
    Thanks for the great article. Im facing issues in start the uwsgi service. It jus fails and dont get any error in the graphite.log too.
    Following is my pip Freeze
    Django==1.8.7
    Twisted==15.4.0
    argparse==1.2.1
    chardet==2.0.1
    colorama==0.2.5
    django-tagging==0.3.1
    html5lib==0.999
    meld3==0.6.10
    psycopg2==2.4.5
    pysqlite==2.6.3
    python-ldap==2.4.10
    python-memcached==1.53
    pytz==2015.7
    requests==2.2.1
    simplejson==3.3.1
    six==1.5.2
    ssh-import-id==3.21
    supervisor==3.0b2
    urllib3==1.7.1
    virtualenv==1.11.4
    wheel==0.24.0
    whisper==0.9.14
    wsgiref==0.1.2
    zope.interface==4.1.3

    Graphite log:
    Mon Nov 30 06:43:49 2015 - *** Starting uWSGI 1.9.17.1-debian (64bit) on [Mon Nov 30 06:43:49 2015] ***
    Mon Nov 30 06:43:49 2015 - compiled with version: 4.8.2 on 23 March 2014 17:15:32
    Mon Nov 30 06:43:49 2015 - os: Linux-3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015
    Mon Nov 30 06:43:49 2015 - nodename: grafana
    Mon Nov 30 06:43:49 2015 - machine: x86_64
    Mon Nov 30 06:43:49 2015 - clock source: unix
    Mon Nov 30 06:43:49 2015 - pcre jit disabled
    Mon Nov 30 06:43:49 2015 - detected number of CPU cores: 4
    Mon Nov 30 06:43:49 2015 - current working directory: /
    Mon Nov 30 06:43:49 2015 - writing pidfile to /run/uwsgi/app/graphite/pid
    Mon Nov 30 06:43:49 2015 - detected binary path: /usr/bin/uwsgi-core
    Mon Nov 30 06:43:49 2015 - setgid() to 33
    Mon Nov 30 06:43:49 2015 - setuid() to 33
    Mon Nov 30 06:43:49 2015 - your processes number limit is 524288
    Mon Nov 30 06:43:49 2015 - your memory page size is 4096 bytes
    Mon Nov 30 06:43:49 2015 - detected max file descriptor number: 524288
    Mon Nov 30 06:43:49 2015 - lock engine: pthread robust mutexes
    Mon Nov 30 06:43:49 2015 - thunder lock: disabled (you can enable it with --thunder-lock)
    Mon Nov 30 06:43:49 2015 - uwsgi socket 0 bound to UNIX address /run/uwsgi/app/graphite/socket fd 3
    Mon Nov 30 06:43:49 2015 - uwsgi socket 1 bound to TCP address 127.0.0.1:3031 fd 5
    Mon Nov 30 06:43:49 2015 - Python version: 2.7.6 (default, Jun 22 2015, 18:01:27) [GCC 4.8.2]
    Mon Nov 30 06:43:49 2015 - Set PythonHome to /opt/graphite
    Mon Nov 30 06:43:49 2015 - *** Python threads support is disabled. You can enable it with --enable-threads ***
    Mon Nov 30 06:43:49 2015 - Python main interpreter initialized at 0x1df70b0
    Mon Nov 30 06:43:49 2015 - your server socket listen backlog is limited to 100 connections
    Mon Nov 30 06:43:49 2015 - your mercy for graceful operations on workers is 60 seconds
    Mon Nov 30 06:43:49 2015 - mapped 218376 bytes (213 KB) for 2 cores
    Mon Nov 30 06:43:49 2015 - *** Operational MODE: preforking ***
    /opt/graphite/webapp/graphite/settings.py:246: UserWarning: SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security
    warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
    Mon Nov 30 06:43:49 2015 - WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1df70b0 pid: 15426 (default app)
    Mon Nov 30 06:43:49 2015 - *** uWSGI is running in multiple interpreter mode ***
    Mon Nov 30 06:43:49 2015 - spawned uWSGI master process (pid: 15426)
    Mon Nov 30 06:43:49 2015 - spawned uWSGI worker 1 (pid: 15442, cores: 1)
    Mon Nov 30 06:43:49 2015 - spawned uWSGI worker 2 (pid: 15443, cores: 1)


    Regards,
    Ravi

    ReplyDelete
  8. Iwould also like to thank you for this tutorial. However, now when I want to redo it on a new server I get a rendering error I cannot resolve: 'LocalTimezone' object has no attribute 'normalize'
    Any insights in this?

    ReplyDelete
    Replies
    1. The problem was solved by installing pytz and restarting uwsgi. In the virtualenv:
      > pip install pytz
      > sudo service uwsgi restart

      Delete
  9. Hello.

    I get the same error as wildling:


    psycopg2.OperationalError: no connection to the server


    First the program works correctly. Then, if I stop and re-run the program (sudo /opt/graphite/bin/python /opt/graphite/bin/carbon-cache.py stop/start). And that's it. Stops working with the error message above. What's happening?

    ReplyDelete
  10. I had to specify django-tagging==0.4 and install python-cairocffi outside the virtualenv, but that took care of it. Thanks for the walkthrough!

    ReplyDelete