Sunday, November 23, 2014

How System Administrators Keep Services Running

Prerequisites: This guide was written for Linux Ubuntu 14.04 users.

Running Processes in the Background


Daemons are background processes that run continually, responding to events and other requests. Unlike foreground processes, a daemon will notify you that it's running, detach itself from the terminal, and allow you to continue running other commands.

The "&" operator

Appending "&" to a command that would normally run a foreground process will run it in the background. node & for instance will run node in the background and return the job number and PID of the process like so: [1] 23471. So if you'd like to return the process to the foreground, run fg 1, with 1 representing the job number of our node process. If you have a process running in the foreground, you can send it to the background by pressing CTRL+Z to suspend and bg to run it in the background. Finally, run jobs to list all running jobs.

# List all running jobs
[1]-  Stopped                 node
[2]+  Stopped                 python
[3]   Running                 find / -name nginx.conf &


service is a program you'll become very familiar with right off the bat. Some common programs that run in the background through service include networking, nginx, and sshd. Running service usually requires root access and the full command for starting the sshd service is as follows: sudo service sshd start. Various services automatically run on startup depending on the configuration of the server.


tmux is short for Terminal MUltipliXer and lets you manage multiple sessions in one terminal, allowing you to attach and detach programs across sessions and run foreground services in the background. Run brew install tmux on Mac OS X or sudo apt-get install tmux on Ubuntu to install. Tmux makes it easy to run a process in a session, allowing you to close the terminal window. This is perfect for mongo dumps or any other operations that take a significant amount of time.

Automating Service Initialization


Whenever you set up a new Linux distribution, you can often install the required daemons you need to run at install time. Distribution vendors normally supply initialization scripts for your services. However, you're going to run into a scenario where you're building from source without an init script or even writing from source. You'll want to write your own init.d script. You can find an example for Nginx at this link.


Upstart is an event based utility for starting and terminating services during system boot and shutdown. Upstart can also be used to monitor running services and compensates for the limitations of system V and dependency-based init systems. Check back for a detailed article about how to write your own upstart script, in the future.

Cron Jobs

cron is a Linux utility that allows you to run a command or script automatically on reboot or at a specified time and date. Cron jobs are useful for scheduling daily migrations and other repetitive tasks. Each user has their own crontab, which allows the commands entered to run with those user's privileges.

# Open up a the crontab as the appropriate user
sudo crontab -u rob -e

# Edit this file to introduce tasks to be run by cron.
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# For more information see the manual pages of crontab(5) and cron(8)
# m h  dom mon dow   command

We append @reboot PORT=9001 NODE_ENV=production /usr/lib/node_modules/forever/bin/forever start -c /usr/bin/node /home/rob/sites/ to the end of the crontab file. The commands kickstarts forever, a utility for keeping node processes running.

Another process we can start on reboot is buildbot, our continuous integration system. We log in to our CI server, open up the crontab for the buildbot user with sudo crontab -u buildbot -e and append @reboot cd /home/buildbot/buildbot && /home/buildbot/buildbot/bin/buildbot start master && /home/buildbot/buildbot/bin/buildslave start slave to the end of the file. As you can see, we're running the version of python running in the applications's virtualenv directory which bypasses the need to activate the virtualenv.


supervisor is another linux utility for keeping services running. Here is the initial setup:

# install
sudo apt-get install supervisor

# edit the config file
sudo vi /etc/supervisor/supervisord.conf

# add the following to your supervisord.conf...
# turn on the web status console

# reload supervisor
supervisorctl reload

Warning: If you get the following error: error: , [Errno 111] Connection refused: file: /usr/lib/python2.7/ line: 224, run supervisord -c /etc/supervisor/supervisord.conf and then try running supervisorctl reload again.

Now browse http://localhost:9001 and you'll see your supervisor's console. If you installed supervisor on a server, navigate to <IPADDRESS>:9001.

Warning: If you get the following error: error: , [Errno 13] Permission denied: file: /usr/lib/python2.7/ line: 224, you'll need to make supervisord's socket accessible to non-root users. Follow these steps:

# create a supervisor group
groupadd supervisor

# add ourselves to this group
usermod -a -G supervisor 

Now update the unix_http_server block of your /etc/supervisor/supervisord.conf file with the following:

file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0770                       ; socket file mode (default 0700)

Then, log out and log in so the new group membership takes effect. Afterwards restart supervisord. Now head over to the supervisor docs to configure your first application with supervisord. I also have an example of how to keep Logstash running at the bottom of my article on setting up Logstash.


Systemd is another system and service manager that is becoming the standard for Linux distros as System V and Upstart get deprecated. Some of Systemd's advantages over System V include the use of sockets to start daemons in parallel and a reduced computational overhead[1].

Head over to this article for more info about Systemd.

Other Useful Commands

Finding Our Which Ports Are Being Used by Running Services

To scan all ports, run the following commands:

# with nmap
sudo nmap -T Aggressive -A -v -p 1-65000

# with netstat
sudo netstat --tcp --udp --listening --program

# with lsof
sudo lsof +M -i4

# with fuser
sudo fuser -v 3143/tcp

# and to kill a process with fuser
sudo fuser -vk 3143/tcp

Now, that's just a handful of techniques we can leverage, but that concludes this article. In follow up articles, we'll go in depth into these are other utilities so check back soon.


1. ^ Wikipedia (23 June 2015). "SystemD"

No comments:

Post a Comment