Saturday, December 6, 2014

Log Rotation

Log rotation, an automated archiving process, is designed to alleviate the maintenance of large numbers of log files on any given server. Under Linux, this process is carried out by a utility called logrotate, and allows for the automatic rotation, compression, deletion, and transmission of logs. Parameters may be set to ensure that logs are processed either periodically or once a threshold has been reached. It is absolutely critical that you rotate your logs or you run the risk of running out of disk space. It will also keep you from having to swim through millions of log entries just to track down a particular event.

Prerequisites: You'll want to set up an Ubuntu instance using this guide. This article was written for Linux Ubuntu users.

Configuration

Most systems will already have a daily log rotation script at either /etc/cron.daily/logrotate or /etc/cron.daily/logrotate.cron. You'll notice there are also designations for hourly, weekly, and monthly in the same location. All of these scripts start by reading the configuration file at /etc/logrotate.conf. Take the time to scan through this file, as it's well commented and fairly straightforward.

# see "man logrotate" for details
# rotate log files weekly
weekly

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}
# system-specific logs may be configured here

So in this configuration file, you'll notice we're rotating our logs weekly, but holding onto a month of backlogs. What we mean by rotation is we log line-by-line entries to the same file for up to a week, then backlog it and create a new file. Compression is off by default. We tell logrotate to store package specific logs in /etc/logrotate.d. So Nginx would store its log configuration in /etc/logrotate.d/nginx. Finally, we tell logrotate what to do with wtmp and btmp logs. These are just classifications of utmp files that keep track of logins and logouts to the system. utmp tracks the current login state per user, wtmp records login and logout history, and btmp records failed login attempts. Now, let's move onto the logrotate.d files. Run ls /etc/logrotate.d to get a list of all the specific package logs that go through rotation.

Understanding Log Files

Here are the contents of /etc/logrotate.d/nginx. In the following sections, we'll be breaking this down, line by line.

/var/log/nginx/*.log {
  weekly
  missingok
  rotate 52
  compress
  delaycompress
  notifempty
  create 0640 www-data adm
  sharedscripts
  prerotate
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
      run-parts /etc/logrotate.d/httpd-prerotate; \
    fi \
  endscript
  postrotate
    [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
  endscript
}

Log file paths

This is indicated by the top level block, which gives logrotate a wildcard(*) or space separated values to target specific logs with this configuration.

/var/log/nginx/*.log /var/log/nginx/log.txt {
...
}

Other parameters

You then have the ability to indicate rotation interval, rotation count, and compression.

weekly # rotate weekly
missingok # move onto next log file without error message if log file is missing
rotate 52 # delete after 52 log files have been created
compress # compress after rotation...
delaycompress # ...but compress on the next rotation cycle
notifempty # don't rotate empty logs
create 0640 www-data adm # file creation and permissions

Prerotate, postrotate and shared scripts

prerotate designates scripts to be run before rotating the file, postrotate runs after, and sharedscripts ensures that the script only runs once rather than every rotated log file.

sharedscripts
prerotate
  if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
    run-parts /etc/logrotate.d/httpd-prerotate; \
  fi \
endscript
postrotate
  [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
endscript

You should have a good overview of logrotate at this point. Make sure to also check out this article on Nginx error logging, as well.

No comments:

Post a Comment