apr2003.tar

Logrotate -- A Backup Solution

Sean D. Conway

A After a system failure, tracking down the media, determining the last known good backup, and starting the recovery process can be tricky. I implemented the solution described in this article to reduce the stress of performing backups. Using a program found in Linux called logrotate, I created a backup solution to consolidate and automate the backup process. Logrotate's configuration file contains the details to create the backups, compress the backup files, provide automatic rotation, and remove stale files.

Logrotate is a utility designed to handle the management of log files. Logrotate maintains a series of log files, automatically rotates the files, and removes the files based on the parameters provided in a configuration file. You can specify the name of any file in the logrotate configuration file, it is treated the same as a log file.

I used logrotate when I worked with Red Hat (RH) 6.0. Since then, I have found it to be part of a standard RH install. The current version is available from:

http://www.rpmfind.net/linux/RPM/

The program can be installed with an rpm -Uvh logrotate-3.6.4-1.i386.rpm (current RH 7.3 version) command. Following are a few points of interest in the logrotate tour:

/usr/sbin/logrotate -- Utility residence
/etc/logrotate.conf -- Default configuration file
/etc/logrotate.d -- Default directory for other configuration files
/var/lib/logrotate.status -- Default state file

On RH, logrotate is run from a daily job directory /etc/cron.daily/logrotate. When logrotate is called, it obtains information about which files it is processing by using the default configuration file /etc/logrotate.conf. Alternate configuration files can be used if optioned. The configuration file defines the file to rotate and the parameters to apply. A configuration file entry may look like this:

#daily
"/var/log/logfile.log" {
        daily
        rotate 7
        compress
        }

Using the configuration shown here, the file logfile.log would be compressed and rotated and given the name of logfile.log.1.gz. After day two, the file logfile.log.1.gz would be rotated to logfile.log.2.gz and new copy of logfile.log.1.gz produced from logfile.log. At the end of the configuration file schedule, the directory would contain seven copies of the database. On the eighth day, the copy with the extension logfile.log.7.gz would be replaced with another rotated copy. A long listing (ls -al) in the directory would display seven copies of the database files, with dates preserved from when they were first created.

The man pages for logrotate explain the parameters used to control logrotate. The remainder of this article describes how logrotate can be used to produce a backup schema.

When planning the backup process, I used a variation on the grandfather-father-son backup tape rotation method. The backup set consists of grandfather(monthly), father(weekly), and son(daily) as shown in Figure 1.

Under this grandfather-father-son implementation method, a full backup is performed every day (Monday-Sunday); a full backup of the oldest daily is preserved every Sunday; a full backup of the oldest weekly is preserved the last day of a four-week cycle. This assumes you start the daily cycle on a Monday. After 1 year, 6 dailies, 3 weeklies, and 12 monthly copies are preserved.

Logrotate works similarly to a stack in programming. As each file is rotated, it is pushed onto the stack and forces other files down the stack. The size of the stack is determined by the rotate value in the logrotate configuration file. Once the stack capacity is reached, the first file in gets pushed out. Figure 1 illustrates the idea.

To produce the backup schema shown in Figure 1, create the configuration file /etc/logrotate_backup.conf from the following listing:

##############################
#
#Name:logrotate_backup.conf
#
#Description:script designed to produce backups
#grandfather(monthly),father(weekly),son(daily)
#
#Author: Sean D. Conway
#
#Wed Aug 28 18:12:59 CDT 2002
#
#Revision:
#
#NOTE:
#replace the path to be backed up.
#don't remove "" good for files that have spaces.
#maintain the "." file extensions this will preserve the frequency
################################

#daily (son)
"/home/oracle/backup/database" {
        daily
        rotate 7
        compress
        }

#weekly (father)
"/home/oracle/backup/database.7.gz" {
        weekly
        rotate 4
        }

#monthly (grandfather)
"/home/oracle/backup/database.7.gz.4" {
        monthly
        rotate 12
        }

This configuration file is divided into three sections: daily, weekly, and monthly. The following listing provides some comments to explain the parameter lines found in the script:

#daily (son)                         <--comments line
"/home/oracle/backup/database" {     <--file to rotate
        daily                        <--rotation schedule
        rotate 7                     <--rotation interval
        compress                     <--compress the file
        }

The parameters here enable the logrotate program to rotate and compress daily a file called database. The rotation interval is seven. At the end of one week, the storage directory will contain the following files: database.1.gz, database.2.gz, database.3.gz, database.4.gz, database.5.gz, and database.6.gz. The weekly configuration would rotate the file database.7.gz to database.7.gz.1. The file database.1.gz would be the most current.

The logrotate utility can access the configuration files using two methods:

1. The configuration file can be placed in the default configuration file directory /etc/logrotate.d. This directory is listed in logrotates default configuration file /etc/logrotate.conf. All files in /etc/logrotate.d are additional configuration files that will be processed by the program.

2. An alternate method is to force the logrotate program to use an alternate configuration file from a command line. The following is the cron entry to support logrotate running each day at 07:00 using a configuration file called /etc/logrotate_backup.conf.

00 07 * * * /usr/sbin/logrotate -f /etc/logrotate_backup.conf

I prefer the second method because it allows more flexible scheduling.

The logrotate backup mechanism is used to support three database servers as shown in Figure 2. Each database server produces a backup file and sends it to a server called a tape_gate. On the tape_gate server, logrotate runs daily and takes care of producing the rotation schema. The complete backup series is exported to tape for offsite storage.

Tape_gate also runs a licensed tape client for accessing the tape silo. This configuration allows the administrators to use the client software to schedule and access the backups. The administrator can use the backups stored locally or pull the file back from the tape storage device. All the file creation dates are preserved, and a simple grep on a month name will produce a list of backup files available for that month.

To get support for this backup solution, I had to address some design limitations. This implementation requires a server to process the backups. The horsepower required for the server is minimal. For this installation, I used a reclaimed Pentium desktop with an additional cost of two hard drives. It does provide an alternate point for accessing backups. I find it easier to access a backup from a disk rather than searching a tape library.

Backup files are moved from the source to tape_gate using secure ftp. At selective network slow periods the additional network traffic is acceptable. Secure ftp provides access to a ready backup from tape_gate if needed.

Table 1 shows the complete set of filenames after a one-year cycle. The drive capacity must be sufficient to support the number of system backups being handled. The cost for hard drive capacity today is lower than it was ten years ago. Sixty-gig IDE drives are common. At 14-MB compressed, I was able to get all the backups on a 60-GB drive. The three-database server backups total 500 MB compressed.

The feature that sold this design was cost. A client license for accessing the tape silo costs $1200. Each of the databases runs a different operating system, so each would have needed a different client license. Instead, two drives cost $300 and $1200 for one client license for tape_gate; I saved $2100. The alternative was to create NFS mount on all the servers.

I have been asked to add additional backups to the server. It seems this common approach to backups for a variety of systems can make life easier for administrators. Certainly, one common backup mechanism for all systems is easier to remember than a different backup method for each system.

Sean D. Conway is a former college instructor turned Network System Specialist for a regional telecommunications company in Canada. Working with administrators, they are responsible for the care and feeding of operational support systems inside the telecommunication cloud. He still dabbles in network theory lectures. His formal electronic engineering technology training and education background are valuable tools in achieving a goal -- making learning about computers and networks easier.