Configuring Red Hat's High-Availability Server

Rafeeq Ur Rehman

This is a 24x7 world, and many businesses need around-the-clock, reliable processing of data. Many UNIX vendors provide high-availability systems for their machines. With a gradual market shift towards Linux, demand for Linux-based, high-availability solutions has also increased. This article takes a quick look at how to configure the Red Hat High-Availability Server, which runs with Red Hat Linux 6.2 and provides a failover clustering and load balancing solution.

There have been many definitions for the term high availability (HA). For simplicity, high availability is a solution that minimizes downtime for a system. High availability can be achieved in many ways, including fault tolerance and computer clustering. To achieve high availability, computer clusters are usually used in a failover server (FOS) configuration where one computer (the backup or failover server) backs up another one (the primary server). The primary server provides some service to end users in normal circumstances.

All servers in a failover cluster use two type of network addresses. Private, or heartbeat, addresses are used to monitor each other; public addresses are used for end users. Initially, all public addresses are configured on the primary server. If the primary server fails (hardware or software), the backup server takes over the job of primary server. This process is transparent to the end user, as public network addresses are migrated from primary server to the backup server. The end user still connects to the same address and sees no difference as the same services and data are also available on the backup server. One example is a Web server where two identical machines, with the same data back up each other. Most of the time, primary and backup servers are connected to a RAID or other storage solution, such as the one provided by EMC Corporation. A failover, high-availability cluster configuration is shown in Figure 1.

A virtual IP (VIP) address is configured on the public network, which is visible to clients. All services are provided using this virtual IP address. This VIP address can be moved from primary to backup server, and vice versa, when required. The active server always keeps the VIP address so the clients always connect to the active server. When the primary server fails, the backup server takes up the VIP and configures it on one of its network adapters connected to the public network. The private network is usually used for heartbeat monitoring, a term used to monitor the health of the other system in a cluster. The heartbeat and public networks may physically be the same network or different ones.

The Red Hat high-availability server can be used in two different configurations. First is the failover server (FOS) configuration, which I describe in this article, and the other one is the Linux virtual server (LVS) configuration. In LVS configuration, the Red Hat server can be used with servers running any operating system to monitor and load balance. This is a much broader application than a simple failover server. In the LVS configuration, multiple FOS clusters can be used to add another layer of reliability to the system.

System Requirements

System requirements for the server depend upon the application that will run on the participating machines. Although it is not necessary for both machines participating in failover configuration to be identical, it helps. You should have as much RAM and CPU power available as you can, but the minimum recommended RAM is 64 MB. Red Hat documentation recommends 1.2 GB disk space for the HA server, but I was able to install it on as little as 300 MB. I believe that with some effort, this may be decreased. You may need additional space for your applications and data.

Installation

When you purchase the High-Availability Server from Red Hat, you get a box with three CDs and an installation guide. The first CD contains Red Hat Linux 6.2 along with the Red Hat High-Availability Server. This is all you need to install the server. The installation process is well explained in the accompanying guide. However, it is not much different from the installation of a simple Red Hat Linux distribution on a PC or server. The only difference is in some options and background images. When you are asked about "Install or Upgrade", you have to select "Cluster Server", which is the default choice. The rest is the same familiar process of installing Linux, which is briefly explained below.

To start the installation, install the first CD and reboot the system. Your machine will start booting from Red Hat Linux 6.2. Here you can choose one of the available installation methods. I always select the "Expert" method, because it allows more flexibility. After that, you have to make Language, Keyboard and Mouse selections. After a welcoming screen, you have to select "Cluster Server". Then there are the usual processes of partitioning the disk, selecting mount points, formatting the partitions, and so on. You then will go to lilo configuration, time zone selection, and setting the root password. In the "Package Group Selection", you can select packages according to your needs. The amount of required disk space depends upon what you select for installation. If you install everything, the disk requirement may go up to 1.5 GB. To configure the server using the Web interface, you must install Apache Web server and Netscape Communicator, both of which are bundled with the installation CD and are selected for installation by default.

There are other installation methods available for the server, but the easiest one is through a bootable CD-ROM. Anybody who has installed Red Hat 6.2 will not find any difficulty in installing the HA server.

Configuration

After completing the installation process, the major part is to configure the High-Availability server. The basic requirement is that you should have TCP/IP networking already configured and working on both primary and backup servers. Thus, you should be able to ping one server from the other. I used IP address 192.168.2.100 for the primary server and 192.168.2.200 for the backup server. These addresses are used for the heartbeat check of the servers. After that you must configure both servers so that the root user can rlogin to the other server without a password. You should block the rest of the world from gaining access to the servers with rlogin. The first step is to configure /etc/hosts.allow and /etc/hosts.deny files on both of the systems. For example, these files on my primary server look like the following:

There is only one line in /etc/hosts.deny file as shown below:

#
# This file describes the names of the hosts which are
# not allowed to use the local INET services.
#
ALL: ALL

This line disables access for all hosts except those mentioned in /etc/hosts.allow file, which is listed below:

#
# This file describes the names of the hosts which are
# allowed to use the local INET services.
#
ALL: 192.168.2.200

This file allows access to host 192.168.2.200, which is the failover server for cluster. Having done that, you should create .rhosts (dot rhosts) file in the home directory of the root user on both of the systems. This will allow access of root user from both of the systems without typing a password. In the simplest case, you can just add the host name of the other server in this file. You also need to edit the /etc/hosts file that maps host names to IP addresses such that the host name of both of the servers can be resolved.

The /etc/lvs.cf file controls failover cluster bootup and who takes control of the primary server. Before going into detail about what is inside this file and how it is configured, look at a sample /etc/lvs.cf file that I created on my test environment:

primary = 192.168.2.100
service = fos
rsh_command = rsh
backup_active = 1
backup = 192.168.2.200
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 18
network = direct
failover Web_Server {
     address = 192.168.2.11 eth0:1
     active = 1
     port = 80
     timeout = 6
     send = "GET / HTTP/1.0\r\n\r\n"
     expect = "HTTP"
     start_cmd = "/etc/rc.d/init.d/httpd start"
     stop_cmd = "/etc/rc.d/init.d/httpd stop"
}

The first part of this file, which is not indented, is a general setup that lists primary and backup server heartbeat IP addresses, the type of high-availability server (in this case, failover designated by "fos"), heartbeat port numbers, and keepalive times. The second part shows a failover configuration for the Web server using port number 80. It shows that the Web server is using the IP address 192.168.2.11, which is configured on interface eth0:1. This is the virtual IP known to end users through DNS. For testing this configuration, I used only one network interface, but I strongly recommend using a separate network interface for heartbeat. For all external users, the public IP address for the Web server is 192.168.2.11, which is initially configured on the primary server. If the primary server fails, the backup server detects it by using heartbeat, configures the same IP address on its eth0:1 interface, and starts the Web server. As shown, the server start and stop scripts are also included in the configuration file.

The syntax of the configuration file must be strictly followed. There should not be any space on any side of the equal sign. A complete list of keywords that can be used in this file is present in the installation guide that comes with the high-availability server.

Once you have finalized this file, it needs to be copied on both of the servers participating in the cluster. After copying this file, you can restart the machines for changes to take effect, or you can use the following command to start the cluster pulse-monitoring daemon:

/etc/rc.d/init.d/pulse start

Whenever you make any change to this file, it should be copied to all of the participating servers. Once the cluster is running, you can list the available Ethernet adapters and configured IP addresses by using the ifconfig -a command. Output similar to the following will be observed in response to this command:

eth0      Link encap:Ethernet  HWaddr 00:E0:29:89:28:59  
          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::e0:2989:2859/10 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1389 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2891 errors:0 dropped:0 overruns:0 carrier:0
          collisions:5 txqueuelen:100
          Interrupt:11 Base address:0xe800 

eth0:1    Link encap:Ethernet  HWaddr 00:E0:29:89:28:59  
          inet addr:192.168.2.11  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:11 Base address:0xe800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:1408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1408 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

Interface eth0:1 comes up as soon as one of the nodes in the cluster takes the role of master server.

Using the Web Interface to Configure the Cluster

The beauty of the configuration process is that it is easy (even for not-so-experienced administrators), with the help of the Web interface. The configuration file (/etc/lvs.cf) shown previously is not generated manually. I used the Web interface to configure the cluster that automatically generates this configuration file. To configure the cluster, start Netscape communicator and go the following URL:

http://localhost/piranha

Your browser screen will look something like the one shown in Figure 2. Using this screen, login to the cluster administration system. The login name is "piranha", and the password for this login ID is created by using the following command:

/usr/sbin/piranha-passwd <password>

After you are logged into the system, the main configuration screen appears as shown in Figure 3 where you can specify heartbeat addresses and other configuration addresses for both of the servers. You can also configure services that are to be monitored by the failover server. Different settings can be changed by using four of the links provided under the "CONTROL/MONITORING" label in the screen shot shown in Figure 3. I hope you find these configuration tips useful.

Price and Availability

The high-availability server can be purchased for $1995 from the Red Hat online store at: http://www.redhat.com.

Rafeeq U. Rehman received his bachelors and masters degrees in Electrical Engineering from the University of Engineering and Technology, Lahore. He is the author of many articles on Linux, and a book on HP-UX system and network administration. His interests include network management, security, and C programming. He can be contacted at: rurehman@yahoo.com.