UltraMonkey on CentOS 4 with BlueQuartz

CONFIGURATION

Server is configured as director to be a part of high availability and load balancing cluster as described here:
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html

CentOS 4 with Bluequartz from Nuonce.net is used as operating system.

PROBLEM

As a part of configuration an alias for loopback interface had to be created so server can accept packets for the virtual service:

DEVICE=lo:0
IPADDR=192.168.6.240
NETMASK=255.255.255.255
NETWORK=192.168.6.0
BROADCAST=192.168.6.255
ONBOOT=yes
NAME=loopback

When heartbeat starts on director, it removes the alias on loopback interface lo and creates virtual IP address on interface eth0.

Here is how all interfaces look like before problem occurs:

[root@server ~]# ip addr show
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

However after certain amount of time the alias with virtual IP appears on loopback interface again:

[root@server ~]# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet 192.168.6.240/32 brd 192.168.6.255 scope global lo:0
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

NOTE

This problem only affects servers acting as load balancers / directors. Real servers running on CentOS / Bluequartz are not affected.

CAUSE

It turns out this problem is caused by swatch which is utilized by BlueQuartz. Every 15 minutes it checks the status of all interfaces that have ONBOOT=yes and brings them up if they are down.

See /usr/sausalito/swatch/bin/am_network.sh for more information.

SOLUTION

In order to prevent swatch from manipulating our virtual IP address, you have to turn off ONBOOT parameter since that what it looks for.

...
ONBOOT=no
...

Then disable heartbeat from starting automatically and make sure it is off.

[root@server ~]# chkconfig heartbeat off
[root@server ~]# chkconfig --list heartbeat | grep heartbeat
heartbeat       0:off   1:off   2:off   3:off   4:off   5:off   6:off

Finally bring the alias and heartbeat up in rc.local. This script file is run once, before all other scripts have run but before the logon prompt appears.

...
ifup lo:0
/etc/init.d/heartbeat start

LINKS

1. Streamline High Availability and Load Balancing
2. How to install Ultra Monkey LVS in a 2-Node HA/LB Setup on CentOS/RHEL4