IBM AIX - HACMP Failover Test

IBM AIX - HACMP Failover Test

A cluster failover test is typically done in three or four phases:

1. Manual Failover

The manual failover is the most important test for a cluster configuration. This test can be invoked on one node by 

node1# smitty clstop

                               Stop Cluster Services

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* Stop now, on system restart or both                 now                    +
  Stop Cluster Services on these nodes               [barney]                +
  BROADCAST cluster shutdown?                         true                   +
* Select an Action on Resource Groups                 Move Resource Groups   +

When stopping the cluster on node 1 the first thing executed is the cluster stop script. It brings down the applications and unmounts all application filesystems. If your application stop script is not able to stop all application processes some filesystems can't be unmounted and the failover fails.

When all resources are down on node 1 HACMP starts to bring up all resources on node 2. The application start script is the last thing hacmp does.

Check that your application is working properly and that all clients can connect. If so the first phase of the failover test is completed.

2. Manual Failback

Switch the resources back to the home node. Again check if everything is fine.

3. Automatic Failover

This test simulates a hardware failure on the active node. The easiest way to simulate is to issue the command

node1# halt -q

on the active node. Check that everything will be brought up on node 2.

4. Partial Hardware Failure

Sometimes only a component fails. Maybe a network switch fails or a storage system becomes unavailable. Test these scenarios to make sure that HACMP is correctly setup - and only starts a failover if needed. These tests also check if your VGs are correctly mirrored over two sites.