dropdown menu


Cluster Manager (clstrmgr)
This is the main PowerHA daemon. It maintains an upated information about the health of the cluster, status of the resource groups and interfaces. After PowerHA is installed clstrmgrES is started from inittab, and it is always running whether cluster services are started or not.

lssrc -ls clstrmgrES                        displays a lots of useful information about cluster

It uses services provided by the RSCT subsystems to monitor the status of the nodes and their interfaces. It receives ibformation from Topology Sevices and uses Group Services for inter-node communication. It invokes the appropriate scripts in response to node or network events.(recovering from SW/HW failures, request to online/offline a node, request to move/online/offline a resource group) It maintains update informations about the resource groups (status, location) A daemon which runs on each cluster nodes.

If clstrmgr hangs or is terminated the default action taken by SRC is to issue halt -q, causing the system to crash. Clstrmgr is dependent on RSCT; if topsvcs or grpsvcs has problems with starting, the clstrmgr will not start either.


Cluster configuration daemon (clconfd)
This keeps CAA cluster configuration information in sync. It wakes up every 10 minutes to synchronize any necessary cluster changes.


Cluster Information Program (clinfo)
Clinfo obtains updated cluster information from the Cluster Manager. It makes information about the state of the cluster, nodes, networks and applications. Used by clstat, and it is optional on cluster nodes and clients.

startsrc -s clinfoES                        starts clinfo (/usr/es/sbin/cluster/etc/rc.cluster this script also starts everything)
stopsrc -s clinfoES                         stops clinfo


Cluster Communication Daemon (clcomd)
All cluster communication is going through clcomd. It must be running before any cluster services can be started. The trusted IP addresses are stored in /etc/cluster/rhosts file. (root.system 0600). Nodes with an empty or missing rhosts file will refuse all POWERHA related communication. The real use of the /etc/cluster/rhosts file is before the cluster is first synchronized. After the first synchronization, ODM classes are populated, and rhosts file is not used (only when adding new node to the cluster.)

Clcomd is started via /etc/inittab entry, which is created during PowerHA install. Clcomd is managed by src (startsrc, stopsrc, refresh; refresh is useful to reread /etc/cluster/rhosts file), and logs are in /var/hacmp/clcomd/clcomd.log. It uses port 6191.

Clcomd used for these tasks:
- verification and synchronization
- global OBM changes and remote command execution (commands listed under /usr/es/sbin/cluster)
- File collections
- C-SPOC and User and password administration (c-spoc commands are in /usr/es/sbin/cluster/cspoc)


netmon.cf file

In PowerHA, heartbeating is used to monitor an adapter’s state over a long period of time. When heartbeating is not working, a decision must be made about whether the local adapter has gone bad or there are other network problems. This decision is made based on whether any network traffic can be seen on the local adapter, using the inbound byte count of the interface. Where VIO is involved, this test becomes unreliable because there is no way to distinguish whether inbound traffic came in from the outside world through VIOS, or from a neighboring virtual I/O client.

A new netmon.cf function was added to support PowerHA in a virtual I/O environment, where PowerHA could not detect a local adapter-down event. This is because traffic being passed between the VIO clients looks like normal external traffic. The new format helps to decide, that adapter is up only, if it can ping a specified target.

The netmon.cf file must be placed in the /usr/es/sbin/cluster directory on all cluster nodes. Up to 32 targets can be provided for each interface. If any specific target is pingable, the adapter will be considered “up.” (The traditional format of the netmon.cf file is not valid in PowerHA v7, and later, and is ignored. Only the !REQD lines are used.)

example content of /usr/es/sbin/clister/netmon.cf:
!REQD en2
!REQD en2

Interface en2 is considered “up” only if it can ping either or



There is a newer solution for the above VIO - PowerHA adapter status problem, and netmon.cf file is not required anymore (keyword: poll_uplink). Using SEA poll_uplink method, SEA can pass up the link status, so no "!REQD" style ping is required any more. (VIOS and AIX 7.1 TL3 needed)

There's no special configuration required on the VIOS/SEA to support this feature, and on the VIO client + PowerHA these steps are needed:

1. smitty clstop                                  <--stop cluster
2. chdev -l <entX> -a poll_uplink=yes -P             <--enable on all virt. interface on all node (with -P reboot is needed)
3. entstat -d <entX> | grep -i bridge               <--after enabling it, it can be checked with entstat, it should show: Bridge Status: Up
4. clmgr -f modify cluster MONITOR_INTERFACES=enable <-- enable poll uplink in PowerHA (CAA)
5. verif. and synch                               <--run verif. and synch.
6. smitty clstart                                 <--start cluster


clhosts file
This file contains IP address information which helps to enable communication between monitoring daemons on clients and the PowerHA cluster nodes. The file resides on all PowerHA cluster servers and clients in the /usr/es/sbin/cluster/etc/ directory.
When a monitor daemon starts up (for example clinfoES on a client), it reads this file to know which nodes are available for communication.
(when running clstat utility from a client, the clinfoES obtains info from this file.)


Address Resolution Protocol (ARP)
The Internet communication protocol used to dynamically map Internet addresses to physical (hardware/MAC) addresses on local area networks.
The /usr/sbin/cluster/etc/clinfo.rc script, which is called by the clinfo utility whenever a network or node event occurs, updates the system’s ARP cache.

clinfo.rc file
PowerHA can be configured to change the MAC address of a network interface by hardware address takeover (HWAT). In a switched enwironment, the network switch might not always get promptly informed of the new MAC. The clinfo.rc script is used to flush the system's ARP cache in order to reflect changes to network IP addresses. (HWAT is only supported when using IPAT via replacement.)
On clients not running clinfoES, you might have to update the local ARP cache by pinging the client from the cluster node. In order to avoid this, add the IP of the client to the PING_CLIENT_LIST variable in the clinfo.rc script (/usr/es/sbin/cluster/etc/clinfo.rc). Through the use of PING_CLIENT_LIST entries, the ARP cache of clients (and other network devices) can be updated.


RSCT (Reliable Scalable Cluster Technology) (earlier: RS/600 Cluster Technology)
It is a software stack, a package of services ("cient subsystems"), which is a prerequisite for HACMP and is packaged with AIX.

-Topology Services: generates heartbeats to monitor nodes, networks and network adapters, diagnoses failures. When a node joins the cluster, topology services adds the adapter information to the machine list.
This "topology" or "connectivity" information is then passed on to group services.

-Group Services: "Client subsystems" (e.g. event management subsystem, RMC subsystems...) forms groups, with a membership list. Provides reliable communication and protocols required for cluster operation. The main daemon is hagsd. Group services coordinates/monitors state changes within the cluster (e.g. node join/leave) and then passes these state changes to the interested subscribers, for example cluster manager or. event management.Enhanced Concurrent mode disks use RSCT group services to control locking.

-Resource Monitoring and Control (RMC): RMC notifies the Cluster Manager about events, so it responds to this event by RSCT. Main damon is rmcd. The process application monitoring uses RMC and therefore does not require any custom script. Dynamic Node Priority (DNP) is calcuated by the use of RMC

-Event Management: Match information about the state of system resources, it initiates the scripts needed managing the cluster.

HACMP relies on topology services for heartbeats and group services for reliable messaging. These services are started prior to the cluster processes. If there are problems with these services the cluster will not start.


SNMP (Simple Network Management Protocol)
It is a popular protocol for network management. It is used for collecting information from, and configuring, network devices, such as servers, printers, hubs, switches, and routers
When giving cldump it says: "Obtaining information via SNMP from Node: aix11..."

lssrc -a | egrep 'snm|mib'        lists the necessary daemons
ls -l /etc | grep snmp            location of conf. files
ls -l /var/tmp | grep snmp        location of log files
/usr/sbinsnmpv3_ssw -n            switch to the non-encrypted version of snmpdv3 agent (it can be used when problems with clstat, cldump)

clsmuxpd (SMUX Peer daemon)
Clusmuxpd provides SNMP support.  (IT is before v 5.3, as clinfo improved)
clinfo is based on SNMP; it queries the clsmuxpd daemon for up-to-date cluster information and provides a simple display of it.


C-SPOC (Cluster Single Point of Control)

It helps managing the entire cluster from a single point. In smitty hacmp or with commmands which are under /usr/es/sbin/cluster/cspoc.
C-SPOC using clcomd for HACMP communication between nodes, so /etc/rhosts file no longer used.
If there is a failure of a C-SPOC function it will be logged in the /tmp/cspoc.log, on the node performing the operation.

cspoc.log             contains the used commands in this file


HACMP start-up:

When PowerHA installed it will create this entry:
hacmp:2:once:/usr/es/sbin/cluster/etc/rc.init >/dev/console 2>&1                    <--it starts clcomdES, clstrmgrES, snmpd, syslogd

If PowerHA configured for IP Address Takeover:
harc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network startup           

When start at system restart option is chosen in C-SPOC (Manage HACMP services)
hacmp6000:2:wait:/usr/es/sbin/cluster/etc/rc.cluster -boot   -A   # Bring up Cluster    <--do not use this option, manual control better


Manual cluster switch:
1. varyonvg
2. mount FS (mount -t sapX11)(mount -t nfs)
3. check nfs: clshowres
   if there are exported fs: exportfs -a
   go to the nfs client: mount -t nfs
4. IP configure (ifconfig)
   grep ifconfig /tmp/hacmp.out -> it will show the command:
     IPAT via IP replacement: ifconfig en1 inet netmask up mtu 1500
     IPAT via IP aliasing: ifconfig en3 alias netmask
   netmask can be found from ifconfig or cltopinfo -i
   (removing ip: ifconfig en3 delete
5. check routing (extra routes could be necessary)
   (removing route: route delete -host or route delete -net



  1. Hi :)

    Need a help. Is there a way to change the node_id attribute of HACMPnode odm stanza?

    Currently my nodes are showing as "2" and "3". But rest of all other clusters are as 1&2. I need to change the node_id value for the problematic cluster to 1&2.

    Any thoughts?


    1. Hi,
      I never had to do that, but I would open an IBM call, as it looks strange....

  2. Agreed - looks strange to me.. Even tried to cahnge the node_id value in HACMP odm, but it goes back to 2&3 on verify and sync :(

  3. Can I update netmon.cf on the live servers (while RG is online)?

    1. Netmon.cf file will be read at cluster startup, since RSCT topology services scans the netmon.cf file during initialization. I think you can update the file, just your update will not take effect until you restart the cluster.

    2. Hi,

      Thanks for your prompt reply.

      So, bringing services down with the unmannaged option and then restart with automatically manage Rg option will activate the netmon.cf?


    3. Hi, as far as I know cluster services are not restarted if you put Resource Group to unmanage and then manage.
      You need a full stop for the cluster, then at the next start it will read the new config file.

  4. Hi,

    May i know, when i can expect your GPFS articles in this blog?


    1. Hi, I don't have any systems with GPFS...when there will be one, where I can test few things, I'll write about it.

  5. HI...

    To start a cluster through SMIT,the command is #smitty cl_start or #smitty clstart.
    in which version it is cl_start and in which versin it is clstart

    To start a cluster does the smitty path is same for 5.5 and 6.1
    I am little bit confused, kindly help me.

    Thank you

    1. Hi, both of them are the same and it is working for me under 6.1, I guess same with 5.5 (Why did not try these by yourself?)

    2. Thank you for replying.

      I have recently completed y course. Now I dont have IBM box to check. Thats why I took your help.

      Thank you

  6. Hi..,
    The df commands show the file system is 100% full,but du commands shows no files,what's the reason?can u give me some solution.

    1. Hi, can you post the outputs of those commands?

  7. Hi,

    I went through your VIO, NIM, General articles and are very helpful, but I never did a HACMP configuration...with your blog I am able to understand the topology and definitions, can you alos please demonstrate or explain on how to configure 2 node cluster step by step ( like what filesets's or SW / how to gather IP's on node A and B and how to input and how to configure disk/network hearbeat, naming, VG's)

    I worked on cluster which is already configured and performed cluster start/stop/sync/manual failover/failback/c-spoc administration {creating users/add pv's etc}, but never configured one.

    if you can provide/demonstrate the procedure it will be very appriciated.

    1. Hi, there is some description at this link: http://aix4admins.blogspot.hu/2011/08/build-and-configure-1.html.

  8. hi
    how to move resource groups in hacmp? what is the pre request for doing that?

  9. Hi, what are the steps to go from "IPAT via IP replacement" to "IPAT via aliasing" with the least effort possible? Thank you

  10. Could u please tel me watz the major difference between Aix p6 and p7 serious???

  11. Can you run C-SPOC commands when only one node is started in the Cluster? Command like unmirrorvg and mirrorvg. Will the 2nd node's ODM be updated?

  12. Which IP is responsible for Heartbeat, either Serivice IP or Boot IP? Can't get exact information in Redbooks & Google too.


  13. What are the differences between PowerHA 5.4/6.1 and 6.1/7.1