dropdown menu

Network - Topolgy

Physical and logical networks

A physical network connects two or more physical network interfaces. These network interfaces (on one physical network) can be grouped together to form a logical network. These logical networks are known by a unique name (for example net_ether_01 in PowerHA). A logical network can be viewed as the group of interfaces used by PowerHA to host one or more service IPs.

During  initial cluster configuration, the discovery process automatically defines the networks and interfaces to them. Discovery process will use the /etc/hosts file, and it will create:
/usr/es/sbin/cluster/etc/config/clip_config: It contains details of the discovered interfaces; used in the F4 SMIT lists.
/usr/es/sbin/cluster/etc/config /clvg_config: It contains details of each physical volume (PVID, volume group name,status, major number, and so on) and a list of free major numbers.


Resources: File systems, service IPs, applications... which are highly available (These can be moved from one node to another.)
Resource Group: Those resources which are grouped together, and moved together during a failover

Topology: It is a general term, which refers to nodes, networks, communication interfaces, and communication adapters.
Boot interface: Any interface that can host a service IP. (Previous versions of PowerHA used the terms boot and standby adapter, in PowerHA7.1 these are collapsed into one term: boot interface)
Base IP: The default IP that is configured on an interface by AIX during startup.
IP alias: An IP address that is added to an interface, rather than replacing its base IP address. It is an AIX function supported by PowerHA.
Service IP: An IP address over which a service is provided. (usualluyit is an alias) It is kept highly available by PowerHA. (It is not part of the topology, it is part of the resources.)


Persistent Node IP:

A persistent node IP is an IP alias that can be assigned permanently to a node. It always stays on the same node (is node-bound) and it can coexists with other IP labels present on the same node. It Is not part of any resource group and this address can be used for administrative purposes because it always points to a specific node regardless of whether PowerHA is running.

The persistent IP labels are defined in the PowerHA configuration, and they become available when the cluster definition is synchronized. A persistent IP label remains available even if PowerHA is stopped, or the node is rebooted. If the interface with the persistent IP label fails while PowerHA is running, the persistent IP label is moved to another interface on the same node.
If the node fails the persistent IP label will no longer be available.


IP address takeover (IPAT) mechanisms

1. IP replacement (PowerHA versions 5.x, 6.x)
2. IP aliasing (PowerHA versions 5.x, 6.x, 7.x)

1. IPAT via IP replacement:

The service IP replaces the existing address on the interface, thus only one service IP can be configured on one interface at one time. The service IP should be on the same subnet as one of the boot IP addresses.

Other interfaces on this node cannot be in the same subnet, and they are called as standby interfaces.
These standby interfaces are used if the boot interface fails. IPAT via IP replacement can save subnets, but requires extra hardware.

If the interface holding the service IP address fails, PowerHA moves the service IP address on another available interface on the same node and on the same network; in this case, the resource group is not affected. If there is no available interface on the same node, the resource group is moved together with the service IP to another node with an available interface on the same logical network. (IP replacement method is not supported on PowerHA 7.1)

2. IPAT via aliasing

The service IP is aliased (using the ifconfig command) onto the interface without removing the underlying boot IP address. This means that more than one service IP label can coexist on one interface.Each boot interface on a node must be on a different subnet. The service IP labels can be on one or more subnets, but they cannot be the same as any of the boot interface subnets. Standby interfaces are not necessary, because all interfaces are labeled as boot interfaces.

By removing the need for one interface per service IP address, IPAT through aliasing is more flexible and in some cases requires less hardware. IPAT through aliasing also reduces fallover time, as it is much faster to add an alias to an interface, rather than removing the base IP address and then apply the service IP address.


PowerHA v7.1 supports IP address takeover (IPAT) only through aliasing.

The service IP is aliased onto the interface without removing the underlying boot IP address by using the ifconfig command. IPAT through aliasing obsoletes the concept of standby interfaces (all network interfaces are labeled as boot interfaces). As IP addresses are added to the interface through aliasing, more than one service IP label can coexist on one interface. PowerHA uses the subnet mask of the base IP address for all IP aliases configured on this network interface.

IPAT through aliasing is supported only on networks that support the gratuitous ARP function of AIX. Gratuitous ARP is when a host sends out an ARP packet before using an IP address and the ARP packet contains a request for this IP. In addition to confirming that no other host is configured with this address, it ensures that the ARP cache on each machine (on the subnet) is updated with this new address.

For IPAT through aliases keep these in mind:
- Each base adapter must be on separate subnets (this is needed to allow heartbeating).
- Service addresses must be on separate subnet from the base subnets.
- Multiple service labels can coexist as aliases on an interface, and these can be on the same subnet.
- In a multiple adapter per network configuration, the persistent IP must be on a different subnet of each of the boot interfaces
- The persistent IP  can be in the same or different subnet as the service.
- All interfaces that belong to the same network must have same subnet mask. (Interfaces on other network can have another subnet mask.)

Not all addresses must be routable outside the VLAN. Only the service and persistent addresses must be routable. The base adapter addresses and any aliases used for heartbeating do not need to be routed outside the VLAN because they are not known to the client side.

During failover to another interface (on the same node) it can happen, that the service IP address is active on both the failed Interface and on the takeover interface. This is needed to preserve routing (routes will no desappear from routing table), but this might cause a DUPLICATE IP ADDRESS error log entry, which you can ignore.


Multicast vs Unicast at PowerHA

PowerHA SystemMirror 7.1 introduced multicast-based (IP-based multicast) communication between the nodes in the cluster. Multicast-based communication provides optimized communication method to exchange heartbeats, and also allows to communicate critical events and messages, in one-to-many method instead of one-to-one method.

CAA built to use kernel-level code to exchange heartbeats over the network, and CAA exploits multicast network communication.

In multicasting nodes are forming a group and exchange messages between each other. A multicast message sent by one node in the group is received by all in the group. This allows for efficient cluster communication where many times messages need to be sent to all nodes in the cluster. For example, a cluster member need to notify the remaining nodes about a critical event, so it will send a single multicast packet with the relevant information.

One important prerequisite, that multicast must be enabled on the switches as well. Before creating a cluster verify that multicast traffic works correclty.

It can be tested with mping. (If mping fails, network admin needs to review switches)

1. mping -v -r -c 5       <--start mping on one node in receiver mode (-r) (-v: verbose, -c:number of pings)
2. mping -v -s -c 5       <--start mping on other node in sender mode (it will choose default ip, with -a you can specify ip)


PowerHA v7.1.0 through v7.1.2 requires to use multicast, however PowerHA v7.1.3 introduced unicast as an option, making multicast optional also.

Using multicast may have a benefit in a multi node (3,4,5...) cluster, but (as I saw) there is no benefit in a 2 node cluster setup. (It creates additional attention and complexity.) In PowerHA v7.1.3 the default method used in cluster configuration is unicast (which in my oppinion is good.)

Changing cluster config from Multicast to Unicast:

1. lscluster -c                 <--check Communication mode
2. smitty cm_define_repos_ip_addr <--change heartbeat mechanism from mulitcast to unicast
3. verify and sync <--verify and synchronize cluster
4. lscluster -c                  <--check again Communication mode


No comments:

Post a Comment