HA - Storage

Storage

Enhanced Concurrent VG

PowerHA is using highly available filesystems on a special VG, which is called "enhanced concurrent volume group" (ECVG). This is basically a usual VG with the possibility to access the VG on all nodes. Any disk which will be used in PowerHA on multiple nodes, must be placed in an enhanced concurrent volume group, which can be used in either concurrent or nonconcurrent setup:
- Concurrent setup: An application runs on all cluster nodes at the same time. In this setup ECVG enables concurrent access to the VG
- Non-concurrent setup:  An application runs on one node at a time. The volume groups are not concurrently accessed, they are accessed by one node at a time.

ECVG has the ability to vary on VG in two modes:
- Active state: The vg behaves the same way as the traditional varyon. Operations can be performed on the vg, and logical volumes and file systems can be mounted.
- Passive state: The passive state allows limited read only access to the volume group descriptor area (VGDA) and the logical volume control block (LVCB).

This type of VG can be created in "smitty hacmp" under C-SPOC --> Storage --> Volume Groups --> Create a Volume Group and then choose both nodes... (In PowerHA we need to create "enhanced concurrent mode volume groups", but this does not necessarily mean that we will use them for concurrent access)

Ensure that the automatic varyon attribute is set to No for shared volume groups.

So, in non-concurrent mode, the node which owns the VG (where appl. is running) can do read-write operations, but on the other node only limited read operations are possible (passive mode). Passive mode is the LVM equivalent of disk fencing, where LVM restricts the higher-level actions, such as NFS and JFS mounts and it does not allow read/write access to files. These different modes can be checked by lsvg command:

# lsvg bb_vg
VOLUME GROUP:       bb_vg 
VG STATE:           active   
VG PERMISSION:      read/write   <--this is the actve node

# lsvg bb_vg
VOLUME GROUP:       bb_vg   
VG STATE:           active     
VG PERMISSION:      passive-only <--this is the standby node

(Also "lsvg -o" will report only those volume groups which are are varied on in active mode.)

Historically, because there was no SCSI locking, a partitioned cluster could quickly vary on a VG, which could cause data corruption. Enhancements in AIX 6.1 and 7.1 introduced JFS2 mountguard. This option prevents a file system from being mounted on more than one system at the same time. PowerHA v7.1 and later automatically enable this feature.

During cluster configuration these volume groups are activated in passive mode. When the resource group comes online on the node, the VG varied on in active mode. When the resource group goes offline, the volume group is varied off to passive mode.

--------------------------

Fast disk takeover

To enable fast disk takeover, PowerHA activates enhanced concurrent volume groups in active or passive states. Fast disk takeover reduces total fallover time by providing faster acquisition of the disks without having to break SCSI reserves. During fast disk takeover, PowerHA skips the extra processing needed to break the disk reserves or update and synchronize the LVM information. It uses enhanced concurrent volume groups, and additional LVM enhancements provided by AIX. When a node fails, the other node changes the volume group state from passive mode to active mode. This change takes approximately 10 seconds and it is at the volume group level. This time impact is minimal compared to the previous method of breaking SCSI reserves.

The active and passive mode flags to the varyonvg command are not documented because they should not be used outside a PowerHA environment. However, you can find it in the hacmp.out log.
Active mode varyon command: varyonvg -n -c -A app2vg
Passive mode varyon command: varyonvg -n -c -P app2vg

When a resource group is brought online, PowerHA checks a disk with lqueryvg to determine if it is an enhanced concurrent volume group or not:
# lqueryvg -p hdisk0 -X
0             <---return code of 0 (zero) indicates a regular non-concurrent volume group (like rootvg)

# lqueryvg -p hdisk8 -X
32           <--return code of 32 indicates an enhanced concurrent volume group

----------------------------

Lazy update

If a volume group under PowerHA is updated directly (that is, without C-SPOC), information on other nodes will be updated only, when PowerHA  brings the vg online on those nodes, but not before.

The volume group time stamp is maintained in VGDA, and the local ODM.  When PowerHA is doing varyon vg, it compares the time stamp in ODM with the VGDA. If the values differ, PowerHA will update ODM with the information in the VGDA.

In normal cicumstances PowerHA does not require lazy update processing for enhanced concurrent volume groups, as it keeps all cluster nodes updated with the LVM information.

----------------------------

Disk fencing

In PowerHA 6.1 and previous releases, when using enhanced concurrent volume groups in a fast disk takeover mode, the VGs are in full read/write (active) mode on the node owning the resource group. Any standby node has the VGs varied on in read only (passive) mode. The passive mode is the LVM equivalent of disk fencing.

Passive mode allows readability only of the volume group descriptor area and the first 4 KB of each logical volume. It does not allow read/write access to file systems or logical volumes. It also does not support LVM operations. However, low-level commands, such as dd, can bypass LVM and write directly to the disk. And when the enhanced concurrent volume group is varied off, there is no write-protect restriction on the disks any more.

PowerHA SystemMirror 7.1 uses the storage framework disk fencing, and it prevents disk write access from the AIX SCSI disk driver layer. It invalidates the possibility of a lower-level operation, such as dd. This new feature prevents data corruption because and it protects the disks even after the enhanced concurrent volume groups varied off.

PowerHA creates a fence group and the AIX SCSI disk driver rejects disk write access and returns the EWPROTECT error when disks are fenced with read only access.

Disk fencing when node and RG comes online:
1. When a PowerHA node comes up a fence group is created for the enhanced concurrent VG and sets the permissions (fence height) to read only. This prevents write access to the disks
2. Prior to vary on the VG, the fence height is set to read write, so the varyonvg to passive mode could work, and when it is done fence height is returned to 'read only'.
3. When the PowerHA resource groups come online, the VG is varied on in active mode, and the fence height is set to read write, and the write access to the disks is granted.

----------------------------

NodeA has changes, we want nodeB to be aware about them:
(for example an fs in a shared vg has been increased without HACMP)

YOU CAN TRY THESE:
I. TRY C-SPOC FIRST: smitty hacmp -> c-spoc -> HACMP Logica Volume Management -> Synchronize a Shared Volume Group Definition


II. MANUAL METHOD:

    1 .NodeA:
    -check lv ownerships:               ls -l /dev/<lvname>
    -if varyoffvg is possible:          varyoffvg <vgname>
    -if varyoffvg is not possible:      varyonvg -bun <vgname>


    2. NodeB:
    -if vg was not exported on the remote node (nodeA)learning import is possible (this preserves lv ownerships under /dev):
                            importvg -L <vgname> -n <hdiskname>
           
    -if learning import not possible and vg already exists, first should be exported:
                            exportvg <vgname>

    -import vg from disk:   importvg -y <vgname> -n -V<maj.numb.> <hdisk>
   
    -!!!!if necessary autovaryon should be set back to no on the standby node: varyonvg -bun <vg> -> chvg -an <vgname> -> varyoffvg
    -if vg was exported/imported lv ownerships will be reset to root.system, check if ok under /dev

    3. NodeA:
    -if varyonvg -bun ... was used, vg should be set back to normal use:
                            varyonvg <vgname>

III. IMPORT VG DEFINITIONS TO HACMP:
    On the node with the incorrect data and the VG NOT varied on export VG: exportvg shared_vg
    On the node with the VG varied on:
    smitty hacmp -> Ext. Conf. -> Ext. Res. Conf. -> HACMP Ext. Res. Group -> Change Show Res. and Attr.:
    "Automatically Import Volume Groups" set to "true", and the VG information will be synchronized immidiately.

--------------------------

Repository disk

PowerHA sends heartbeat packets between all communication interfaces (adapters) in the network to determine the status of the adapters and nodes. Cluster Aware AIX (CAA) provides an extra heartbeat path over SAN or Fibre Channel (FC) adapters. This SAN heartbeating (also known as sancomm) is not mandatory, however it provides an additional heartbeat path for redundancy.

Cluster Aware AIX maintains cluster related configurations (such as node list, cluster tunables etc.) on a special disk device called the repository disk. If a node loses access to the repository disk, the cluster continues operating, but the affected node is considered to be in degraded mode. If in this situation a network problem also occurs on this node, then PowerHA does not allow this node to operate, because a split brain situation could happen (partitioned cluster). In other situations where only the network interfaces have failed, the CAA and the storage framework use the repository disk to do all the necessary communication. (The tie breaker is an optional feature you can use to prevent a partitioned cluster, also known as split brain. If specified as an “arbitrator” (in split and merge policy), the tie breaker decides which partition of the cluster survives. A node that succeeds in placing a SCSI reserve on the tie breaker disk wins, and hence survives. The loser is rebooted.)

# /usr/lib/cluster/clras dumprepos            <--displaying the content of the repository disk (including the CAA tunables and cluster configuration)
HEADER
 CLUSRECID: 0xa9c2d4c2
 Name: caa_cl
...
...
 Multicast: 228.1.1.48
 config_timeout : 240
 node_down_delay : 10000
 node_timeout : 20000
 link_timeout : 0
 deadman_mode : a
 repos_mode : e
 hb_src_lan : 1
 hb_src_san : 2
 hb_src_disk : 1
 site_up : e
 site_down : e
DISKS none
NODES numcl numz uuid shid flags name
 1 0 25bc9128-784a-11e1-a79d-b6fcc11f846f 1 00000000 caa1
 1 0 25e7a7dc-784a-11e1-a79d-b6fcc11f846f 2 00000000 caa2

A backup repository disk can be also configured, which is an empty disk to be used for rebuilding the cluster repository if the current repository disk encounters any failure. To see which disk is reserved as a backup disk, use the clmgr -v query repository command or the odmget HACMPsircol command.

Since PowerHA 7.1.1 after the failure of a repository disk  some cluster maintenance tasks are still possible. This feature is called as "repository disk resilience", and moving Resource Groups, bringing a Resource Group online/offline is still possible, even with a failed repository disk. Because all repository configuration is also maintained in memory, CAA can recreate the configuration information when a new  repository disk is provided, which can be done online with no cluster impact.

Replacing a repository disk from the command line:
1. # clmgr modify cluster caa_cl REPOSITORY=hdiskX                     <-- replace a repository disk
2. # clmgr verify cluster CHANGES_ONLY=yes SYNC=yes                <--verify and synchronize PowerHA configuration
3. # /usr/sbin/lscluster -d                                                                      <--after sync completed, verify that the new repository disk works

(A respository disk can be replaced using smitty hacmp as well: Problem Determination Tools -->  Select a new Cluster repository disk)

When an additional node joins the cluster, CAA uses the ODM information to locate the repository disk. In case the ODM entry is missing the ODM entry can be repopulated and the node forced to join the cluster using an undocumented option of clusterconf. This assumes the administrator knows the hard disk name for the repository disk: # clusterconf -r hdiskX

--------------------------

FS extension after new storage has been added
1. cfgmgr - on both nodes
2. add PVID - check LUNs are the same
3. add LUN to the vg in smitty hacmp (it will do necessary actions on both nodes)
   (smitty hacmp -> system man. -> hacmp log. vol. man. -> shared vol. gr. -> set charact. of a ... -> add a volume ...)
4. lv, fs extension in smitty hacmp:
    1. usual fs extension: hacmp is counting in 512byte blocks. 1MB=2048

    or

    2. if we want to choose which disks should be used:
        -first increase lv (choose disks)
        -lsfs -q (it will show new lv size in 512byte blocks)
        -increase fs by the output of lsfs -q

--------------------------

If PVIDs are not consistent (on nodeB):
nodeA:    varyoffvg sharedvg

nodeB:    rmdev -dl hdisk3 (this will not delete data on the disk, only removes ODM definitions)
          rmdev all inconsistent disks
          cfgmgr
          importvg -V123 -yshared_vg hdisk3  (ODM will be updated with new values)
          chvg -an sharedvg
          varyoffvg sharedvg

nodeA:    varyonvg sharedvg


-----------------------------------

Error: timestamp is different for a VG on the 2 nodes:

This could be because timestamp on the disk and ODM is different for the vg (at least on 1 node)

You can check and compare the timestamps from ODM and disk:
-ODM:
    lsattr -El <vgname>                                    <--there will be a line for the timestamp
    odmget CuAt | grep -p timestamp | grep -p <vgname>     <--there will be a line for the timestamp

-disk:
    lqueryvg -Tp <hdisk>                                   <--it can be any disk from the vg

You can check on both nodes.

Correction:
    -increase in smitty hacmp
    or
    -importvg:
        node1: varyonvg -bun ...          <--break the reservation
        node2: importvg -L ...            <--updates the timestamp
        node2: lsattr -El ...             <--verify the values
        node1: varyonvg -n ...            <--restores the reservation

-----------------------------
Importing vgs with ESS:

-if hdisks has pvid:
    importvg -y vg_name -V major# hdisk1
    hd2vp vg_name

-if vpath has only pvid:
    importvg -y vg_name -V major# vpath0

-if neither hdisks nor vpath has pvid:
    chdev -l vpath0 -a pv=yes
    importvg -y vg_name -V major# vpath0

-----------------------------