STORAGE - MPIO

MPIO (Multipath I/O):


Multipathing

A path describes a route from the HBA port in the host (through the switches in the fabric) into a storage port on the storage array. When more than one HBA ports are cabled, a host can access a LUN through more than one path, which is called multipathing. If any of the components along the path fails, the server selects another path. The process of detecting a failed path and switching to another path is called path failover. 

In general 4 or 8 paths per disk is recommended, (or up to 16 paths for rare situations.) Extra, unnecessary redundancy could have some negative effect (each paths takes extra memory, error recovery could take longer)

Patch Control Module (PCM): It is responsible for controlling multiple paths. Each storage device requires a PCM. PCM is a storage vendor supplied code that handles path management.  It can be a separate (3rd party) software (driver) or AIX has a native PCM package, which comes with the base operating system, which is sometimes called AIXPCM, MPIOPCM or just MPIO.

--------------------------------------------------------

AIX multipathing and MPIO history

At the end of the 90s and early 2000 SAN and FC technology got more popular and storage vendors (EMC, Hitachi, IBM...) came up with their own multipathing solutions. IBM developed SDD. With SDD each path was called hdisk and on top of these a "super device" (vpath) has been created to do path management. It was capable of path failover and load balancing, but with the increased number of hdisks and vpath devices the maintenance was a bit complex. Around 2002 AIX 5.2 has introduced a new feature called MPIO. Without installing any 3rd party or vendor filestes and without creating extra hdisks devices, the new AIX MPIO feature was capable of discovering multiple paths. It recognized that these paths belonged to the same LUN (hdisk), so only one hdisk device was created. It made administration easier, but it was lacking some capabilites, for example load-balancing. To solve this problem IBM created the SDDPCM package (for IBM storage), which was based on MPIO and it was capable for load-balancing. SDDPCM was very popular, but during the years the default AIX MPIO was also getting better and better, until it reached all the capabilities of SDDPCM, so IBM decided to withdraw SDDPCM from the market in 2020.

Before AIX 5.2 to use multipathing, storage vendor codes were needed, for example IBM provided the SDD package. Since AIX 5.2, a default AIX installation is capable for path management and this feature is called MPIO (MPIO is an acronym for Multipath I/O, which is a mutlipathing solution.) MPIO allows a disk to have multiple paths, but it has only 1 entry per hdisk in the ODM. With MPIO design, the path management is off-loaded from the disk driver to an AIX kernel extension called: PCMKE (Path Control Module Kernel Extension). The reason to have a separate PCMKE is to make it easier for disk vendors, such as EMC, Hitachi or IBM, to adopt the AIX MPIO solution. Additionally 4 new AIX commands have been added to AIX 5.2 to manage the device paths: mkpath, rmpath, lspath, chpath.  In AIX 7.1 TL3 a new command: lsmpio, has been introduced. This command is similar to lspath, but displays additional information about the MPIO storage devices.

UDID (Unique device identifier)

Every MPIO-capable device must provide a unique identifier that allows the device to be distinguished from any other device in the system. It is called UDID . When the cfgmgr command runs, it requests the UDID for the device and it is compared with the UDIDs stored in ODM to determine if a newly discovered device needs to be defined or the device already exists and only a new path needs to be defined.

--------------------------------------------------------

ODM Update

The AIX MPIO infrastructure allows IBM or third-party storage vendors to supply ODM definitions, which are often referred to as a host attachment kit. A host attachment fileset from the disk vendor is needed to update the ODM to support a specific storage, then AIX can recognize and appropriately configure the disk. Without this, disks are configured using generic ODM definitions. Based on the host attachment filesets disks will be configured as MPIO or non-MPIO devices.

For example if we have a 3rd party storage (like Hitachi or EMC), and we don't install any additional packages, then probably AIX will detect disks as "Other FC SCSI Disk Drive". A device will be discovered as an "MPIO other FC device" only if the device has been certified with one of the AIX default PCMs and no vendor-provided ODM definitions have been installed.  Certification does not guarantee that all device capabilities can be used. (Usually only fail_over is possible but no round_robin or load_balancing.) After the ODM is updated, and the host attachment contained MPIO definitions, the device will be discovered as an MPIO device (for example EMC storage could look like "EMC MPIO FC disks") and one of the AIX PCMs can do the path management. If host attachment contained non-MPIO definitions, then disks will be recognized as non-MPIO (like "EMC CLARiiON FCP RAID 1/0 Disk") and the vendor supplied driver needs to be installed for path management. After the ODM update, the newly discoverd disks will have vendor configured default values (like queue_depth, reserve_policy etc.)

--------------------------------------------------------

MPIO disk settings
https://developer.ibm.com/articles/au-aix-mpio/

# lsattr -El hdisk15
PCM             PCM/friend/fcpother         Path Control Module              False
PR_key_value    none                        Persistant Reserve Key Value     True+
algorithm       round_robin                 Algorithm                        True+
clr_q           no                          Device CLEARS its Queue on error True
dist_err_pcnt   0                           Distributed Error Percentage     True
dist_tw_width   50                          Distributed Error Sample Time    True
hcheck_cmd      inquiry                     Health Check Command             True+
hcheck_interval 60                          Health Check Interval            True+
hcheck_mode     nonactive                   Health Check Mode                True+
...
...
timeout_policy  retry_path                  Timeout Policy                   True+
unique_id       2611200173800005102BE072810XIV03IBMfcp Unique device identifier  False
ww_name         0x500173800051019           FC World Wide Name               False

--------------------

algorithm: (determines how I/O should be distributed across the paths)
- fail_over: I/O is distributed to one path at a time, if it fails next enabled path is selected, depending on path priority.
           VSCSI disks use fail_over, if SCSI-2 reserves is used (reserve_policy=single_path) fail_over is the only possible algorithm.
           By default,  priority is set to 1 (highest priority) and it can range from 1 to 255 (which is the lowest priority).
           set priority: # chpath -l hdisk1 -a priority=255 -p fscsi0 -w 20080022a10bb2d5,1000000000000
           check priority # lspath -l hdisk1 -a priority -F value -p fscsi0 -w 20080022a10bb2d5,1000000000000
   
- round_robin: I/O is distributed to all enabled paths. Paths with same prio. has equal I/O, otherwise higher prio. path utilized more
           To achieve optimal performance during failure, make sure the ordered path list alternate paths between separate fabrics.

- shortest_queue: Similar to round_robin, but when load increases it favors path with fewest I/O  and path priority is ignored.
           If one path is slow (congestion in the SAN), other less-congested paths are used for more I/O

If using SCSI-2 reserves or vSCSI disks, then fail_over must be used. For other situations, shortest_queue (if available) or round_robin enable maximum use of the SAN resources.

--------------------

hcheck_cmd: (the command that will be sent to the disk for healthcheck)
- test_unit_rdy: Test Unit Ready (TUR) command, this is the default setting
- inquiry: In clustered environments with this can have greater control of SCSI reserve and release

In general the default setting should be used, but if there are reservation locks on the disks, use the inquiry option. If test unit ready is used and the backing device is reserved, then test unit ready fails and log an error on the client.

--------------------

hcheck_mode: (determines which paths will be tested by the health checker. Disabled or Missing paths are never checked
(Disabled paths must be recovered manually with chpath, Missing paths with cfgmgr first. If disk is not open (VG is varied off), nohealth checking is done.)

- nonactive: paths with no active I/O are checked (or paths in failed state).
(At round_robin and shortest_queue all paths are busy, so only failed paths are checked. If disk is idle then any paths can be checked (without a pending I/O.))

- enabled: The PCM checks all enabled paths, even paths that have other active I/O

- failed:  PCM checks paths that are marked as failed.

The default value is nonactive, and most of the cases this should not be changed.

--------------------

hcheck_interval:  (interval, in seconds, at which MPIO checks path availability).

- hcheck_interval=0:  disables health check mechanism, any failed paths need to be recovered manually

- hcheck_interval=(small value): the health check commands use the disk’s queue_depth and they receive a higher priority than normal I/O, so a small health check interval can quickly use up a lot of bandwidth on the SAN if there are a large number of disks.

If a device has only one non-failed path and an error is detected on that last path, AIX will send a health check command on all failed paths before retrying the I/O. If there are good paths to use, AIX will discover it and uses it before failing user I/O, regardless of the health check interval setting.

So, for most cases, the default value is appropriate and the general rule for hcheck_interval is, that it should be greater than or equal to rw_timeout. It is much more likely to be a good idea to increase the health check interval than to decrease it. Better performance is achieved when hcheck_interval is slightly greater than the rw_timeout value on the disks.

https://support.purestorage.com/Solutions/IBM/AIX/AIX_Recommended_Settings
PureStorage ODM definition sets hcheck_interval setting to 10 as opposed to IBM recommendation of 30. hcheck_interval setting is set to lower value than the rw_timeout as we are not checking active paths and a lower hcheck_interval will not have any SAN performance impact.

--------------------

timeout_policy: (when an I/O operation fails to complete within the rw_timeout value, what action PCM should take)

- retry_path: command is retried on the same path. This is likely to lead to delays in the I/O recovery.
        (Only after several failures will AIX fail the path and try the I/O on an another path.)

- fail_path:  AIX will fail the path after a single command timeout. Failing the path forces the I/O to be retried on a different path.
         With this setting recovery can be quicker and also the detection of a situation where all paths have failed is also quicker.
         A path that is failed due to timeout policy can later be recovered automatically by the AIX health check commands.

- disable_path: path will be disabled after a single command timeout. Disabled path can be recovered only manually (chpath)

The recommended setting is fail_path.

--------------------------------------------------------

Active/Passive paths

A Storage Processor or Storage Controller manages or controls the disk array at storage side. Many storage arrays have two or more controllers (Storage Processors) and connecting HBA ports to each controllers protects against controller failures. A device that has multiple controllers can designate one controller as the active or preferred controller. For such a device, the PCM uses just the paths to the active or preferred controller as long as there is at least one such path that is enabled and not failed.

Disk arrays can be:
- active/active: it allows access to the LUNs simultaneously through all the storage processors. All the paths are active at all times.
- active/passive: one SP is actively servicing a given LUN. The other SP acts as backup for the LUN (and may be actively serving other LUN I/O).

--------------------------------------------------------

manage_disk_drivers

manage_disk_drivers lists storage models and their drivers. These drivers can be AIX MPIO (like AIX_AAPCM or AIX_APPCM) and non MPIO drivers, when a third-party mulitpath driver is installed. In this case the AIX MPIO feature can be disabled by selecting the AIX_non_MPIO option. (AAPCM means Active/Active and APPCM means Active/Passive)

# manage_disk_drivers -l
Device              Present Driver        Driver Options   
2810XIV             AIX_AAPCM             AIX_AAPCM,AIX_non_MPIO
DS4100              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4200              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4300              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4500              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4700              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4800              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS3950              AIX_APPCM             AIX_APPCM
DS5020              AIX_APPCM             AIX_APPCM
DCS3700             AIX_APPCM             AIX_APPCM
DS5100/DS5300       AIX_APPCM             AIX_APPCM
DS3500              AIX_APPCM             AIX_APPCM
XIVCTRL             MPIO_XIVCTRL          MPIO_XIVCTRL,nonMPIO_XIVCTRL
2107DS8K            NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
IBMFlash            NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
IBMSVC              NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO


NO_OVERRIDE:
The "NO_OVERRIDE" option (as a Present Driver) indicates that the configuration is not overridden by using "manage_disk_drivers -d (device) -o...". If SDDPCM is installed it will substitue AIX MPIO driver for SVC, DS8k (that is why it is not listed by manage_disk_drivers), and by default SDDPCM takes precedence. In this case NO_OVERRIDE would mean SDDPCM will be used (we did not override it by a manual change). If SDDPCM is not installed, then the AIX default PCM is used.

# manage_disk_drivers -d IBMSVC -o AIX_AAPCM     <---use the AIX default PCM even if SDDPCM is installed
# manage_disk_drivers -l                         <--after reboot, AIX PCM is in control (not SDDPCM, which can be uninstalled now)
Device              Present Driver        Driver Options
IBMSVC              AIX_AAPCM             NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO

Changing the driver will reset disk attributes (queue_depth, algorithm, reserve_policy..) to the new default values of the new driver.

--------------------------------------------------------

lsmpio

The command lsmpio was introduced in AIX 7.1 TL3, and it displays information about the MPIO storage devices (those devices which are controlled by PCM). It is similar to lspath, but with much more details.

# lsmpio -l hdisk1234
name   path_id status  path_status     parent connection
===============================================================================
hdisk1234  0   Enabled Opt,Sel,Deg,Rsv fscsi0 500a098186a7d4ca,0008000000000000
hdisk1234  1   Enabled Non             fscsi0 500a098196a7d4ca,0008000000000000
hdisk1234  2   Enabled Opt,Sel         fscsi1 500a098186a7d4ca,0008000000000000
hdisk1234  3   Enabled Non             fscsi1 500a098196a7d4ca,0008000000000000


status: (same as with lspath)
enabled:   path is configured and operational. It will be considered when paths are selected for IO.
disabled:  path has been manually disabled and won't be considered when paths are selected for IO. (set back to enabled with 'chpath')
failed:    path is unusable due to IO failures. It won't be selected for IO. (after problem is fixed, remove path and rediscover it)
defined:   path is not configured and used for io operation (rmpath -l will explicitly turn path off)
missing:   path was not detected after reboot, but it was there earlier in the system (these can be recovered with 'cfgmgr')
detected:  path was detected during boot, but it was not configured (this status should never appear, only during boot)

It is best to manually disable paths before storage maintenance (rmpath). AIX MPIO stops using any disabled or Defined paths, so no error detection or recovery will be done. This ensures that the AIX host does not go into extended error recovery during a scheduled maintenance. After the maintenance is complete, the paths can be re-enabled with cfgmgr. (When disabling multiple paths for multiple LUNs, rmpath is simpler than chpath, as it does not have to be run on a per-disk basis.)

path_status: (more detailed path status)
Opt - optimized path: preferred path, which is attached to a preferred controller. PCM selects preferred paths whenever possible.
Non - non-optimized path: this path is not considered as preferred path. PCM avoids this path, unless all preferred paths fail.
Act - active path: (on a device that has active and passive controllers) the PCM selects active paths for I/O operations.
Pas - passive path: (on a device that has active and passive controllers) the PCM avoids the selection of passive paths.
Sel - selected path: this path is selected for I/O operations when lsmpio command was issued.
Rsv - reservation conflict: could be that multiple hosts accessing the same disk
Fai - failure: path experienced a failure. (If status is Enabled and Fai: MPIO leaves one path in Enabled state, even when all paths have errors.)
Deg - degraded: there were errors, which causing to temporarily avoid to use this path. (Additional errors can cause path to fail)
Clo - closed: if all paths are closed, the device is closed.  If some paths are closed, then those had errors and MPIO tries to periodically recover those.

--------------------------------------------------------

smitty mpio
genkex | grep pcm                    show loaded pcm (like /usr/lib/drivers/sddpcmke)

lspath                                lists paths (lspath -l hdisk46)
lspath -l hdisk0 -HF "name path_id parent connection path_status status"    more detailed info about a device (like lsdev for devices)
lspath -AHE -l hdisk0 -p vscsi0 -w "810000000000"    display attrib. for given path and connection (-w) (-A is like lsattr for devices)
                                      (if only 1 path exist to parent device connection can be omitted: lspath -AHE -l hdisk0 -p vscsi0)
lspath -l hdisk1 -a priority -F value -p fscsi0 -w 20080022a10bb2d5,1000000000000  check priority

chpath                                changing path state (enabbled, disabled)
chpath -s enabled -l hdisk -p vscsi0  it will set the path to enabled status
chpath -l hdisk1 -a priority=255 -p fscsi0 -w 20080022a10bb2d5,1000000000000     change priority

rmpath -l hdiskX -p vscsi0 -w 870000000000   put path in defined state  (-w can be omitted if only 1 path exist to parent device)
rmpath -dl hdiskX -p fcsiY           dynamically remove all paths under a parent adapter from a supported storage MPIO device
                                     (-d: deletes, without it puts it to define state)
                                     (The last path cannot be removed, the command will fail if you try to remove the last path)

mkpath ...                            makes a path into Available state

lsmpio                                lists addtional info about paths (which path is selected)
lsmpio -q                             shows disks with its size
lsmpio -ql hdiskX                     shows disk serial number (LUN ID)
lsmpio -Sl hdisk0 | grep Path         shows path statistics (which path was used mostly in the past)
lsmpio -ar                            list parent adapter and remote port information (-a: adapter (local), -r: remote port)
lsmpio -are                           list error statistics for local and remote (-e: error)
lsmpio -z                             reset statistics

While checking errors in lsmpio -are:
If 1 HBA shows error on all remote storage ports (but other HBA has no errors), then it is a problem with the adapter, cable or switch.
If 2 HBAs show errors on 1 particular storage port, then the error is at the storage port side (or the cable or the switch port.)
If 2 HBAs show erros on the same number of storage ports, then most likely switch has errors (probably at the ISL (inter switch link) between switches)

--------------------------------------------------

Failed path handling:
(there were Hitachi disks in Offline (E) state, but they were not unconfigured earlier)
    -lspath | grep -v Enab
    -rmpath -p fscsiX -d
    -cfgmgr -l fcsX
    -lspath | grep -v Enab
    -dlnkmgr view -lu -item

--------------------------------------------------

Change adapter setting online:

rmpath -d -p vscsi0                             <--removes all paths from adapt. (rmpath -dl hdisk0 -p vscsi0, it removes only specified path)
rmdev -l vscsi0                                 <--puts adapter into defined state
chdev -l vscsi0 -a vscsi_err_recov=fast_fail    <--change adapter setting (if -P is used it will be activated after reboot)
cfgmgr -l vscsi0                                <--configure back adapter

--------------------------------------------------------

SDDPCM migration to AIX PCM
https://www.ibm.com/support/pages/migrate-aixpcm-using-managediskdrivers-command>

Below steps should be done during downtime when no application is using the disks.
If the disks are shared among multiple nodes in a cluster or VIOS configuration, the reservation policy must be checked and changed before the disk is opened. If this is not changed, other nodes in the cluster or VIOS configuration will lose access to the shared disks until the reservation policy is changed.

lsattr -El hdiskX                               <--check current disk attributes (save output) (lsmpio should not work at this point)
manage_disk_drivers -l                          <--list disk drivers
manage_disk_drivers -d IBMSVC -o AIX_AAPCM      <--switching disk driver to MPIO (manage_disk_drivers -l will show new value)
shutdown -Fr                                    <--reboot
lsattr -El hdiskX                               <--check new value (lsmpio should work now)
chdev -P -l hdisk0 -a queue_depth=X -a reserve_policy=no_reserve  <--set back needed values for disks
chdef -a reserve_policy=no_reserve -c disk -s fcp -t mpioosdisk   <--set predefined values in ODM (or chdef -a reserve_policy=no_reserve -c disk -s fcp -t aixmpiods8k)
shutdown -Fr                                    <--reboot
installp -u devices.fcp.disk.ibm.mpio.rte devices.sddpcm.72.rte  <--remove host attachment and sddpcm filesets (if sddpcm is removed both should be removed!!)
shutdown -Fr                                    <--reboot

Check again current and ODM settings to be sure SDDPCM removal did not overwrite something, because removing SDDPCM will set the queue to the default value of 20
If needed to roll back to SDDPCM: # manage_disk_drivers -d IBMSVC -o NO_OVERRIDE