dropdown menu

STORAGE - MPIO

MPIO (Multipath I/O):


Multipathing

A path describes a route from the HBA port in the host (through the switches in the fabric) into a storage port on the storage array. When more than one HBA ports are cabled, a host can access a LUN through more than one path, which is called multipathing. If any of the components along the path fails, the server selects another path. The process of detecting a failed path and switching to another path is called path failover. 

In general 4 or 8 paths per disk is recommended, (or up to 16 paths for rare situations.) Extra, unnecessary redundancy could have some negative effect (each paths takes extra memory, error recovery could take longer)

Patch Control Module (PCM): It is responsible for controlling multiple paths. Each storage device requires a PCM. PCM is a storage vendor supplied code that handles path management.  It can be a separate (3rd party) software (driver) or AIX has a native PCM package, which comes with the base operating system, which is sometimes called AIXPCM, MPIOPCM or just MPIO.

--------------------------------------------------------

AIX multipathing and MPIO history

At the end of the 90s and early 2000 SAN and FC technology got more popular and storage vendors (EMC, Hitachi, IBM...) came up with their own multipathing solutions. IBM developed SDD. With SDD each path was called hdisk and on top of these a "super device" (vpath) has been created to do path management. It was capable of path failover and load balancing, but with the increased number of hdisks and vpath devices the maintenance was a bit complex. Around 2002 AIX 5.2 has introduced a new feature called MPIO. Without installing any 3rd party or vendor filestes and without creating extra hdisks devices, the new AIX MPIO feature was capable of discovering multiple paths. It recognized that these paths belonged to the same LUN (hdisk), so only one hdisk device was created. It made administration easier, but it was lacking some capabilites, for example load-balancing. To solve this problem IBM created the SDDPCM package (for IBM storage), which was based on MPIO and it was capable for load-balancing. SDDPCM was very popular, but during the years the default AIX MPIO was also getting better and better, until it reached all the capabilities of SDDPCM, so IBM decided to withdraw SDDPCM from the market in 2020.



Before AIX 5.2 to use multipathing, storage vendor codes were needed, for example IBM provided the SDD package. Since AIX 5.2, a default AIX installation is capable for path management and this feature is called MPIO (MPIO is an acronym for Multipath I/O, which is a mutlipathing solution.) MPIO allows a disk to have multiple paths, but it has only 1 entry per hdisk in the ODM. With MPIO design, the path management is off-loaded from the disk driver to an AIX kernel extension called: PCMKE (Path Control Module Kernel Extension). The reason to have a separate PCMKE is to make it easier for disk vendors, such as EMC, Hitachi or IBM, to adopt the AIX MPIO solution. Additionally 4 new AIX commands have been added to AIX 5.2 to manage the device paths: mkpath, rmpath, lspath, chpath.  In AIX 7.1 TL3 a new command: lsmpio, has been introduced. This command is similar to lspath, but displays additional information about the MPIO storage devices.

UDID (Unique device identifier)
Every MPIO-capable device must provide a unique identifier that allows the device to be distinguished from any other device in the system. It is called UDID . When the cfgmgr command runs, it requests the UDID for the device and it is compared with the UDIDs stored in ODM to determine if a newly discovered device needs to be defined or the device already exists and only a new path needs to be defined.

--------------------------------------------------------

ODM Update

The AIX MPIO infrastructure allows IBM or third-party storage vendors to supply ODM definitions, which are often referred to as a host attachment kit. A host attachment fileset from the disk vendor is needed to update the ODM to support a specific storage, then AIX can recognize and appropriately configure the disk. Without this, disks are configured using generic ODM definitions. Based on the host attachment filesets disks will be configured as MPIO or non-MPIO devices.

For example if we have a 3rd party storage (like Hitachi or EMC), and we don't install any additional packages, then probably AIX will detect disks as "Other FC SCSI Disk Drive". A device will be discovered as an "MPIO other FC device" only if the device has been certified with one of the AIX default PCMs and no vendor-provided ODM definitions have been installed.  Certification does not guarantee that all device capabilities can be used. (Usually only fail_over is possible but no round_robin or load_balancing.) After the ODM is updated, and the host attachment contained MPIO definitions, the device will be discovered as an MPIO device (for example EMC storage could look like "EMC MPIO FC disks") and one of the AIX PCMs can do the path management. If host attachment contained non-MPIO definitions, then disks will be recognized as non-MPIO (like "EMC CLARiiON FCP RAID 1/0 Disk") and the vendor supplied driver needs to be installed for path management. After the ODM update, the newly discoverd disks will have vendor configured default values (like queue_depth, reserve_policy etc.)

--------------------------------------------------------

MPIO disk settings
https://developer.ibm.com/articles/au-aix-mpio/

# lsattr -El hdisk15
PCM             PCM/friend/fcpother         Path Control Module              False
PR_key_value    none                        Persistant Reserve Key Value     True+
algorithm       round_robin                 Algorithm                        True+
clr_q           no                          Device CLEARS its Queue on error True
dist_err_pcnt   0                           Distributed Error Percentage     True
dist_tw_width   50                          Distributed Error Sample Time    True
hcheck_cmd      inquiry                     Health Check Command             True+
hcheck_interval 60                          Health Check Interval            True+
hcheck_mode     nonactive                   Health Check Mode                True+
...
...
timeout_policy  retry_path                  Timeout Policy                   True+
unique_id       2611200173800005102BE072810XIV03IBMfcp Unique device identifier  False
ww_name         0x500173800051019           FC World Wide Name               False

--------------------

algorithm: (determines how I/O should be distributed across the paths)
- fail_over: I/O is distributed to one path at a time, if it fails next enabled path is selected, depending on path priority.
           VSCSI disks use fail_over, if SCSI-2 reserves is used (reserve_policy=single_path) fail_over is the only possible algorithm.
           By default,  priority is set to 1 (highest priority) and it can range from 1 to 255 (which is the lowest priority).
           set priority: # chpath -l hdisk1 -a priority=255 -p fscsi0 -w 20080022a10bb2d5,1000000000000
           check priority # lspath -l hdisk1 -a priority -F value -p fscsi0 -w 20080022a10bb2d5,1000000000000
   
- round_robin: I/O is distributed to all enabled paths. Paths with same prio. has equal I/O, otherwise higher prio. path utilized more
           To achieve optimal performance during failure, make sure the ordered path list alternate paths between separate fabrics.

- shortest_queue: Similar to round_robin, but when load increases it favors path with fewest I/O  and path priority is ignored.
           If one path is slow (congestion in the SAN), other less-congested paths are used for more I/O

If using SCSI-2 reserves or vSCSI disks, then fail_over must be used. For other situations, shortest_queue (if available) or round_robin enable maximum use of the SAN resources.

--------------------

hcheck_cmd: (the command that will be sent to the disk for healthcheck)
- test_unit_rdy: Test Unit Ready (TUR) command, this is the default setting
- inquiry: In clustered environments with this can have greater control of SCSI reserve and release

In general the default setting should be used, but if there are reservation locks on the disks, use the inquiry option. If test unit ready is used and the backing device is reserved, then test unit ready fails and log an error on the client.

--------------------

hcheck_mode: (determines which paths will be tested by the health checker. Disabled or Missing paths are never checked
(Disabled paths must be recovered manually with chpath, Missing paths with cfgmgr first. If disk is not open (VG is varied off), nohealth checking is done.)

- nonactive: paths with no active I/O are checked (or paths in failed state).
(At round_robin and shortest_queue all paths are busy, so only failed paths are checked. If disk is idle then any paths can be checked (without a pending I/O.))

- enabled: The PCM checks all enabled paths, even paths that have other active I/O

- failed:  PCM checks paths that are marked as failed.

The default value is nonactive, and most of the cases this should not be changed.

--------------------

hcheck_interval:  (interval, in seconds, at which MPIO checks path availability).

- hcheck_interval=0:  disables health check mechanism, any failed paths need to be recovered manually

- hcheck_interval=(small value): the health check commands use the disk’s queue_depth and they receive a higher priority than normal I/O, so a small health check interval can quickly use up a lot of bandwidth on the SAN if there are a large number of disks.

If a device has only one non-failed path and an error is detected on that last path, AIX will send a health check command on all failed paths before retrying the I/O. If there are good paths to use, AIX will discover it and uses it before failing user I/O, regardless of the health check interval setting.

So, for most cases, the default value is appropriate and the general rule for hcheck_interval is, that it should be greater than or equal to rw_timeout. It is much more likely to be a good idea to increase the health check interval than to decrease it. Better performance is achieved when hcheck_interval is slightly greater than the rw_timeout value on the disks.

https://support.purestorage.com/Solutions/IBM/AIX/AIX_Recommended_Settings
PureStorage ODM definition sets hcheck_interval setting to 10 as opposed to IBM recommendation of 30. hcheck_interval setting is set to lower value than the rw_timeout as we are not checking active paths and a lower hcheck_interval will not have any SAN performance impact.

--------------------

timeout_policy: (when an I/O operation fails to complete within the rw_timeout value, what action PCM should take)

- retry_path: command is retried on the same path. This is likely to lead to delays in the I/O recovery.
        (Only after several failures will AIX fail the path and try the I/O on an another path.)

- fail_path:  AIX will fail the path after a single command timeout. Failing the path forces the I/O to be retried on a different path.
         With this setting recovery can be quicker and also the detection of a situation where all paths have failed is also quicker.
         A path that is failed due to timeout policy can later be recovered automatically by the AIX health check commands.

- disable_path: path will be disabled after a single command timeout. Disabled path can be recovered only manually (chpath)

The recommended setting is fail_path.

--------------------------------------------------------

Active/Passive paths

A Storage Processor or Storage Controller manages or controls the disk array at storage side. Many storage arrays have two or more controllers (Storage Processors) and connecting HBA ports to each controllers protects against controller failures. A device that has multiple controllers can designate one controller as the active or preferred controller. For such a device, the PCM uses just the paths to the active or preferred controller as long as there is at least one such path that is enabled and not failed.

Disk arrays can be:
- active/active: it allows access to the LUNs simultaneously through all the storage processors. All the paths are active at all times.
- active/passive: one SP is actively servicing a given LUN. The other SP acts as backup for the LUN (and may be actively serving other LUN I/O).

--------------------------------------------------------

manage_disk_drivers

manage_disk_drivers lists storage models and their drivers. These drivers can be AIX MPIO (like AIX_AAPCM or AIX_APPCM) and non MPIO drivers, when a third-party mulitpath driver is installed. In this case the AIX MPIO feature can be disabled by selecting the AIX_non_MPIO option. (AAPCM means Active/Active and APPCM means Active/Passive)

# manage_disk_drivers -l
Device              Present Driver        Driver Options   
2810XIV             AIX_AAPCM             AIX_AAPCM,AIX_non_MPIO
DS4100              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4200              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4300              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4500              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4700              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS4800              AIX_APPCM             AIX_APPCM,AIX_fcparray
DS3950              AIX_APPCM             AIX_APPCM
DS5020              AIX_APPCM             AIX_APPCM
DCS3700             AIX_APPCM             AIX_APPCM
DS5100/DS5300       AIX_APPCM             AIX_APPCM
DS3500              AIX_APPCM             AIX_APPCM
XIVCTRL             MPIO_XIVCTRL          MPIO_XIVCTRL,nonMPIO_XIVCTRL
2107DS8K            NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
IBMFlash            NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
IBMSVC              NO_OVERRIDE           NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO


NO_OVERRIDE:
The "NO_OVERRIDE" option (as a Present Driver) indicates that the configuration is not overridden by using "manage_disk_drivers -d (device) -o...". If SDDPCM is installed it will substitue AIX MPIO driver for SVC, DS8k (that is why it is not listed by manage_disk_drivers), and by default SDDPCM takes precedence. In this case NO_OVERRIDE would mean SDDPCM will be used (we did not override it by a manual change). If SDDPCM is not installed, then the AIX default PCM is used.

# manage_disk_drivers -d IBMSVC -o AIX_AAPCM     <---use the AIX default PCM even if SDDPCM is installed
# manage_disk_drivers -l                         <--after reboot, AIX PCM is in control (not SDDPCM, which can be uninstalled now)
Device              Present Driver        Driver Options
IBMSVC              AIX_AAPCM             NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO

Changing the driver will reset disk attributes (queue_depth, algorithm, reserve_policy..) to the new default values of the new driver.

--------------------------------------------------------

lsmpio

The command lsmpio was introduced in AIX 7.1 TL3, and it displays information about the MPIO storage devices (those devices which are controlled by PCM). It is similar to lspath, but with much more details.

# lsmpio -l hdisk1234
name   path_id status  path_status     parent connection
===============================================================================
hdisk1234  0   Enabled Opt,Sel,Deg,Rsv fscsi0 500a098186a7d4ca,0008000000000000
hdisk1234  1   Enabled Non             fscsi0 500a098196a7d4ca,0008000000000000
hdisk1234  2   Enabled Opt,Sel         fscsi1 500a098186a7d4ca,0008000000000000
hdisk1234  3   Enabled Non             fscsi1 500a098196a7d4ca,0008000000000000


status: (same as with lspath)
enabled:   path is configured and operational. It will be considered when paths are selected for IO.
disabled:  path has been manually disabled and won't be considered when paths are selected for IO. (set back to enabled with 'chpath')
failed:    path is unusable due to IO failures. It won't be selected for IO. (after problem is fixed, remove path and rediscover it)
defined:   path is not configured and used for io operation (rmpath -l will explicitly turn path off)
missing:   path was not detected after reboot, but it was there earlier in the system (these can be recovered with 'cfgmgr')
detected:  path was detected during boot, but it was not configured (this status should never appear, only during boot)

It is best to manually disable paths before storage maintenance (rmpath). AIX MPIO stops using any disabled or Defined paths, so no error detection or recovery will be done. This ensures that the AIX host does not go into extended error recovery during a scheduled maintenance. After the maintenance is complete, the paths can be re-enabled with cfgmgr. (When disabling multiple paths for multiple LUNs, rmpath is simpler than chpath, as it does not have to be run on a per-disk basis.)

path_status: (more detailed path status)
Opt - optimized path: preferred path, which is attached to a preferred controller. PCM selects preferred paths whenever possible.
Non - non-optimized path: this path is not considered as preferred path. PCM avoids this path, unless all preferred paths fail.
Act - active path: (on a device that has active and passive controllers) the PCM selects active paths for I/O operations.
Pas - passive path: (on a device that has active and passive controllers) the PCM avoids the selection of passive paths.
Sel - selected path: this path is selected for I/O operations when lsmpio command was issued.
Rsv - reservation conflict: could be that multiple hosts accessing the same disk
Fai - failure: path experienced a failure. (If status is Enabled and Fai: MPIO leaves one path in Enabled state, even when all paths have errors.)
Deg - degraded: there were errors, which causing to temporarily avoid to use this path. (Additional errors can cause path to fail)
Clo - closed: if all paths are closed, the device is closed.  If some paths are closed, then those had errors and MPIO tries to periodically recover those.

--------------------------------------------------------

smitty mpio
genkex | grep pcm                    show loaded pcm (like /usr/lib/drivers/sddpcmke)

lspath                                lists paths (lspath -l hdisk46)
lspath -l hdisk0 -HF "name path_id parent connection path_status status"    more detailed info about a device (like lsdev for devices)
lspath -AHE -l hdisk0 -p vscsi0 -w "810000000000"    display attrib. for given path and connection (-w) (-A is like lsattr for devices)
                                      (if only 1 path exist to parent device connection can be omitted: lspath -AHE -l hdisk0 -p vscsi0)
lspath -l hdisk1 -a priority -F value -p fscsi0 -w 20080022a10bb2d5,1000000000000  check priority

chpath                                changing path state (enabbled, disabled)
chpath -s enabled -l hdisk -p vscsi0  it will set the path to enabled status
chpath -l hdisk1 -a priority=255 -p fscsi0 -w 20080022a10bb2d5,1000000000000     change priority

rmpath -l hdiskX -p vscsi0 -w 870000000000   put path in defined state  (-w can be omitted if only 1 path exist to parent device)
rmpath -dl hdiskX -p fcsiY           dynamically remove all paths under a parent adapter from a supported storage MPIO device
                                     (-d: deletes, without it puts it to define state)
                                     (The last path cannot be removed, the command will fail if you try to remove the last path)

mkpath ...                            makes a path into Available state

lsmpio                                lists addtional info about paths (which path is selected)
lsmpio -q                             shows disks with its size
lsmpio -ql hdiskX                     shows disk serial number (LUN ID)
lsmpio -Sl hdisk0 | grep Path         shows path statistics (which path was used mostly in the past)
lsmpio -ar                            list parent adapter and remote port information (-a: adapter (local), -r: remote port)
lsmpio -are                           list error statistics for local and remote (-e: error)
lsmpio -z                             reset statistics

While checking errors in lsmpio -are:
If 1 HBA shows error on all remote storage ports (but other HBA has no errors), then it is a problem with the adapter, cable or switch.
If 2 HBAs show errors on 1 particular storage port, then the error is at the storage port side (or the cable or the switch port.)
If 2 HBAs show erros on the same number of storage ports, then most likely switch has errors (probably at the ISL (inter switch link) between switches)

--------------------------------------------------

Failed path handling:
(there were Hitachi disks in Offline (E) state, but they were not unconfigured earlier)
    -lspath | grep -v Enab
    -rmpath -p fscsiX -d
    -cfgmgr -l fcsX
    -lspath | grep -v Enab
    -dlnkmgr view -lu -item

--------------------------------------------------

Change adapter setting online:

rmpath -d -p vscsi0                             <--removes all paths from adapt. (rmpath -dl hdisk0 -p vscsi0, it removes only specified path)
rmdev -l vscsi0                                 <--puts adapter into defined state
chdev -l vscsi0 -a vscsi_err_recov=fast_fail    <--change adapter setting (if -P is used it will be activated after reboot)
cfgmgr -l vscsi0                                <--configure back adapter

--------------------------------------------------------

SDDPCM migration to AIX PCM
https://www.ibm.com/support/pages/migrate-aixpcm-using-managediskdrivers-command>

Below steps should be done during downtime when no application is using the disks.
If the disks are shared among multiple nodes in a cluster or VIOS configuration, the reservation policy must be checked and changed before the disk is opened. If this is not changed, other nodes in the cluster or VIOS configuration will lose access to the shared disks until the reservation policy is changed.

lsattr -El hdiskX                               <--check current disk attributes (save output) (lsmpio should not work at this point)
manage_disk_drivers -l                          <--list disk drivers
manage_disk_drivers -d IBMSVC -o AIX_AAPCM      <--switching disk driver to MPIO (manage_disk_drivers -l will show new value)
shutdown -Fr                                    <--reboot
lsattr -El hdiskX                               <--check new value (lsmpio should work now)
chdev -P -l hdisk0 -a queue_depth=X -a reserve_policy=no_reserve  <--set back needed values for disks
chdef -a reserve_policy=no_reserve -c disk -s fcp -t mpioosdisk   <--set predefined values in ODM (or chdef -a reserve_policy=no_reserve -c disk -s fcp -t aixmpiods8k)
shutdown -Fr                                    <--reboot
installp -u devices.fcp.disk.ibm.mpio.rte devices.sddpcm.72.rte  <--remove host attachment and sddpcm filesets (if sddpcm is removed both should be removed!!)
shutdown -Fr                                    <--reboot

Check again current and ODM settings to be sure SDDPCM removal did not overwrite something, because removing SDDPCM will set the queue to the default value of 20
If needed to roll back to SDDPCM: # manage_disk_drivers -d IBMSVC -o NO_OVERRIDE

33 comments:

Anonymous said...

some very useful information... Thanks.

Anonymous said...

Hi AIX,

Whether the Hitachi LUNs can be accessed without dlinkmgr software ie., by using default SDDPCM driver.

aix said...

The answer to your question is yes and no.
Yes, Hitachi LUNs can be accessed without dlnkmgr software, but SDDPCM is not for Hitachi LUNs.
You have to differentiate SDDPCM and PCM (native AIX PCM)
SDDPCM is for IBM storage only, for this you need to install additional fileset, for example devices.sddpcm53.rte and then you can use pcmpath commands. But for native AIX PCM (MPIO) you don't have to install addtional software. AIX, with a base operating system install, is capable to use some third party devices (ie. Hitachi) as MPIO devices (lspath, rmpath...)
But you need to ask Hitachi support as well, if the model you have is capable for using this way.

aix said...

Hi,

I have received this from Jose, and I would like to share:

"I was interested in a summary table of my disks so I wrote the below script, pcmpath query essmap was as well ok for me but gave me further info

display the disk in lspath in “ No hdiskxx size mb No-paths” with no root authority
p="/usr/sbin/lspath";for i in `$p| awk ' !/Missing/ {print $2}'|sort|uniq `;do echo "$i; `getconf DISK_SIZE /dev/$i` mb; `$p| awk ' !/Missing/ &&/'$i' / {print $2}'|wc -l|sed 's/ //g'`" ; done|cat -n

Output
1 hdisk0; 70006 mb; 1
2 hdisk1; 70006 mb; 1
3 hdisk2; 20480 mb; 4
4 hdisk25; 20480 mb; 4
5 hdisk26; 20480 mb; 4"

Anonymous said...

Very helpful.
Is there anyway, I can locate the network adapter location by turn-on the light, as I have 3 Network adapter in P550 Machine in production environment. My oslevel is AIX52TL4.
Thanks in advance.

aix said...

Hi, on AIX 5.3, if you issue the command: diag -> Task Selection -> Hot Plug Task -> PCI Hot Plug Manager -> Identify a PCI Hot Plug Slot
This will blink the light at that location on AIX 5.3, you should try, maybe it works on AIX 5.2 as well

Anonymous said...

Thank for your help. It work.

aix said...

Welcome :)

Siva said...

Hi,

Please explain briefly about Missing, Failed and Defined states in lspath output.

Regards,
Siva

Anonymous said...

hello,

if I have the command pcmpath then I have everything right sddpcm? stg ds8k

aix said...

hi, I would say yes...if it works correctly.

aix said...

Hi,

failed
Indicates that the path is configured, but it has had IO failures that have rendered it unusable. It will not be considered when paths are selected for IO.
defined
Indicates that the path has not been configured into the device driver.
missing
Indicates that the path was defined in a previous boot, but it was not detected in the most recent boot of the system.

Anonymous said...

i am having MPIO on vio , i use lspath display 1 of the SAN disk in vio server.
# lspath -l hdisk12 -H -F "name parent connection path_id"
name parent connection path_id

hdisk12 fscsi0 500507630700067a,4060400500000000 0
hdisk12 fscsi0 50050763070b067a,4060400500000000 1
hdisk12 fscsi1 500507630710067a,4060400500000000 2
hdisk12 fscsi1 50050763071b067a,4060400500000000 3

how to explain connection and path_id ?

aix said...

Hi,
both of them can be used tu uniquely identify paths, for example with chpath commands.

The connection information differentiates the multiple path instances that share the same logical parent (adapter). (SCSI ID and LUN ID of the device associated with this path.)


path_id: Indicates the ID of the path, it is used to uniquely identify a path

Anonymous said...

Hi,

enabled
Indicates that the path is configured and operational. It will be considered when paths are selected for IO.
disabled
Indicates that the path is configured, but not currently operational. It has been manually disabled and will not be considered when paths are selected for
IO.
failed
Indicates that the path is configured, but it has had IO failures that have rendered it unusable. It will not be considered when paths are selected for
IO.

defined
Indicates that the path has not been configured into the device driver.
missing
Indicates that the path was defined in a previous boot, but it was not detected in the most recent boot of the system.
detected
Indicates that the path was detected in the most recent boot of the system, but for some reason it was not configured. A path should only have this status
during boot and so this status should never appear as a result of the lspath command.
Regards
Virender Kumar

Anonymous said...

Hi,

I need to findlut the MPIO package version which is installed in AIX. And need to know which version of HBA cards are using?

Thank you

Anonymous said...

Hi,

how can we check the disk raid level from AIX 7.1 Machine.

Note: In AIX 5.3 it is #lsattr -El hdisk8 | grep -i raid

Thanks

Marcel said...

Hi ,
try the following :
# lslpp -L '-a' devices.common.IBM.mpio.rte
or :
# smit list_installed

Anonymous said...

Hi AIX man !
I have this environment were a HDS disk is connected to a LPAR. The point is, even with mpio installed, there is not 2 paths to the disks. So, my question is, what are the required packages required at AIX to configure multipath?

AIX 6100-04
devices.common.IBM.mpio.rte
6.1.5.0 COMMITTED MPIO Disk Path Control Module
devices.fcp.disk.Hitachi.array.mpio.rte
5.4.0.0 COMMITTED AIX MPIO Support for Hitachi
5.4.0.1 APPLIED AIX MPIO Support for Hitachi
5.4.0.2 APPLIED AIX MPIO Support for Hitachi
5.4.0.3 APPLIED AIX MPIO Support for Hitachi
5.4.0.4 APPLIED AIX MPIO Support for Hitachi
5.4.0.5 APPLIED AIX MPIO Support for Hitachi
devices.fcp.disk.Hitachi.modular.mpio.rte
6.0.0.0 COMMITTED AIX MPIO Support for Hitachi
6.0.0.1 APPLIED AIX MPIO Support for Hitachi
devices.common.IBM.mpio.rte
6.1.5.0 COMMITTED MPIO Disk Path Control Module

Anonymous said...

Hi All
I have a question ? I am trying to install Hitachi software 6001 which is already their in my . file after running everything the output is coming failed.
This is what i am getting can anyone tell me where i am doing mistake ?

cannot open /output/hitachi/odm/mpio/6000/HTC_MPIO_Modular_ODM_6000I: No such file or directory
Please mount volume 1 on /output/hitachi/odm/mpio/6000/HTC_MPIO_Modular_ODM_6000I
...and press Enter to continue installp: An error occurred while running the restore command.
Use local problem reporting procedures.

installp: CANCELED software for:
devices.fcp.disk.Hitachi.modular.mpio.rte 6.0.0.0

Thanks.

yogesh c said...

How to collect the MPIO related error logs/ event logs on AIX?

Anonymous said...

Hi Admin,
please could you help me to understand the difference between MPIO and SDDPCM .
also, what are the advantages for moving from MPIO/SDD to SDDPCM ?

Thanks in adv.

lifer said...

I have have question about reserve_policy=single_path . If this setting is configured, does it stops from other HBA loging (PLOGI) on to storage?

Michael said...

Is there any timeouts in FC adapter (fcsX) or device driver (fscsiX) that we can tune?
we have a need to extend the time that AIX spent on the alternate path when failover.
We observed AIX failed quickly when the LUNs on the alternate path were not yet up,
and when AIX failed and bubbled up the error to the application, the application failed.
In other OSes, we can extend this duration to 300 seconds but in AIX we do not know
what it is and what is the default value. lsattr -El fcsX or fscsiX do not show any
relevant attributes.

Thanks.
Michael

aix said...

Probably there are some parameters in MPIO which could help, I suggest checking MPIO best practices: https://www.ibm.com/developerworks/aix/library/au-aix-mpio/

Michael said...

Thank you very much for the reply.

I came across the following statement in the page you mentioned:

https://www.ibm.com/developerworks/aix/library/au-aix-mpio/

"AIX implements an emergency last gasp health check to recover paths when needed. If a device has only one non-failed path and an error is detected on that last path, AIX sends a health check command on all of the other failed paths before retrying the I/O, regardless of the health check interval setting. This eliminates the need for a small health check interval to recover paths quickly. If there is at least one good path, AIX discovers it and uses it before failing user I/O, regardless of the health check interval setting."

From the traces we have, since the primary path was gone (rebooted and switch sent RSCN), the alternate path was still alive just the LUNs were in transition (not yet active), AIX checked the state of LUNs (using Test Unit Ready first, then resent 8 failed IOs, then sent Start/Stop Unit and 8 failed IOs, then repeat the Start/Stop Unit and 8 IOs sequences), after N seconds, AIX failed the IOs and the application on AIX failed. We yet to capture a longer traces which shows the duration of this check, but we need to know what timeout it is, what is the default value, can it be extended. We need AIX to check at most 300 seconds so the application on AIX can survive the LUNs takeover/failback. Thanks.

Michael said...

From the application log, the entire LUNs takeover took around 80 seconds, and AIX spent 15-20 seconds on checking the primary path before failover to alternate path, so the AIX spent less than 60 seconds checking the alternate path (failed user IOs after that). The length of time for LUNs takeover varies, but in our test we used a predefined configuration for verification. Thanks.

Anonymous said...

Hi,

Can anyone advise me on this pls..
While Storage array(EMC VNX)SP's rebooting AIX LPARS went to hung state due to VIO not able to failover the paths.what could be the reason like why MPIO is not worked on VIO as expected?

Anonymous said...

Hi,

i need to mirror the rootvg by removing alt disk, but it showing two disks for each vg & it looks like bellow. pls suggest me. recently VCS upgrade was done.

# lspv |grep root
hdisk4 00c9445e536966b4 altinst_rootvg
hdisk5 00f6a9b3db3026ce rootvg active
hdisk6 00f6a9b3db3026ce rootvg active
hdisk7 00c9445e536966b4 altinst_rootvg

Thamba said...

Hi All,

My company is introducing PURE storage for AIX server.
we have a combination of vscsi + NPIV close to 350 AIX client partitions.

Just started implementing and came to know there is no inbuilt command or package to check the local storage LUNS on the AIX client server.

Has anyone gone through this situation or i am looking for any workaround for the same.

Currently using NETAPP and i have been using this SANLUN inbuilt commands so far.

Regards,
Thamba

Unknown said...

How to get the LUN Name in storage to Hdisk value in OS ? Earlier we used "mipo_get_config -Av" command with IBM DS storage for this

aix said...

lsmpio -ql hdiskX

Ziya said...
This comment has been removed by the author.