STORAGE - EMC

EMC:

EMC comes with the default settings. These settings are set by EMC technicians for the optimized performance.
It is not recommended to change these configurations:

queue_depth=16
init_link     al
dyntrk       yes
fc_err_recov fast_fail

Because of queue_depth attributes which is 16 you may need to change num_cmd_elems attributes on FC card too.
num_cmd_elems >= queue_depth * LUN numbers.

Maximum num_cmd_elems  could be 1024 OR 2048 which depend FC card type.

------------------
installed filesets:

root@aix14: /etc/emc # lslpp -L | grep EMC
  EMC.Symmetrix.aix.rte      5.3.0.3    C     F    EMC Symmetrix AIX Support
  EMC.Symmetrix.fcp.rte      5.3.0.3    C     F    EMC Symmetrix FCP Support
  EMCpower.base              5.3.1.0    C     F    PowerPath Base Driver and
  EMCpower.encryption        5.3.1.0    C     F    PowerPath Encryption with RSA
  EMCpower.migration_enabler
  EMCpower.mpx               5.3.1.0    C     F    PowerPath Multi_Pathing

------------------

EMCPOWERRESET:

EMC has developed a binary called emcpowerreset for removing disk reservations, held by PowerPath devices, in the event that a node crashes. This binary is required for any HACMP installations on AIX 5.1, and higher when running PowerPath version 3.0.3 and higher.

To determine the different emcpowerreset versions, run the following command:

cksum emcpowerreset (or cksum /usr/lpp/EMC/Symmetrix/bin/emcpowerreset)

Version 1 = 1108394902 7867
Version 2 = 1955156125 10311

The emcpowerreset binary takes as options two parameters, and these two parameters are automatically passed to the binary whenever it is invoked within the HACMP script logic.


Reset reservation bit:
If you run into not being able to access an hdiskpowerX disk, you may need to reset the reservation bit on it:

# /usr/lpp/EMC/Symmetrix/bin/emcpowerreset fscsiX hdiskpowerX

------------------

smitty powerpath

powermt version                    shows installed powerpath version
powermt display                    shows short overview of hba, paths, managed classes
powermt display options            shows some settings for PowerPath
powermt display paths              shows the paths and dead paths to the storage port
powermt display ports              shows the storage ports information
powermt display dev=all            shows detailed info of all devices
powermt display dev=hdiskpowerX    shows detailed info of the specified device
powermt display every=<x seconds>  shows io stats on each adapter in the specified interval (good for checking load balancing)

/usr/lpp/EMC/Symmetrix/bin # ./inq.aix64_51    shows detailed info about the disks (type, serial numbers...)

powermt remove hba=1 dev=all   removes all pathes from an hba (it asks before reomving the last active path)
powermt config                 configures back the pathes

powermt disable hba=0          disables specified hba (it will show failed, errpt entry will also generated) - enable is needed later
powermt enable hba=0           enables the specified hba

powermt remove hba=2 dev=0     remove only 1 path from the specified device (powermt remove hba=2 dev=hdisk11)

powermt set mode=standby hba=1 dev=all    sets the adapter to standby mode (it will be activated only if the other path failed)

powermt unmanage class=hitachi powermt will not manage any hitachi devices
powermt manage class=hitachi   powermt will manage hitachi devices (not recommended, can cause trouble)

powermt save file=<path>       saves powerpath configuration settings
powermt load file=<path>       loads back powerpath configuration settings

powermt restore dev=all        it does an I/O path check, if a previously dead path is alive, it will be marked as alive
                               (If you have dead I/O paths, and you fixed something, you can request PowerPath to re-check the paths.)

powermt check                  check the I/O Paths
                               (for example, if you manually removed an I/O path, it will detect a dead path and remove it from the list.)

lsdev -Ct power                shows the powerpath0 device (which is the pseudo parent of the hdiskpowerX devices)
rmdev -dl powerpath0           this is needed for the full removing of powerpath

powermt watch every=5          it shows in 5 seconds interval fc adapter statistics (IO/sec). It is good for checkng if load balancing works.

--------------------------
removing a LUN:
powermt display dev=19
powermt remove dev=19
rmdev -dl hdiskpower19
rmdev -dl hdisk59
rmdev -dl hdisk77
--------------------------

"powermt check" checks specified paths and, if desired, removes from the PowerPath configuration any paths marked dead.

 Warning: storage_system I/O path path_name is dead
 Do you want to remove it (y/n/a/q)?

 Valid responses are:

 y    Remove the dead path, and continue checking remaining paths.
 n    Do not remove the path, but continue checking remaining paths.
 a    Remove the dead path and any subsequent paths marked dead.
 q    Do not remove the dead path, and exit the command. Any paths that were already removed remain removed.

--------------------------

Install:

0. preparations:
    -set FC adapter settings if needed: (init_link, dyntrk, fc_err_recov, num_cmd_elems)
    -remove already assigned EMC disks (if you see Other  MPIO... (which is EMC) remove them, so cfgmgr will bring up correctly later)
   

1. ODM extension and Powerpath:
    -ODM_def_EMC.AIX.5.3.0.3:
        install only these:
            -EMC Symmetrix AIX Support Software
            -EMC Symmetrix FCP Support Software

  (Symmetrix MPIO and Symmmetrix Powerpath can't be installed on 1 system together.)

    -Powerpath_AIX.5.3.SP1:
        EMCpower: everything can be installed


2. add licence key (/mnt/Storage-Treiber/EMC/PowerPath_license)
    emcpreg -add XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

    you can check if it is added:
    emcpreg -list or powermt check_registration

  (license key can be copied from an already installed system)


3. specifics for RAC,VIO (dual) and HACMP

    RAC and VIO (dual):
    set the reservation setting to no reserve in the ODM

    we have a script: /mnt/Storage-Treiber/EMC/emc_reserve.sh
        you will be asked:
        1. Set reserve_lock=no in the PdAt for Symmetrix FCP devices.
        2. Set reserve_lock=yes in the PdAt for Symmetrix FCP devices.

        choose #1

    you can check the settings in the ODM: odmget PdAt | grep -p reserve | grep -p SYMMETRIX
    (deflt. should be on no)
    (reserve should be removed only for dual vio environment)


    HACMP:
    (full instruction: Host Connectivity Guide pg. 348)
    emcpowerreset utilit should be run, for HACMP to be able handle disk reservations correctly.
   
    It can be found here: /usr/lpp/EMC/Symmetrix/bin
    (makes sure it exists and root is the owner)

    smitty hacmp -> Ext. Conf. -> Ext. Res. Conf. -> HACMP Ext. Resources Conf. -> Conf. Custom Disk Methods -> Add Cust. Disk

    * Disk Type (PdDvLn field from CuDv)                 [disk/pseudo/power]
    * Method to identify ghost disks                     [SCSI3]                                                      +
    * Method to determine if a reserve is held           [SCSI_TUR]                                                   +
    * Method to break a reserve                          [/usr/lpp/EMC/Symmetrix/bin/emcpowerreset]                   +
        Break reserves in parallel                        true                                                        +
    * Method to make the disk available                  [MKDEV]                                                      +
   
    Configure the same custom disk processing method on each node in the cluster and synchronize the cluster resources.
    (if you want NFS exports, same major numbers are needed for the hdispowerX devices)

4. Configure Powerpath to not manage other disks (ESS, Hitachi)
    (this is very important, otherwise Powerpath try to manage them, and can cause problems)
    change this  (with vi) in /etc/emc/mpaa.lams

    root@aix14: /etc/emc # cat mpaa.lams
    global:version:5.3.0
    managed:symm
    managed:clariion
    unmanaged:hitachi
    unmanaged:invista
    unmanaged:hpxp
    unmanaged:ess
    unmanaged:hphsx

    (this file will be used when configuring the paths)


5. Run config manager

    cfgmgr -S
    (this will config the devices sequentially. This way recommended to avoid troubles of parallel config.)
    (normal cfgmgr is OK as well)

    if needed, run : powermt config

    you can setup this link as well:
    ln -s /usr/lpp/EMC/Symmetrix/bin/inq.aix64_51 /usr/sbin/inq
--------------------------

problem with bosboot after pprootdev fix:

pprootdev fix, can help if rootvg is on hdiskpower devices and we have problems with bosboot.
"bosboot -a" wants to use that disk, what is used during boot (bootinfo -b), this disk should be in the output of "lsvg -p" after we did pprootdev fix. It can happen that pprootdev fix will put rootvg to a different disk, so bosboot will fail

Solution: put the unneeded paths to failed state, then do pprootdev fix, so rootvg will be only on that disk what is available (what is used during boot). Then the other paths can be put back to available.

Commands for determining which disk are needed:
-odmget -q"name=rootvg and attribute=pv" CuAt | grep value
        value = "00080e82a47d2e5d0000000000000000"
        value = "00080e828106fa470000000000000000"


-odmget -q "value=00080e828106fa470000000000000000 and attribute=pvid" CuAt        <--this disks list is important
CuAt:
        name = "hdisk12"
        attribute = "pvid"
        value = "00080e828106fa470000000000000000"
        type = "R"
        generic = "D"
        rep = "s"
        nls_index = 2

CuAt:
        name = "hdisk8"
        attribute = "pvid"
        value = "00080e828106fa470000000000000000"
        type = "R"
        generic = "D"
        rep = "s"
        nls_index = 2

--------------------------

emcp_xcryptd has high CPU usage:

This daemon is doing encryption for EMC. It is usually not in use in AIX.

You can check if encryption is being used and disbale this process with these steps:

# powervt xcrypt -info -dev all            <--lists all hdiskpower devices, if it is encrypted or not
# /etc/rc.emcp_xcryptd stop                <--stops emc_xryptd
# rmitab rcxcrypt                          <--remove from inittab