dropdown menu

STORAGE - EMC

EMC:

EMC comes with the default settings. These settings are set by EMC technicians for the optimized performance.
It is not recommended to change these configurations:

queue_depth=16
init_link     al
dyntrk       yes
fc_err_recov fast_fail

Because of queue_depth attributes which is 16 you may need to change num_cmd_elems attributes on FC card too.
num_cmd_elems >= queue_depth * LUN numbers.

Maximum num_cmd_elems  could be 1024 OR 2048 which depend FC card type.

------------------
installed filesets:

root@aix14: /etc/emc # lslpp -L | grep EMC
  EMC.Symmetrix.aix.rte      5.3.0.3    C     F    EMC Symmetrix AIX Support
  EMC.Symmetrix.fcp.rte      5.3.0.3    C     F    EMC Symmetrix FCP Support
  EMCpower.base              5.3.1.0    C     F    PowerPath Base Driver and
  EMCpower.encryption        5.3.1.0    C     F    PowerPath Encryption with RSA
  EMCpower.migration_enabler
  EMCpower.mpx               5.3.1.0    C     F    PowerPath Multi_Pathing

------------------

EMCPOWERRESET:

EMC has developed a binary called emcpowerreset for removing disk reservations, held by PowerPath devices, in the event that a node crashes. This binary is required for any HACMP installations on AIX 5.1, and higher when running PowerPath version 3.0.3 and higher.

To determine the different emcpowerreset versions, run the following command:

cksum emcpowerreset (or cksum /usr/lpp/EMC/Symmetrix/bin/emcpowerreset)

Version 1 = 1108394902 7867
Version 2 = 1955156125 10311

The emcpowerreset binary takes as options two parameters, and these two parameters are automatically passed to the binary whenever it is invoked within the HACMP script logic.


Reset reservation bit:
If you run into not being able to access an hdiskpowerX disk, you may need to reset the reservation bit on it:

# /usr/lpp/EMC/Symmetrix/bin/emcpowerreset fscsiX hdiskpowerX

------------------

smitty powerpath

powermt version                    shows installed powerpath version
powermt display                    shows short overview of hba, paths, managed classes
powermt display options            shows some settings for PowerPath
powermt display paths              shows the paths and dead paths to the storage port
powermt display ports              shows the storage ports information
powermt display dev=all            shows detailed info of all devices
powermt display dev=hdiskpowerX    shows detailed info of the specified device
powermt display every=<x seconds>  shows io stats on each adapter in the specified interval (good for checking load balancing)

/usr/lpp/EMC/Symmetrix/bin # ./inq.aix64_51    shows detailed info about the disks (type, serial numbers...)

powermt remove hba=1 dev=all   removes all pathes from an hba (it asks before reomving the last active path)
powermt config                 configures back the pathes

powermt disable hba=0          disables specified hba (it will show failed, errpt entry will also generated) - enable is needed later
powermt enable hba=0           enables the specified hba

powermt remove hba=2 dev=0     remove only 1 path from the specified device (powermt remove hba=2 dev=hdisk11)

powermt set mode=standby hba=1 dev=all    sets the adapter to standby mode (it will be activated only if the other path failed)

powermt unmanage class=hitachi powermt will not manage any hitachi devices
powermt manage class=hitachi   powermt will manage hitachi devices (not recommended, can cause trouble)

powermt save file=<path>       saves powerpath configuration settings
powermt load file=<path>       loads back powerpath configuration settings

powermt restore dev=all        it does an I/O path check, if a previously dead path is alive, it will be marked as alive
                               (If you have dead I/O paths, and you fixed something, you can request PowerPath to re-check the paths.)

powermt check                  check the I/O Paths
                               (for example, if you manually removed an I/O path, it will detect a dead path and remove it from the list.)

lsdev -Ct power                shows the powerpath0 device (which is the pseudo parent of the hdiskpowerX devices)
rmdev -dl powerpath0           this is needed for the full removing of powerpath

powermt watch every=5          it shows in 5 seconds interval fc adapter statistics (IO/sec). It is good for checkng if load balancing works.

--------------------------
removing a LUN:
powermt display dev=19
powermt remove dev=19
rmdev -dl hdiskpower19
rmdev -dl hdisk59
rmdev -dl hdisk77
--------------------------

"powermt check" checks specified paths and, if desired, removes from the PowerPath configuration any paths marked dead.

 Warning: storage_system I/O path path_name is dead
 Do you want to remove it (y/n/a/q)?

 Valid responses are:

 y    Remove the dead path, and continue checking remaining paths.
 n    Do not remove the path, but continue checking remaining paths.
 a    Remove the dead path and any subsequent paths marked dead.
 q    Do not remove the dead path, and exit the command. Any paths that were already removed remain removed.

--------------------------

Install:

0. preparations:
    -set FC adapter settings if needed: (init_link, dyntrk, fc_err_recov, num_cmd_elems)
    -remove already assigned EMC disks (if you see Other  MPIO... (which is EMC) remove them, so cfgmgr will bring up correctly later)
   

1. ODM extension and Powerpath:
    -ODM_def_EMC.AIX.5.3.0.3:
        install only these:
            -EMC Symmetrix AIX Support Software
            -EMC Symmetrix FCP Support Software

  (Symmetrix MPIO and Symmmetrix Powerpath can't be installed on 1 system together.)

    -Powerpath_AIX.5.3.SP1:
        EMCpower: everything can be installed


2. add licence key (/mnt/Storage-Treiber/EMC/PowerPath_license)
    emcpreg -add XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

    you can check if it is added:
    emcpreg -list or powermt check_registration

  (license key can be copied from an already installed system)


3. specifics for RAC,VIO (dual) and HACMP


    RAC and VIO (dual):
    set the reservation setting to no reserve in the ODM

    we have a script: /mnt/Storage-Treiber/EMC/emc_reserve.sh
        you will be asked:
        1. Set reserve_lock=no in the PdAt for Symmetrix FCP devices.
        2. Set reserve_lock=yes in the PdAt for Symmetrix FCP devices.

        choose #1

    you can check the settings in the ODM: odmget PdAt | grep -p reserve | grep -p SYMMETRIX
    (deflt. should be on no)
    (reserve should be removed only for dual vio environment)


    HACMP:
    (full instruction: Host Connectivity Guide pg. 348)
    emcpowerreset utilit should be run, for HACMP to be able handle disk reservations correctly.
   
    It can be found here: /usr/lpp/EMC/Symmetrix/bin
    (makes sure it exists and root is the owner)

    smitty hacmp -> Ext. Conf. -> Ext. Res. Conf. -> HACMP Ext. Resources Conf. -> Conf. Custom Disk Methods -> Add Cust. Disk

    * Disk Type (PdDvLn field from CuDv)                 [disk/pseudo/power]
    * Method to identify ghost disks                     [SCSI3]                                                      +
    * Method to determine if a reserve is held           [SCSI_TUR]                                                   +
    * Method to break a reserve                          [/usr/lpp/EMC/Symmetrix/bin/emcpowerreset]                   +
        Break reserves in parallel                        true                                                        +
    * Method to make the disk available                  [MKDEV]                                                      +
   
    Configure the same custom disk processing method on each node in the cluster and synchronize the cluster resources.
    (if you want NFS exports, same major numbers are needed for the hdispowerX devices)

4. Configure Powerpath to not manage other disks (ESS, Hitachi)
    (this is very important, otherwise Powerpath try to manage them, and can cause problems)
    change this  (with vi) in /etc/emc/mpaa.lams

    root@aix14: /etc/emc # cat mpaa.lams
    global:version:5.3.0
    managed:symm
    managed:clariion
    unmanaged:hitachi
    unmanaged:invista
    unmanaged:hpxp
    unmanaged:ess
    unmanaged:hphsx

    (this file will be used when configuring the paths)


5. Run config manager

    cfgmgr -S
    (this will config the devices sequentially. This way recommended to avoid troubles of parallel config.)
    (normal cfgmgr is OK as well)

    if needed, run : powermt config

    you can setup this link as well:
    ln -s /usr/lpp/EMC/Symmetrix/bin/inq.aix64_51 /usr/sbin/inq
--------------------------

problem with bosboot after pprootdev fix:

pprootdev fix, can help if rootvg is on hdiskpower devices and we have problems with bosboot.
"bosboot -a" wants to use that disk, what is used during boot (bootinfo -b), this disk should be in the output of "lsvg -p" after we did pprootdev fix. It can happen that pprootdev fix will put rootvg to a different disk, so bosboot will fail

Solution: put the unneeded paths to failed state, then do pprootdev fix, so rootvg will be only on that disk what is available (what is used during boot). Then the other paths can be put back to available.

Commands for determining which disk are needed:
-odmget -q"name=rootvg and attribute=pv" CuAt | grep value
        value = "00080e82a47d2e5d0000000000000000"
        value = "00080e828106fa470000000000000000"


-odmget -q "value=00080e828106fa470000000000000000 and attribute=pvid" CuAt        <--this disks list is important
CuAt:
        name = "hdisk12"
        attribute = "pvid"
        value = "00080e828106fa470000000000000000"
        type = "R"
        generic = "D"
        rep = "s"
        nls_index = 2

CuAt:
        name = "hdisk8"
        attribute = "pvid"
        value = "00080e828106fa470000000000000000"
        type = "R"
        generic = "D"
        rep = "s"
        nls_index = 2

--------------------------

emcp_xcryptd has high CPU usage:

This daemon is doing encryption for EMC. It is usually not in use in AIX.

You can check if encryption is being used and disbale this process with these steps:

# powervt xcrypt -info -dev all            <--lists all hdiskpower devices, if it is encrypted or not
# /etc/rc.emcp_xcryptd stop                <--stops emc_xryptd
# rmitab rcxcrypt                          <--remove from inittab


15 comments:

Anonymous said...

Hey, I think your website might be having browser compatibility issues.
When I look at your blog site in Firefox, it looks fine but when opening in Internet
Explorer, it has some overlapping. I just wanted to give you a
quick heads up! Other then that, great blog!
Also see my web page :: click here

aix said...

Yes... I'm still looking for solution, looked over many many tips for "drop down menu problem with IE", but did not work for me :(
Your blog is very clear and the design is awesome, but I saw you might have the same problem. If I select "CLIENTS" on the main menu, then I put mouse over "COMPANY", I see the same overlapping. If you could fix that and share the solution with me...I would be really grateful.

Anonymous said...

I have a question...when a lun is assigned to aix server, how to identify those disk?? Is there any naming convention like hdiskpower.

aix said...

If you use EMC storage, "powermt display dev=all" will show LUN id of all disks.

Anonymous said...

Hi,

After the EMC ODM and powerpath upgrade, the powerpath associated disks went to defined state and are not listed under powermt display o/p

What I did is mentioned below

1) Upgraded EMC ODM from 5.3 to 6.0
2) varied off datavg
3) stopped the navi agent
4) Upgraded powerpath from 5.0 to 5.5
5) started navi agent, cfgmgr, powermt config, powermt save

But it did not display the devices
lsedv -Cc disk, shows the powerpath deivces in defined state.
powermt registeration looks good

Can you help on this

aix said...

Hi, your steps looks good, I would do the same...probably I would check if powerpath upgrade can be done in 1 step from 5.0 to 5.5 or it needs 2 steps. I would suggest to open an EMC call...

Anonymous said...

Hey Gurus,
Have you got the solution for this powerpath upgrade issue which u face, pls share ur findings...

Anonymous said...

Hello

Was wondering if there is a setting in the VMAX SAN STORAGE for a driver so that NODE A and NODE B can see the same set of DISKS from the SAN

Thank You


kmalusare said...

we have faced the similar issue during and after powerpath upgrade ....... issue we found is reservation policy of some of the hdisks ( manging hdiskpower ) was changed ....in powerpath reservation policy of ALL the hdisks controlling hdiskpower should be same ..i.e. either it can be single_path or no_reserve

Anonymous said...

This is old, but thought I'd answer anyway. Upgrading powerpath from 5.0 to 5.5 should not be done as an actual upgrade. Always remove the old, and install the new, part of which is removing all of your hdiskpowerX devices. The reason to do an uninstall/install is the powerpath0 device. Just upgrading the software doesn't recreate this device, as well as EMC is not good about cleaning up old ODM entries. After installing the new software, DO NOT run cfgmgr. You have to do a "powermt config" first, then a cfgmgr, otherwise, the powerpath0 device will not be created prior to creating hdiskpower devices, and your device major ids may incur a conflict.

aix said...

Thx a lot for this description!

Yogesh Malkoti said...

what is the recommendation for reserve_policy to use for EMC storage in shared GPFS environment. current settings are pr_exclusive with hitachi storage and it working well. As we are moving to emc storage, EMC recommends to use pr_shared. what is the difference between those, since pr_exclusive also working fine.

Anonymous said...

i am getting below error while clearing the pvid of powerdisk.
Method error (/etc/methods/chgpowerdisk):
0514-062 Cannot perform the requested function because the
specified device is busy.

how to clear this

Storage Tutorials said...

Very informative. Keep sharing. Let us know if you can submit guest articles on our website too.

XFS Filesystem said...

This blog is awesome containing plethora of information.