Adapter

Adapter

Connector can be PCI:

-FC HBA - Fibre Channel Host Bus Adapter
    fcsx - hardware adapter,
    fscsix - fc-scsi protocol device (logical device) (scsi protocol over the fc protocol)
     
   fcstat fcsX                 fiber channel statistics report


-SCSI HBA - SCSI Host Bus Adapter

    sisscsiaX - SCSI ADAPTER (internal disks are connected to this)

    lsdev -C | grep sisscsia   list the scsi adapters
    lsdev -Cs scsi -H          lists the scsi devices with location code

----------------------------------------
A host bus adapter is a physical Fibre Channel adapter that can be assigned to a logical partition.
----------------------------------------
----------------------------------------

ADAPTER SETTINGS:

root@aix21: / # lsattr -El fscsi0
attach       switch       How this adapter is CONNECTED         False   <-- shows actual connection protocol (None,switch,al lehet)
dyntrk       no           Dynamic Tracking of FC Devices        True    <-- dynamically follows SAN changes
fc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True    <-- 3. delayed_fail or fast_fail
scsi_id      0x133300     Adapter SCSI ID                       False
sw_fc_class  3            FC Class for Fabric                   True

root@aix21: / # lsattr -El fcs0
bus_intr_lvl  305        Bus interrupt level                                False
bus_io_addr   0xffc00    Bus I/O address                                    False
bus_mem_addr  0xfffbf000 Bus memory address                                 False
init_link     al         INIT Link flags                                    True    <--al or pt2pt
intr_priority 3          Interrupt priority                                 False
lg_term_dma   0x800000   Long term DMA                                      True    <--deafault 0x200000, max. 0x8000000
max_xfer_size 0x100000   Maximum Transfer Size                              True    <--default is 100000, max 1000000
num_cmd_elems 200        Maximum number of COMMANDS to queue to the adapter True    <--default 200, max is 1024 or 2048
pref_alpa     0x1        Preferred AL_PA                                    True
sw_fc_class   2          FC Class for Fabric                                True

(Below descriptions are from "DS8800 Performance Monitoring and Tuning" Redbook.)

attach:(sets the Fiber Channel topology)
    none: if no cable has been plugged in
    al: all devices are in a loop or ring, similar to token ring networking. The failure of one device causes a break in the ring.
    switch: all devices or loops of devices are connected to Fibre Channel switches, similar conceptually to modern Ethernet implementations  

dyntrk:
    If it is on yes, it enables dynamic changing of fibre channel cable connections on switch ports or storage ports.
    We have changed few StoragePorts to another Switch last saturday ==> that means, that the HardwarePath changed for these servers,
    where the Option "dyntrk" is set to no. --> reconfigure your disks

fc_err_recov:
    It can enhance fast I/O Failure for FC devices after link events (like link lost) in a switched environment.
    delayed_fail: waits some time before acting.
    fast_fail is useful where multipathing software is used. This would support faster failover to alternate paths

init_link:
    from online help: "Do not change this attribute unless directed by IBM support."
    al: arbitrared loop : first tries al, then pt2pt, if not succes then link remains down
    pt2pt: tries pt2pt, if not succes then link remains down

lg_term_dma:
    it controls the DMA memory resource that an adapter driver can use (default value is 0x200000, maximum value is 0x8000000)
    One change is to increase the value of lg_term_dma to 0x400000 or higher.
    If you have a dual port adapter the maximum value is divided between the adapter ports.
    Using the maximum value for a dual-port adapter can cause the second port performance to fail

max_xfer_size:
    Controls the maximum I/O size the adapter device driver will handle and the memory area used by the adapter.
    Default value is max_xfer_size=0x100000, which means the memory area is 16 MB.
    2 I/O DMA settings are available 16MB and 256/128MB: lsattr -l fcs0 -a max_xfer_size -R
    0x100000    <= 16 M
    0x200000    <= 256 M or 128 M depending on PCI Bus Architecture
    0x400000    <= 256 M or 128 M depending on PCI Bus Architecture
    ...
    Sometimes changed to 0x200000 which increases the memory region size to 128MB and potentially can improve the adapter’s bandwidth.
    if DMA_ERR appears in errpt(which can casuse SAN booting issues too), then chaning back max_xfer_size to the default value can help
    Since AIX 6.1 the DMA memory area for virt. FC adapters is always 128 MB even with the default max_xfer_size
    (max_xfer_size is a DMA memory area, but it is different from the lg_term_dma (which is used for I/O control))

num_cmd_elems:
    Maximum number of commands to be queued to the adapter (range can be checked: lsattr -Rl fcs0 -a num_cmd_elems)
    When a large number of storage devices are configured, you can increase this to improve performance.
    if queue_depth=16  and we have 10 disks, num_cmd_element should be  minimum 160

-----------------------

num_io_queues
(https://www.ibm.com/developerworks/aix/library/au-aix-performance-improvements-fc-fcoe-trs/index.html)

The FC driver stack includes a SCSI protocol driver and an adapter driver. The adapter driver maintains a SCSI I/O queue for I/O requests to the adapter. Due to I/O serialization through a single I/O queue scaling issues have been observed with extreme I/O transactions per second (IOPS) and small I/O sizes. To improve the number of IOPS on smaller block sizes, the multiqueue functionality is introduced with the 16 Gb FC starting from AIX 6.1 TL9 SP8, 7.2 TL1 and VIOS 2.2.4.30.

To support multiple I/O queues, the num_io_queues should be tuned. This indicates the number of I/O queues configured in the adapter driver (each I/O queue is associated with a hardware work queue in the HBA). All I/O requests issued to a particular hdisk will be mapped to the same SCSI I/O queue. Each SCSI I/O queue can service multiple hdisks, however I/O request to a given hdisk cannot be distributed to multiple SCSI I/O queues. The default value is 1 and the maximum is 16.

The HBA's direct memory access (DMA) should be also sufficient to distribute I/O requests across multiple queues. The ODM attribute, io_dma, controls the amount of I/O DMA region for the adapter. The default value of the io_dma is 64 and this should be increased to 256. In case only the num_io_queues are changed without increasing the io_dma value, the adapter instance will be configured with a single SCSI I/O queue and "29FA8C20 Additional FC Adapter Information error" will be logged in errpt.

# lsattr -El fcs0 | grep -e num_io_queues -e io_dma
io_dma          64        IO_DMA                                     True
num_io_queues   1         Desired number of IO queues                True

Steps to change num_io_queues and io_dma for a 16 Gb FC HBA:
# rmdev -Rl fcs0                        <--unconfigure the fcs device
# chdev -l fcs0 -a io_dma=256           <--changing io_dma to 256
# chdev -l fcs0 -a num_io_queues=16     <--changing num_io_queues to 16 (it can be other value as well)
# cfgmgr -l fcs0                        <--configure back fcs device

# lsattr -El fcs0 | grep -e num_io_queues -e io_dma
io_dma          256          IO_DMA                                     True
num_io_queues   16           Desired number of IO queues                True

With this change he number of IOPS for random read operations with a block size of 4KB (for a single FC HBA port) increased by approximately 2.5 times. The achieved IOPS count of 390,000 is very close to the line speed for a single FC HBA port. The IOPS gain for random read operations with a block size of 4 KB in the NPIV configuration is almost equivalent when the number of clients is six or more.

-----------------------

WWPN for an FC Adapter:

It can be obtained with "lscfg":
# lscfg -vpl fcsX | grep Netw
        Network Address.............21000025FF8338A0

Storage Admins likes in this format:
(with colons ':' and with small letters)

# lscfg -vpl fcsX | grep Netw | cut -c 37-54 | tr '[A-Z]' '[a-z]' | sed 's/../&:/g;s/:$//'

21:00:00:25:ff:83:38:a0

-----------------------

HOT PLUG adapter replacement:
0. lsslot -c pci  (check if the adapter hot pluggable)

1. paths are better in offline (C) state: dlnkmgr offline -hba 01.01

-------
it can be done with "smitty mpio" as well:
> MPIO Management > MPIO Path Management > Remove Paths  > Disable Paths under a Parent
-------

2. rmdev -Rl fcs1 <--(-R: unconfigures the device and its children, -l: specifies the logical device)

3. adapter replacement:
diag --> task selection  --> hot plug task ---> PCI Hot Plug Manager ---> Replace/Remove...

After starting it, adapter will blink.
During replacement we should leave this menu point in open,after replacement we can close this menu point in diag (enter)
(If this menu option is in open state, it will lock the adapter and configuring it is not possible, so should be closed.)

4. cfgmgr, the new device will come up with new WWN (lscfg -vpl fcsX, lsattr -El fscsiX)

5. mail to storage team, for mapping the storage to the new WWN

6. Another cfgmgr will bring up the new disks

7. put adapter to online state: dlnkmgr online -hba 01.01

------------------------

Adapter replacement with SDD dynamically (hot plug):

datapath query adapter                        <--identify the adapter to be replaced
datapath remove adapter X                     <--X is the adapter number to be removed ("select counts" shows I/O still running on other adapt.)
rmdev -dl fcsX -R                             <--remove fcs0 and all of its child devices (lsdev -Cc disk, should not show any device)
diag --> task selection  --> hot plug task ---> PCI Hot Plug Manager ---> Replace/Remove...
(from command line: lscfg -vpl fcs0| grep Adapter -> drslot -R -c pci -s P1-I8 <--P1-I8 is the slot location)
go to the ESS specialist to update the WWN of the new adapter (the zone information of the fabric switch must be updated)
cfgmgr                                        <--check with lsdev -Cc adapter, lsdev -Cc disk
addpaths                                      <--configure the new adapter and its child devices to SDD

------------------------


Adapter parameter change with SDD:

lsattr -El fcsX, lsattr -El fscsiX            <--checking the needed settings (num_cmd_elements, fc_err_recov,...)
rmdev -dl dpo -R                              <--remove SDD vpath devices
rmdev -l fcsX -R                              <--put fc adapter and its children devices to Defined state
chdev -l fcsX -a num_cmd_elems=1024
chdev -l fscsiX -a fc_err_recov=fast_fail
cfgmgr -l fcsX                                <--configiure the fc adapter, children devices and hdisks
cfallvpath                                    <--configure SDD vpath devices (if they were removed)

------------------------

PCIDMA (DMA) error:

If there are too many FC adapters and too many LUNs attached to the adapter, this will lead to issues configuring the LUNs. Errors will look like:

LABEL: DMA_ERR
IDENTIFIER: 00530EA6

Resource Name: PCIDMA
Resource Class: NONE
Resource Type: NONE

So if you get these errors, you'll need to change the max_xfer_size back to the default value. Also note that if you are booting from SAN, if you encounter this error, you won't be able to boot, so be sure to have a back out plan if you plan to change this and are booting from SAN.

------------------------