dropdown menu

Adapter

Connector can be PCI:

-FC HBA - Fibre Channel Host Bus Adapter
    fcsx - hardware adapter,
    fscsix - fc-scsi protocol device (logical device) (scsi protocol over the fc protocol)
     
   fcstat fcsX                 fiber channel statistics report


-SCSI HBA - SCSI Host Bus Adapter

    sisscsiaX - SCSI ADAPTER (internal disks are connected to this)

    lsdev -C | grep sisscsia   list the scsi adapters
    lsdev -Cs scsi -H          lists the scsi devices with location code

----------------------------------------
A host bus adapter is a physical Fibre Channel adapter that can be assigned to a logical partition.
----------------------------------------
----------------------------------------

ADAPTER SETTINGS:

root@aix21: / # lsattr -El fscsi0
attach       switch       How this adapter is CONNECTED         False   <-- shows actual connection protocol (None,switch,al lehet)
dyntrk       no           Dynamic Tracking of FC Devices        True    <-- dynamically follows SAN changes
fc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True    <-- 3. delayed_fail or fast_fail
scsi_id      0x133300     Adapter SCSI ID                       False
sw_fc_class  3            FC Class for Fabric                   True

root@aix21: / # lsattr -El fcs0
bus_intr_lvl  305        Bus interrupt level                                False
bus_io_addr   0xffc00    Bus I/O address                                    False
bus_mem_addr  0xfffbf000 Bus memory address                                 False
init_link     al         INIT Link flags                                    True    <--al or pt2pt
intr_priority 3          Interrupt priority                                 False
lg_term_dma   0x800000   Long term DMA                                      True    <--deafault 0x200000, max. 0x8000000
max_xfer_size 0x100000   Maximum Transfer Size                              True    <--default is 100000, max 1000000
num_cmd_elems 200        Maximum number of COMMANDS to queue to the adapter True    <--default 200, max is 1024 or 2048
pref_alpa     0x1        Preferred AL_PA                                    True
sw_fc_class   2          FC Class for Fabric                                True

(Below descriptions are from "DS8800 Performance Monitoring and Tuning" Redbook.)

attach:(sets the Fiber Channel topology)
    none: if no cable has been plugged in
    al: all devices are in a loop or ring, similar to token ring networking. The failure of one device causes a break in the ring.
    switch: all devices or loops of devices are connected to Fibre Channel switches, similar conceptually to modern Ethernet implementations  

dyntrk:
    If it is on yes, it enables dynamic changing of fibre channel cable connections on switch ports or storage ports.
    We have changed few StoragePorts to another Switch last saturday ==> that means, that the HardwarePath changed for these servers,
    where the Option "dyntrk" is set to no. --> reconfigure your disks

fc_err_recov:
    It can enhance fast I/O Failure for FC devices after link events (like link lost) in a switched environment.
    delayed_fail: waits some time before acting.
    fast_fail is useful where multipathing software is used. This would support faster failover to alternate paths

init_link:
    from online help: "Do not change this attribute unless directed by IBM support."
    al: arbitrared loop : first tries al, then pt2pt, if not succes then link remains down
    pt2pt: tries pt2pt, if not succes then link remains down

lg_term_dma:
    it controls the DMA memory resource that an adapter driver can use (default value is 0x200000, maximum value is 0x8000000)
    One change is to increase the value of lg_term_dma to 0x400000 or higher.
    If you have a dual port adapter the maximum value is divided between the adapter ports.
    Using the maximum value for a dual-port adapter can cause the second port performance to fail

max_xfer_size:
    Controls the maximum I/O size the adapter device driver will handle and the memory area used by the adapter.
    Default value is max_xfer_size=0x100000, which means the memory area is 16 MB.
    2 I/O DMA settings are available 16MB and 256/128MB: lsattr -l fcs0 -a max_xfer_size -R
    0x100000    <= 16 M
    0x200000    <= 256 M or 128 M depending on PCI Bus Architecture
    0x400000    <= 256 M or 128 M depending on PCI Bus Architecture
    ...
    Sometimes changed to 0x200000 which increases the memory region size to 128MB and potentially can improve the adapter’s bandwidth.
    if DMA_ERR appears in errpt(which can casuse SAN booting issues too), then chaning back max_xfer_size to the default value can help
    Since AIX 6.1 the DMA memory area for virt. FC adapters is always 128 MB even with the default max_xfer_size
    (max_xfer_size is a DMA memory area, but it is different from the lg_term_dma (which is used for I/O control))

num_cmd_elems:
    Maximum number of commands to be queued to the adapter (range can be checked: lsattr -Rl fcs0 -a num_cmd_elems)
    When a large number of storage devices are configured, you can increase this to improve performance.
    if queue_depth=16  and we have 10 disks, num_cmd_element should be  minimum 160

-----------------------

num_io_queues
(https://www.ibm.com/developerworks/aix/library/au-aix-performance-improvements-fc-fcoe-trs/index.html)

The FC driver stack includes a SCSI protocol driver and an adapter driver. The adapter driver maintains a SCSI I/O queue for I/O requests to the adapter. Due to I/O serialization through a single I/O queue scaling issues have been observed with extreme I/O transactions per second (IOPS) and small I/O sizes. To improve the number of IOPS on smaller block sizes, the multiqueue functionality is introduced with the 16 Gb FC starting from AIX 6.1 TL9 SP8, 7.2 TL1 and VIOS 2.2.4.30.

To support multiple I/O queues, the num_io_queues should be tuned. This indicates the number of I/O queues configured in the adapter driver (each I/O queue is associated with a hardware work queue in the HBA). All I/O requests issued to a particular hdisk will be mapped to the same SCSI I/O queue. Each SCSI I/O queue can service multiple hdisks, however I/O request to a given hdisk cannot be distributed to multiple SCSI I/O queues. The default value is 1 and the maximum is 16.


The HBA's direct memory access (DMA) should be also sufficient to distribute I/O requests across multiple queues. The ODM attribute, io_dma, controls the amount of I/O DMA region for the adapter. The default value of the io_dma is 64 and this should be increased to 256. In case only the num_io_queues are changed without increasing the io_dma value, the adapter instance will be configured with a single SCSI I/O queue and "29FA8C20 Additional FC Adapter Information error" will be logged in errpt.

# lsattr -El fcs0 | grep -e num_io_queues -e io_dma
io_dma          64        IO_DMA                                     True
num_io_queues   1         Desired number of IO queues                True

Steps to change num_io_queues and io_dma for a 16 Gb FC HBA:
# rmdev -Rl fcs0                        <--unconfigure the fcs device
# chdev -l fcs0 -a io_dma=256           <--changing io_dma to 256
# chdev -l fcs0 -a num_io_queues=16     <--changing num_io_queues to 16 (it can be other value as well)
# cfgmgr -l fcs0                        <--configure back fcs device

# lsattr -El fcs0 | grep -e num_io_queues -e io_dma
io_dma          256          IO_DMA                                     True
num_io_queues   16           Desired number of IO queues                True

With this change he number of IOPS for random read operations with a block size of 4KB (for a single FC HBA port) increased by approximately 2.5 times. The achieved IOPS count of 390,000 is very close to the line speed for a single FC HBA port. The IOPS gain for random read operations with a block size of 4 KB in the NPIV configuration is almost equivalent when the number of clients is six or more.

-----------------------

WWPN for an FC Adapter:

It can be obtained with "lscfg":
# lscfg -vpl fcsX | grep Netw
        Network Address.............21000025FF8338A0


Storage Admins likes in this format:
(with colons ':' and with small letters)
# lscfg -vpl fcsX | grep Netw | cut -c 37-54 | tr '[A-Z]' '[a-z]' | sed 's/../&:/g;s/:$//'
21:00:00:25:ff:83:38:a0

-----------------------

HOT PLUG adapter replacement:
0. lsslot -c pci  (check if the adapter hot pluggable)

1. paths are better in offline (C) state: dlnkmgr offline -hba 01.01

-------
it can be done with "smitty mpio" as well:
> MPIO Management > MPIO Path Management > Remove Paths  > Disable Paths under a Parent
-------

2. rmdev -Rl fcs1 <--(-R: unconfigures the device and its children, -l: specifies the logical device)

3. adapter replacement:
diag --> task selection  --> hot plug task ---> PCI Hot Plug Manager ---> Replace/Remove...

After starting it, adapter will blink.
During replacement we should leave this menu point in open,after replacement we can close this menu point in diag (enter)
(If this menu option is in open state, it will lock the adapter and configuring it is not possible, so should be closed.)

4. cfgmgr, the new device will come up with new WWN (lscfg -vpl fcsX, lsattr -El fscsiX)

5. mail to storage team, for mapping the storage to the new WWN

6. Another cfgmgr will bring up the new disks

7. put adapter to online state: dlnkmgr online -hba 01.01

------------------------

Adapter replacement with SDD dynamically (hot plug):

datapath query adapter                        <--identify the adapter to be replaced
datapath remove adapter X                     <--X is the adapter number to be removed ("select counts" shows I/O still running on other adapt.)
rmdev -dl fcsX -R                             <--remove fcs0 and all of its child devices (lsdev -Cc disk, should not show any device)
diag --> task selection  --> hot plug task ---> PCI Hot Plug Manager ---> Replace/Remove...
(from command line: lscfg -vpl fcs0| grep Adapter -> drslot -R -c pci -s P1-I8 <--P1-I8 is the slot location)
go to the ESS specialist to update the WWN of the new adapter (the zone information of the fabric switch must be updated)
cfgmgr                                        <--check with lsdev -Cc adapter, lsdev -Cc disk
addpaths                                      <--configure the new adapter and its child devices to SDD

------------------------


Adapter parameter change with SDD:

lsattr -El fcsX, lsattr -El fscsiX            <--checking the needed settings (num_cmd_elements, fc_err_recov,...)
rmdev -dl dpo -R                              <--remove SDD vpath devices
rmdev -l fcsX -R                              <--put fc adapter and its children devices to Defined state
chdev -l fcsX -a num_cmd_elems=1024
chdev -l fscsiX -a fc_err_recov=fast_fail
cfgmgr -l fcsX                                <--configiure the fc adapter, children devices and hdisks
cfallvpath                                    <--configure SDD vpath devices (if they were removed)

------------------------

PCIDMA (DMA) error:

If there are too many FC adapters and too many LUNs attached to the adapter, this will lead to issues configuring the LUNs. Errors will look like:

LABEL: DMA_ERR
IDENTIFIER: 00530EA6

Resource Name: PCIDMA
Resource Class: NONE
Resource Type: NONE

So if you get these errors, you'll need to change the max_xfer_size back to the default value. Also note that if you are booting from SAN, if you encounter this error, you won't be able to boot, so be sure to have a back out plan if you plan to change this and are booting from SAN.

------------------------

40 comments:

  1. This is a terrific page! Very helpful, thank you!

    ReplyDelete
  2. Well done. Thanks

    ReplyDelete
  3. very good link please continue the with latest updates

    ReplyDelete
  4. OK...and I'm happy I could help you :)

    ReplyDelete
  5. for a newbie AIX system admin, your blog is such a relief, helping specially in Sev1s.. :)

    ReplyDelete
  6. Hey there! Someone in my Myspace group shared this site with us
    so I came to check it out. I'm definitely enjoying the information. I'm book-marking and will be tweeting this to my followers! Exceptional blog and fantastic style and design.
    My page: Central London

    ReplyDelete
    Replies
    1. Thanks! I am appreciating your contribution to this blog!
      (When I travel to London and I need a personal trainer, I know where to go :))

      Delete
    2. Very good explantion. Thanks for the post.

      Delete
  7. AWESOME !

    This site itself is having bunch of knowledge !!!

    Thanks a LOT for preserving/upating this.

    bookmarked this on ALL browsers I'm using :)

    Santosh Bhalerao

    ReplyDelete
  8. This is really very helpful.. Thanks a lot.

    ReplyDelete
  9. what is the fiber port replacement process in aix p740 server??

    ReplyDelete
    Replies
    1. it is written above at adapter replacement section...

      Delete
  10. Hi,

    How i can change init_link if i can see "false" at the end of the line ?

    Many thanks

    ReplyDelete
    Replies
    1. Hi, if it is false, it means user cannot change that line. I would get in contact with IBM in this case.

      Delete
  11. Hi
    How to check the Multipath status?how to check whether multipath software is installed or not?Please explain me some EMC powerpath important commands.if the hdisk is coming from SAN,How to check its path,Multipath status,Lun ID.

    Sorry for asking so many Q;)

    ReplyDelete
    Replies
    1. Description about EMC can be found here: http://aix4admins.blogspot.hu/2011/05/emc-emc-comes-with-default-settings.html

      Delete
  12. Nice post! Didn't know about the lsslot. Really love the detail here

    I didn't see "lsdev -Cc adapter" explicitly mentioned above.. Just to add, I like this just for seeing on the spot if I have a dual port or single port

    e.g.
    bash-3.2# lsdev -Cc adapter
    fcs0 Available 03-00 4Gb FC PCI Express Adapter (df1000fe)
    fcs1 Available 03-01 4Gb FC PCI Express Adapter (df1000fe)
    fcs2 Available 04-00 4Gb FC PCI Express Adapter (df1000fe)
    fcs3 Available 04-01 4Gb FC PCI Express Adapter (df1000fe)
    fcs4 Available 06-00 4Gb FC PCI Express Adapter (df1000fe)
    fcs5 Available 06-01 4Gb FC PCI Express Adapter (df1000fe)
    bash-3.2#

    I have three dual ports

    Just wanted to add!

    ReplyDelete
  13. So, This site very nice ! Thanks

    ReplyDelete
    Replies
    1. Can you answer me about Adapter in AIX? what's the difference between PCI, IDE and SCSI in AIX? I have search on the internet but I can't understand?

      Please .. ! Thanks

      Delete
  14. You saved my life :)

    ReplyDelete
  15. could you please help on one of my common problem is. scenario is: my /tmp is 2% now. but after one minute i get alert /tmp is 100% full. but when i see top usage files. I donot find any big files.. then i use #fuser -kuxc /tmp and sudden i find that again /tmp is 2 %.. please suggest me . how can i find a process which reserves all the /tmp space.

    ReplyDelete
  16. Hi, thanks for the very useful link.
    Have a question:
    There are options we can use for fcstat (-d; -e; -z etc).

    http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

    However, the fcstat -d did not work.
    Anyone can shred a light?

    ReplyDelete
  17. Hi ... I think the answer is yes to these 2 questions that I have but I need to verify please. Would appreciate.

    1) Can I reboot my HMC without interuptions to my attached host?
    2) Can I upgrade my HMC without interruptions to my attached host?

    Thank you.

    ReplyDelete
    Replies
    1. Yes, you can do both of them without interruption in the service on the running LPARs.

      Delete
  18. I really self learned AIX stuff with your blogs .. Its awesome work by you ....!

    ReplyDelete
  19. I replaced a pci card in storage N6210 wich connect to IBM server now I want to change the WWN in server which running with OS AIX 1.0 plz any one tell me the command for change WWN no

    ReplyDelete
  20. Hi
    fcs adapter is reporting continuous "Link errors". Adapter was not connected to any switch, Is there a way we can stop these errors

    ReplyDelete
    Replies
    1. Hi, probably you could try to put it in defined state or remove cables if it is not needed.

      Delete
  21. Hi ,
    In case that I need replace FC cable during suspicion to some problem . Do you have some procedure to do it ?
    Is fcsX must be switched to offline ? What is the best practice to do it ?
    Thanks in advance .

    ReplyDelete
    Replies
    1. Hi, if you have multiple paths to the storage (multiple cables), then plugging out one cable will cause that this path will be OFFLINE or DEGRADED, but other paths should work perfectly, so your LUNs are still available (please prepare for some errpt entries.) You can also deconfigure (disable) one path so when you plug out the cable AIX will not notice anything. Hope it helps.

      Delete
    2. Hi !
      Thanks for quick responce !
      Now all path works , but because ot the issue happened twice ,
      this time we decided to replace FC cables .
      What is the way to "deconfigure (disable) one path" and how to restore it after replacing FC cable .
      Here is the errors from lats time issue happened .
      Thanks in advance .

      Mar 30 07:56:25 fscsi2    T FCP_ERR6           Relogin 10000000C9EF**** to port 0x1FF340 rejected; EIO
      Mar 30 07:56:25 fscsi0    T FCP_ERR6           Relogin 10000000C9EF**** to port 0x0BFF00 rejected; EIO

      Delete
    3. Hi, it depends on the storage you use. (EMC, Hitachi has different commands etc..) I recommend on a test server do some tests before touching the production server.

      Delete
  22. Hi.. please advise what are the possibilities to fail wrap plug test on fcs adapter



    ReplyDelete
    Replies
    1. Hi, maybe this?
      https://www.unix.com/aix/99780-wrap-plug-test.html

      Delete