PERF. - fcstat

fcstat

The fcstat command reports statistics directly from the FC adapter firmware and the FC driver. Protocols such as TCP/IP are designed to tolerate packet loss and out-of-order packets with minimal disruption, but the FC protocol is in-tolerant of missing, damaged or out-of-order frames and is incapable of re-transmitting a single missing frame.

This moves error recovery into the SCSI layer and can result in waiting for commands to timeout. In some cases an error frame is not detected by either the target or the initiator, so it just waits for completion until 30 or 60 seconds to timeout. These are often the result of a physical layer problems such as a damaged fibre channel cable, faulty or degraded laser in SFP’s (in a storage controller, switch or host) or perhaps a failing a ASIC in a switch or a slow draining device causing frames to be discarded. Regardless of the cause, identifying and resolving fibre channel transport related problems are necessary before any I/O performance tuning is attempted.

It is also important to ensure the SCSI layer does not overwhelm the Target Ports or LUNs with excessive I/O requests. Increasing num_cmd_elems may result in driving more I/O to a storage device resulting in even worse I/O service times. (errpt, and iostat can help uncover some of these problems.) However acceptable I/O service time can differ.  For example, some shops demand less than 2 ms service times where others may tolerate 11 ms. The disk technology affects expected I/O service time, as does the availability of write and/or read cache.

If queuing in the disk driver is occurring, (iostat shows non-zero value in qfull) this should be resolved first like increasing queue_depth, or adding additional storage resources (if io service times are too high).  After ensuring there are no fibre channel physical layer problems, average I/O response times are in good range (not exceeding 15 ms) and there is no queuing (qfull) in the disk driver, then we can tune the adapter.

-----------------------------------

In normal way fcstat resets statistics when server is rebooted or the fcs device is reconfigured. fcstat -Z fcsX can be useful for daily monitoring because it resets statistics.

fcstat fcsX        shows fc adapter statistics
fcstat -D fcsX     shows additional fcs related details
fcstat -e fcsX     shows all stats, which includes the device-specific statistics (driver statistics, link statistics, and FC4 types)
fcststat -Z fcsx   resets statistics

-----------------------------------

root@aix1:/ # fcstat fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
(adapter/pciex/df1000f114108a0)
Serial Number: 1C041083F7
Option ROM Version: 02781174
ZA: U2D1.11X4                                      <--firmware version
World Wide Node Name: 0x20000000C9A8C4A6           <--adapter WWN
World Wide Port Name: 0x10000000C9A8C4A6           <--adapter WWPN
FC-4 TYPES:
 Supported: 0x00000120000000000000000000000000000000000000
 Active:    0x00000100000000000000000000000000000000000000
Class of Service: 3
Port Speed (supported): 8 GBIT                     <--8Gb adapter
Port Speed (running):   8 GBIT                     <--running at 8Gb
Port FC ID: 0x6df640                               <--adapter FC ID (first 2 digits after x will show switch id, here 6d)
Port Type: Fabric                                  <--connected in Fabric
Attention Type: Link Up                            <--link status

Seconds Since Last Reset: 270300                   <--adapter is collecting stats since this amount seconds

        Transmit Statistics     Receive Statistics
        -------------------     ------------------
Frames: 2503792149              704083655
Words:  104864195328            437384431872

LIP Count: 0
NOS Count: 0
Error Frames: 0                                    <--affects io when frames are damaged or discarded
Dumped Frames: 0                                   <--affects io when frames are damaged or discarded
Link Failure Count: 0
Loss of Sync Count: 8
Loss of Signal: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 31                          <--fast increase may result in buffer to buffer credit problems, damaged FC frames, discards
Invalid CRC Count: 0                               <--affects io when frames are damaged or discarded                                               

...
Elastic buffer overrun count: 0                    <--may occur with link failures

IP over FC Adapter Driver Information
  No DMA Resource Count: 3207
  No Adapter Elements Count: 126345

FC SCSI Adapter Driver Information
  No DMA Resource Count: 3207                      <--IOs queued at the adapter due to lack of DMA resources (increase max_xfer_size)
  No Adapter Elements Count: 126345                <--IO was temporarily blocked/queued (increase num_cmd_elems)
  No Command Resource Count: 133                   <--there was no free cmd_elems (increase num_cmd_elems)

IP over FC Traffic Statistics
  Input Requests:   0
  Output Requests:  0
  Control Requests: 0
  Input Bytes:  0
  Output Bytes: 0

FC SCSI Traffic Statistics
  Input Requests:   6777091279
  Output Requests:  2337796
  Control Requests: 116362
  Input Bytes:  57919837230920
  Output Bytes: 39340971008

Adapter Effective max transfer value:  0x100000   <--value set in the kernel regardless of ODM (must be equal or greater than hdisk max_coalesce)

-----------------------------------

Port FC ID
We can get some information about the switch in hexa. Here 0x6df640, which are six hexa digits:
1st 2 digits after x: domain id of the SAN switch, we can call it "switch id" (here 6d)
2nd 2 digits after x: port ID (but could be some virtualized interpretation as well, here f6),
3rd 2 digits after x: loop id if in loop mode (00)

Checking "switch id", will show if ports of an FC adapter are connected to different fabrics (switches) or not. Keep in mind, that there may be more switches in a Fabric, so multiple "switch ids" are not guarantee for multiple Fabrics.

If we check a 4-port adapter, and if the first 2 hexa digits are the same, we can say that we are connected to the same switch.
fcs0: Port FC ID: 0xd1e6c0                  <--Fabric 1 (switch id: d1)
fcs1: Port FC ID: 0xd1e7c0                  <--Fabric 1 (switch id: d1)
fcs2: Port FC ID: 0x6de6c0                  <--Fabric 2 (switch id: 6d)
fcs3: Port FC ID: 0x6de7c0                  <--Fabric 2 (switch id: 6d)


Error frames, Dumped frames, Invalid CRC count:
These may be the result of a physical transport layer problem which may result in damaged fiber channel frames as they arrive at the adapter. These are usually not incrementing on frames being transmitted but rather frames received.

For each CRC errors, AIX will log an errpt entry indicating a damaged frame. CRC errors can occur anywhere in the fabric and are usually related to a bad sfp or bad FC cable. These errors will affect I/O processing for a single read or write operation but the driver will retry these. These are the most difficult to troubleshoot.


Link Failure Count, Loss of Sync Count, Loss of Signal:
It indicates the health of the physical link between the switch and the host HBA. If these error counters increase daily we generally suspect a problem with an sfp or FC cable between the switch and the FC HBA. These can affect I/O processing on the host.


Invalid Tx Word Count:
These are incremented when the HBA receives damaged words from the switch. In many cases this will not affect I/O processing but is an early indication of a problem. On certain switch models this may be due to an improper port fill word setting. If not, this may indicate a bad sfp or cable between the HBA and the switch. This error counter is only relevant for communications at the physical layer / Tx / Rx between the switch and the HBA.


Elastic buffer overrun count:
This counter could increment due to Link Failure Count, Loss of Sync Count, Loss of Signal, Invalid Tx Word Count or old unsupported host HBA adapter firmware levels.

-----------------------------------

No DMA Resource Count:
It means additional I/O DMA memory is needed to initiate (larger) I/O’s from the adapter. When the adapter driver is unable to initiate an I/O request due to no free DMA resource, the "No DMA Resource" counter is incremented and the I/O request waits. Increasing max_xfer_size can help in this situation.

No Adapter Elements Count:
number of times since boot, an IO was temporarily blocked due to an inadequate num_cmd_elems. If it shows non-zero values increaseing num_cmd_elems can help.

No Command Resource Count:
When the adapter driver is unable to initiate an I/O request due to no free cmd_elems (num_cmd_elems), the "No Command Resource" counter is incremented and the I/O request waits for adapter buffer resources (checking for free command elements for the adapter). Resources will be available when a currently running I/O request is completed. Increasing num_cmd_elems can help to avoid this situation.

If the "No Command Resource Count" and/or the "No DMA Resource Count"  continues to increment, (and the max_xfer_size and num_cmd_elems are set to maximum values), then the adapter I/O workload capability has been exceeded. In this case I/O load should be reduced by moving load to additional resources, like adding additional FC adapters and balancing the I/O work load. Another workaround would be to reduce the num_cmd_elems.

-----------------------------------

fcstat -D fcsX  can display additional info:
(Values preceded by a 0x are in hex. All values below are reported in hex, not decimal.)

Driver Statistics:
  Number of interrupts:   76534
  Number of spurious interrupts:    0
  Long term DMA pool size:    0x800000
  I/O DMA pool size:  0x1000000                        <--currently active I/O DMA pool size in the driver

  FC SCSI Adapter Driver Queue Statistics              <--adapter driver
    Number of active commands:   0
    High water mark of active commands:    11
    Number of pending commands:  0
    High water mark of pending commands:   1
    Number of commands in the Adapter Driver Held off queue:  0
    High water mark of number of commands in the Adapter Driver Held off queue:   0

  FC SCSI Protocol Driver Queue Statistics             <--protocol driver
    Number of active commands:   0
    High water mark of active commands:    11
    Number of pending commands:  4
    High water mark of pending commands:   5


Number of active commands:
Represents the I/O workload”.  Active commands are commands that have left the adapter driver and have been handed off to the adapter hardware for transport to the end device. These commands have not received a completion status and are considered active.

High watermark of active commands:
The "high water mark of active commands" represents the peak (highest) number of active commands. If I/O service times are low and if the high water mark of active commands is around the num_cmd_elems then increasing the num_cmd_elems may improve I/O performance. In certain error recovery scnerios the "high water mark of active commands" could increase up to the num_cmd_elems limit. When tuning, clear these counters and monitor them for few days, that there are no errors.

High watermark of pending commands:
The "high water mark of pending commands" represents the peak (highest) number of pending commands. (These are pending because the number of active commands reached the num_cmd_limits and the additional commands above that limit are pending.)

If high water mark for active + pending is near to or is exceeding the num_cmd_elems, we recommend increasing num_cmd_elems to cover this water mark to improve the IO performance. Rule to follow: num_cmd_elems > (High water mark for active commands + High water mark for pending commands)

The increase for num_cmd_elems is always recommended to be done gradually until 'No Command Resource Count' counter stops increasing.

If with large sequenial IOs (like backups), there are high avg read and write service timees  and number of active/peak commands are also high (but there are no physical layer problems, no queuing in the adapter and disk) then the storage server is unable to service these I/O requests in a timely manner or the I/O load is greater than the LUN / storage controller capability (like handling within a ~15ms window). Solution could be adding additional storage resources, like distributing the I/O work load to additional LUNs and/or storage controllers

-----------------------------------

Link to some IBM desctiptions: https://www.ibm.com/support/pages/node/6198385

-----------------------------------

Adabter busy %

There are no busy% for adapters in AIX. They are derived from the disk stats. The adapter busy% is simply the sum of the disk busy%.
So if the adapter busy% is, for example, 350% then you have 3.5 disks busy on that adapter. Or it could be 7 disks at 50% busy or 14 disks at 25% or ....

There is no way to determine the adapter busy and in fact it is not clear what it would really mean. The adapter has a dedicated on-board CPU that is always busy (probably no real OS) and we don't run nmon of these adapter CPUs to find out what they are really doing.

-----------------------------------

PERF. - NETWORK VIRTUALIZATION

Network - SEA, Virtualization


viostat            monitors storage performance
seastat            statistics about SEA (first this should be enabled for the SEA: chdev -dev ent10 -attr accounting=enabled)
                   then: seastat -d ent10

----------------------

topas -E:

#  ifconfig <SEA/en8> up            <--SEA interface must be in up state

# topas -E                          <--topas -E will show detailed info
Topas Monitor for host:                Interval:
===============================================================================
Network                                KBPS   I-Pack   O-Pack    KB-In   KB-Out
Topas Monitor for host:        aix10-vios1erval:   2   Thu Nov 15 10:51:27 2012
===============================================================================
Network                                KBPS   I-Pack   O-Pack    KB-In   KB-Out
ent8 (SEA)                              2.3     13.5      6.0      1.5      0.9
  |\--ent3 (VETH CTRL)                  2.4      3.5      3.5      1.2      1.2
  |\--ent2 (EC PHYS)                    1.3      9.0      1.5      0.9      0.4
  |  |\--ent1 (PRIM)                    0.9      4.5      1.5      0.5      0.4
  |   \--ent0 (PRIM)                    0.4      4.5      0.0      0.4      0.0
  |\--ent4 (VETH)                       0.6      1.0      4.5      0.1      0.5
   \--ent5 (VETH)                       0.4      3.5      0.0      0.4      0.0
en6                                     0.0      0.0      0.0      0.0      0.0
lo0                                     0.0      0.0      0.0      0.0      0.0

----------------------

Important details about Virt. Ethernet Adapters:

- Virtual Ethernet by default cannot provide 10Gb. A quote from VIO Wiki: "The transmission speed of Virtual Ethernet adapters is in the range of 1-3 Gigabits per second, depending on the transmission (MTU) size. " It supports MTU sizes: 1500, 9000 and additionally 65280. A SEA (on top of 2x10Gb LACP Eth. Chan.) with 1 Virt. Trunk Adapter (with default settings) will not to provide 20Gb/s bandwidth.

- On a Power 8 server with Jumbo Frames, I could reach on a SEA with 1 Virt. Adapter max. 13 Gb/s. (Without tuning, network speed was 2-3 Gb/s). If you want to use 20Gb/s  (Eth. Channel with 2x10Gb LACP) you need at least 2 active Virt. Adapter in a SEA. (In SEA load-sharing mode at least 4 trunk adapters, 2-2 will be active on each VIO.)

- When using Virt. Ethernet and SEA, server must have enough free CPU (with 10Gb it will use extra CPU)
(Virtual Ethernet traffic is generally heavier than virtual SCSI traffic and Virtual Ethernet connections take up more CPU cycles than Physical Ethernet adapters. The reason is that modern Physical Ethernet adapters are taking over some work from system CPU for example: checksum computation and verification, packet reassembly...)

----------------------

Network Tuning:
(These values are recommended by Gareth Coats)

- SEA returns the maximum aggregate bandwidth when two Virtual network adapters are configured to it rather than one. It is not possible to achieve line speed of 10Gbit Ethernet through
an SEA with a single attached Virtual Network – it is possible with two virtual network adapters.
- An individual process communicating to a remote process will see an average of 450MB/sec bandwidth through VIO servers using 10Gbit Ethernet
- For an LPAR to achieve the maximum throughput over virtual networks the LPAR must communicate with 2 (or more) virtual networks and use more than one process on each virtual network.

Using the correct configuration options is essential, in each LPAR:
- enable largesend
- use the maximum available mtu size (64k)
- ensure each 10Gbit Ethernet adapter has largesend, flow_control, large_receive and jumbo_frames enabled
- Etherchannel adapters should use 8023ad mode, src_dst_port hash mode and have jumbo frames enabled
- SEA adapter itself needs to have largesend, large_receive and jumbo_frames enabled

----------------------

Jumbo frames:

With Jumbo frames turned on, each device (which is affected during network communication) has to be able to handle increased size packets. (Netw. Switch, Phys. Adapters, Eth. Chan., client LPAR). You cannot mix MTU sizes within a network, unless you have a suitable router which can fragment large packets to smaller MTU size if needed (consult with network team)

Virtual Ethernet adapters supports MTU of 64K as well, which can have a huge benefit with 10Gbit adapters. These packets will automatically be divided into the physical MTU before leaving the hardware. (VIO layer will handle the conversion to and from the MTU of 9000.)

The MTU size you can use can depend on the device you are using, so if 65280 does not work try 65536 (the full 64K), 65394 (64K minus overhead), 65390 (64K minus VLAN overhead) or you can try usual jumbo frame MTU=9000 .

When mixed MTU size (1500 and 9000) traffic goes through on a Virt. adapter in a SEA, bandwidth will be dropped down to MTU 1500. (So make sure LPARs which are using the same Virt. Adapter in a SEA have same mtu size setting.)

----------------------

Flow Control:

With 10 Gbit Ethernet it is very useful to turn on flow control to stop the need for re-transmission. Flow control prevents re-transmissions (and the resulting time outs, delays) by stopping transmissions at source when any buffer on the path approaches overflow. (Otherwise it is very easy at high bandwidth to completely fill buffers on switches and adapters so that transmitted packets are dropped. )

Check 10G adapters and ask network team about this feature at switch side.

$ lsdev -dev ent4 -attr | grep flow
flow_ctrl       yes               Request flow control

----------------------

Large Send, Large Receive:

If these are turned on (large send and large receive) TCP stack can build a message up to 64 KB and send it in one call. (with 1500 bytes that would take 44 calls.)
Turning these on can increase network throughput massively
(One test showed 1Gb/s without largesend, 3.8Gb/s with largesend,  much lower CPU in sender LPAR and in sending VIO, all with MTU at 1500, without  jumbo frames.)

Physical adapters should have by deafult large_send (and large_receive) on:
#lsattr -El ent0 |grep large
large_receive yes              Enable receive TCP segment aggregation
large_send    yes              Enable hardware Transmit TCP segmentation

For SEA  large_receive is turned off by default, so it disables this feature at underlying physical adapters.
(To use this feature, it should be turned on at SEA as well):
# lsattr -El ent11 | grep large
large_receive yes        Enable receive TCP segment aggregation
largesend     1         Enable Hardware Transmit TCP Resegmentation

At AIX LPAR side, largesend  is called mtu_bypass:
# lsattr -El en0 | grep mtu_bypass
mtu_bypass    on              Enable/Disable largesend for virtual Ethernet

Some cautions:
- large send and large receive provide benefits only when using large packets (>1500 byte).
- as large receive aggregates multiple incoming packets, this might have negative impact on latency. (higher appl. latency, more re-transmissions, or packet drops as an extreme example)
(Normally large receive works fine in mixed workload environments with moderate network demands. )
-a receiver with large receive enabled can for example communicate with a sender where large send is disabled, but on a lower throughput and with more cpu overhead on the sender side

----------------------

ISNO (Interface Specific Network Options)

Some parameters (tcp_sendspace, tcp_recvspace..) have been added for each network interface and are only effective for TCP (and not UDP) connections.  AIX sets default values for these, for both MTU 1500 and for jumbo frame mode (MTU 9000), which provides good performance. Values set manually for an individual interface take precedence over the systemwide values set with the no command.

Default values set by AIX:

# ifconfig -a
en8: flags=1e080863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 112.186.59.8 netmask 0xffffff00 broadcast 162.86.159.255
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1


Default TCP settings are usually sufficient, but if the TCP send, receive and/or rfc1323 is set, they should be changed to match the above table, unless the settings on the adapter are larger.

If you are setting the tcp_recvspace value to greater than 65536, set the rfc1323 value to 1 on each side of the connection. If you do not set the rfc1323 value on both sides of the connection, the effective value for the tcp_recvspace tunable will be 65536.

----------------------


No Resource Error:

For high speed traffic necessary resources should be available. An example for these resources are the buffer spaces with differet size: tiny, small, medium... If these buffer spaces are not ready during network traffic, we can get "No resource Error", and other errors as well (Packets Dropped, Hypervisor Receive Failure).

# entstat -d ent1

ETHERNET STATISTICS (ent1) :                                          
Device Type: Virtual I/O Ethernet Adapter (l-lan)                    
Hardware Address: 41:ca:14:e7:26:9b                                  
Elapsed Time: 12 days 2 hours 3 minutes 31 seconds                  
                                                                     
Transmit Statistics:       Receive Statistics:                        
--------------------       -------------------                        
Packets: 5912589961        Packets: 26139812411                      
Bytes: 712365989202        Bytes: 712351516630458                    
Interrupts: 0              Interrupts: 6812561727                    
Transmit Errors: 0         Receive Errors: 0                          
Packets Dropped: 0         Packets Dropped: 81212309      <--attention needed      
...                                                                                            
Max Collision Errors: 0    No Resource Errors: 16113801   <--attention needed
...                                                                      
Hypervisor Send Failures: 0                                          
  Receiver Failures: 0                                                
  Send Errors: 0                                                      
Hypervisor Receive Failures: 16113801                      <--attention needed


For Virtual Ethernet adapers the above errors can be caused by incorrect buffer allocation:
Min Buffers: this is the number of buffers initially provided by the server (number of pre-allocated buffers)
Max Buffers: the absolute maximum (upper limit) of allocated buffers
Max Allocated: in the past what was the highest number of allocated buffers

The buffer allocation history can checked at the end of entstat output:

...
Receive Information                                                  
  Receive Buffers                                                    
    Buffer Type              Tiny    Small   Medium    Large     Huge
    Min Buffers               512      512      128       24       24
    Max Buffers              2048     2048      256       64       64
    Allocated                 513      535      148       28       64
    Registered                512      510      127       24       13
                                                                     
    History                                                          
      Max Allocated           576      951      133       64       64   <--all of them is above Min, additionally Large and Huge reached Max
      Lowest Registered       502      502       64       12       11
...

Initially server provides the value of Min Buffers (pre-allocated) and if later network traffic demands more, a so called "post-allocation" will occur.
This action takes time and it can negatively affect response time (especially at high speed workloads.)  After this Max Allocated will show a new higher value.
If Max Allocated reaches the value of Max Buffers it is a hint for bottleneck in latency and throughput. (It probably means network traffic would need more buffers, but server cannot provide more as it cannot go above the value of Max Buffers.)

In short:
- it is not optimal if Max Allocated is above of Min Buffers
- it is even worse if Max Allocated reached the value of Max Buffers

Situation ca be solved by tuning the buffers on all bridging Virt. Adapter configured for SEA.

Commands to change these:
chdev -l entX -a max_buf_huge=128 -P
chdev -l entX-a min_buf_huge=64 -P
chdev -l entX -a max_buf_large=128 -P
chdev -l entX -a min_buf_large=64 -P
chdev -l entX-a max_buf_medium=512 -P
chdev -l entX-a min_buf_medium=256 -P
chdev -l entX-a max_buf_small=4096 -P
chdev -l entX-a min_buf_small=2048 -P
chdev -l entX-a max_buf_tiny=4096 -P
chdev -l entX-a min_buf_tiny=2048 -P


----------------------

Some additional considerations:

-Data Cache Block Flush (dcbflush): This allows the virtual Ethernet device driver to flush the processor’s data cache of any data after it has been received. It increases CPU utilization but also increases throughput: # chdev -l entX -a dcbflush_local=yes –P (need a reboot to take effect)

-Dog thread (Thread): By enabling the dog threads feature, the driver queues the incoming packet to the thread and the thread handles calling IP, TCP, and the socket code. Enable this parameter as your LPAR grows in CPU resources. # ifconfig enX thread or # chdev -l enX -a thread=on

-Disabled Threading on SEA: Threaded mode helps ensure that virtual SCSI and the Shared Ethernet Adapter can share the processor resource appropriately. IBM documentation talks only about when using VSCSI, so if only using NPIV, consider disabling this parameter. The performance gain could be between 16-20 percent for MTU 1500 and 31-38 percent for Jumbo Frames. To disable threading, use the chdev over the SEA with -attr thread=0 option.

PERF. - IOSTAT

IOSTAT - FCSTAT:

IOPS (I/O per second) for a disk is limited by queue_depth/(average IO service time). Assuming a queue_depth of 3, and an average IO service time of 10 ms, this equals to 300 IOPS for the hdisk. And for many applications this may not be enough throughput.

When an application does an IO request, it is queued at each layer from the app. to the disk:
-fs: filesystem buffer (fsbuf)
-lvm: volume group buffer (pbuf)
-multipath driver (optional)
-hdisk: disk driver queue (queue_depth)
-adapter: FC adapter driver queue (num_cmd_elems)
-SAN fabric devices buffer/cache
-Storage server cache

-----------------------------------

IOSTAT:

We can collect data using iostat during the peak load of the system (for 15-30 mins using 15-30s intervals). It can be executed in parallel with additional data collection tools (nmon …) to establish the performance across all the layers.

iostat -s            print the System throughput report since boot
iostat -d hdisk1 5   display only 1 disk statistics (hdisk1)
iostat -a 5          shows adapter statistics as well


iostat -DRTl 5 2     shows 2 statistics in 5 seconds interval
            -D       extended report
            -R       resest min. max values at each interval
            -T       adds timestamp
            -l       long listing mode


Disks:              xfers                            read                            write                           queue           
------  -------------------------------- ------------------------------  ----------------------------- ---------------------------------
         %tm  bps     tps  bread  bwrtn   rps    avg    max   time fail  wps   avg    max  time  fail  avg   min   max   avg   avg  serv
         act                                     serv   serv  outs             serv   serv  outs       time  time  time  wqsz  sqsz qfull
hdisk4  13.9  212.2K  13.9 156.7K  55.5K   9.6   14.9   40.4   0    0    4.4   0.6    1.1     0    0   0.0   0.0   0.0   0.0   0.0   0.0
hdisk5  20.1  246.4K  16.5 204.8K  41.6K  12.4   16.2   49.8   0    0    4.2   0.6    1.3     0    0   0.0   0.0   0.0   0.0   0.0   0.0
hdisk6  19.7  282.4K  17.1 257.9K  24.5K  14.7   13.6   37.3   0    0    2.4   0.6    1.1     0    0   0.0   0.0   0.0   0.0   0.0   0.0
hdisk7  18.7  300.3K  20.3 215.4K  84.9K  13.1   13.9   39.7   0    0    7.2   0.7    5.6     0    0   0.0   0.0   0.0   0.0   0.0   0.0

-----------------------------------

xfers (transfers):

%tm act:     percent of time the device was active (we can see if disk load is balanced correctly or not, 1 used heavily others not)
bps:         amount of data transferred (read or written) per second (default is in bytes per second)
tps:         number of transfers per second
             (Transfer is an I/O request, and multiple logical requests can be combined into 1 I/O request with different size)
bread:       amount of data read per second (default is in bytes per second)
bwrtn:       amount of data written per second (default is in bytes per second)

%tm_act:
The OS has a sensor, regularily asking the disk if it is busy or not. When the disks answers half of the times "I'm busy", then the "% tm_act" will be 50%. If the disk answers every time "I'm busy" then tm_act will be 100%, etc.. A disk answers with "busy", when there are requested operations not yet fulfilled, read or write. If many very small requests go to the disk the chance of the sensor asking exactly when one such operation is still open goes up - much more so than the real activity of the disk.

So, "100% busy" does not necessarily mean the disk is at the edge of its trasnfer bandwidth. It could mean either that because the disk is getting relatively few but big requests (example: stream I/O) but it could also mean that the disk is getting a lot of requests which are relatively small so that the disk is occupied most of the time, but not using its complete transfer bandwith.
To find out which is the case analyse the corresponding "bread" and "bwrtn" column from iostat.

General rule of thumb, if %tm_act greater than 70%, than probably better to migrate something to other disks as well, the more drives that your data hits, the better.

-----------------------------------

read/write:

rps/wps:     number of read/write transfers per second.
avgserv:     average service time per read/write transfer (default is in milliseconds)
timeouts:    number of read/write timeouts per second
fail:        number of failed read/write requests per second

-----------------------------------

queue:

avgtime:     average time spent in the wait queue (waiting to get sent to the disk, the disk's queue is full) (default is in millisecs)
avgwqsz:     average wait queue size (waiting to be sent to the disk)
avgsqsz:     average service queue size (this can't exceed queue_depth for the disk)
sqfull:      number of times the service queue becomes full per sec. (rate per sec. at which I/O requests are submitted to a full queue)

-----------------------------------

Tuning considerations

read avgserv: should be less than 10ms
write avgserv: should be less than 3ms
avgtime, serv qfull, avgwqsize: should be 0

If avgtime, serv qfull and avgwqsize (often) are above 0, the queue is overloaded and the queue_depth variable should be increased. (If I/O service times (avgserv) are poor increasing queue_depth does not help because I/Os will wait at the storage rather than in the queue.)

There is an "in process" and a "wait" queue at each layer, the "in process queue" is sometimes referred to as the "service" queue. Once the queue limit is reached, the IOs go into to "wait queue" until an IO completes and a slot is freed up in the service queue. From the application's point of view, the length of time to do an IO is its service time plus the time it waits in the hdisk wait queue (avgserv+avgtime). Time spent in the wait queue indicates increasing queue_depth may improve performance.

During tuning very useful to run "iostat -D" which shows statistics since system boot, (for this history statistics in sys0 should be iostat=true: lsattr -El sys0).

When you increase the queue_depths (so more IOs will be sent to the disk subsystem), the IO service times are likely to increase (avgserv), but throughput will also increase. If IO service times start approaching the disk timeout value, then you're submitting more IOs than the disk subsystem can handle. If you start seeing IO timeouts and IO completing errors in the error log, then this is the time to look for hardware problems or to make the pipe smaller.

# lsattr -El hdisk400 | grep timeout
rw_timeout    40                 READ/WRITE time out value        True

If read/write max service times are 30 or 60 seconds or close to what the read/write time out value is set to, this likely indicates a command timeout or some type of error recovery the disk driver experienced.

A general rule for tuning queue_depths, is that one can increase queue_depths until IO service times start exceeding ~10-15 ms for small random reads or writes or one isn't filling the queues. Once IO service times start increasing, we've pushed the bottleneck from the AIX disk and adapter queues to the disk subsystem.

ndisk can be used to test what are the limits. Caches and IO service times will affect test results. Read cache hit rates typically increase the second time you run a test. After the first test the cache should be flushed (umount). Write cache helps performance until the write caches fill up at which time performance goes down, so longer running tests with high write rates can show a drop in performance over time. For write caches, consider monitoring the cache to see if it fills up and run your tests long enough to see if the cache continues to fill up faster than the data can be off loaded to disk.

The downside of setting queue depths too high, is that the disk subsystem won't be able to handle the IO requests in a timely fashion, and may even reject the IO or just ignore it. This can result in an IO time out, and IO error recovery code will be called.


-----------------------------------

iowait

%iowait is the percentage of time the CPU is idle AND there is at least one I/O in progress (all CPUs averaged together). A system with four CPUs and one thread doing I/O will report a maximum of 25% iowait. A system with 12 CPUs and one thread doing I/O will report a maximum of 8.3% iowait. High I/O wait does not mean definitely an I/O bottleneck or zero I/O wait dos not mean there is no I/O botleneck. If aix is waiting on a write and has nothing else to do it will keep looking for the incoming i/o completion and in this time 'book' all its time to an i/o wait because that is all it is trying to do. The opposite situation when an application may be busy processing other requests while IOs are taking a long time to complete, and in this case we will see low iowait percentage.

-----------------------------------

PERF. - TOPAS, NMON

TOPAS - NMON:


TOPAS

Reports selected local and remote system statistics. The topas command requires the bos.perf.tools and perfagent.tools file sets to be installed on the system.

Navigation is possible between the columns with the narrow keys (<-, ->, ...)

------------------------

Default View (small letters):

- c n d f p – cool!!!    = CPU Network Disk Filesystem Processes

- c --> CPUs           c --> graph
                       c --> all
                       c --> off

- n --> networks       n --> totals
                       n --> each interface
                       n --> off

- d --> disks          d --> totals
                       d --> each disk
                       d --> off

- f --> filesystems    f --> totals
                       f --> each filesystems
                       f --> off

- p --> processes      p --> top 20 processes
                       p --> off

- if present: t = tape, w=WLM, @=WPARs
- a=reset all

------------------------

Detailed View (capital letters):

- D --> Disks in full detail    m --> Multi-path I/O (only if it is used)
                                d --> adapter view (scsi, vscsi)

                      in adapter view (d):
                                f --> disks (devices) attached to that adaper (first navigate to an adapter with arrows then hit f)
                                v --> virtual adapters only (vscsi)
               
- E --> Ethernet (shows adapters, SEA, Eth. Chan....) (SEA interface must be in up state)

- F --> Filesystems (more details than "f")

- L --> LPAR settings (SMT, physc, %entc..) and individual CPUs (logical CPUs) usage

- P --> Processes details (CPU%, TIME, Page space usage; page space usage is shown only here!!!)

- T --> Tape if there is a ATAPE device attached

- V --> Volume group statistics
                       in volume group view:
                                f --> LVs in the VG (first navigate to a vg then hit f)

- W -->  WLM then @ --> WPAR

------------------------

CEC or Cross Partitions View or Whole Machine

topas -C    or    topas and hit C

On HMC the "Allow performance information collection" should be enabled for the LPAR.

topas -C might not be able to locate partitions residing on other subnets. To circumvent this, create a $HOME/Rsi.hosts file containing the fully qualified host names for each partition (including domains), one host per line.

- s and d --> Shared CPU and Dedicated CPU sections
- g --> Global
- m --> Memory pool = AMS stats from Hypervisor (select 0 hit f - note:only 1 pool)
- p --> CPU pool stats
- v --> VIO Server/Client disk use f to select the VIOS

------------------------

Navigation between topas and nmon:

------------------------

NMON (Nigel's Monitor):


c, C    CPU usage (c: small view  C:large view)
l -> #  it shows physical cpu usage
m       memory and paging statistics
n       network interface view
k       kernel statistics

t       processes --> [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds
A       AIO processes

.       displays only busy disks and processes

D       disk statistics (read/write KB/s)
d       disk statistics with graph (same as D just with graph)
a       adapter I/O statistics (read/write KB/s, %busy)
^       Fibre channel adapter statistics (fcstat, ^ then a hit e.g. space)

j       jfs view
V       volume group statistics (read/write KB/s)

p       shared processor logical partition view
O       Shared Ethernet adapter statistics ("O" means OCean, SEA=sea)


nmon -k < disklist >    Reports only the disks in the disk list. (e.g. nmon -k hdisk1,hdisk2 only with original nmon)


If you use the same set of keys every time the nmon command is started, you can place the keys in the NMON shell variable.
For example, you can run the following command:
export NMON=mcd    (it will display by default memory, CPU, disk statistics)

------------------------

Capturing NMON data to file:

nmon -f -s "seconds" -c "count"

capture of a busy hour with 10 seconds interval: nmon -f -s 10 -c 360
(60minsx60secs=3600 secs-> with a 10 seconds interval it is 360 snapshots)

capture of a day with 5 mins interval: nmon -f -s 300 -c 288
(86400 seconds in a day divided by 300 (5 mins)= 288 snapshots)

For a detailed graph you can increase the count number to 600-700, but it has no value to go above that.
(Every point in a graph takes 3 pixels, so 600x3=1800 pixels are needed on your screen to see that graph)

-m <dir>    give output directory to nmon file
-T          captures top processes with command arguments (-t captures only top processes without command arguments)
-N          add NFS stats
-^          add FC stats
-O          add VIOS SEA stats

------------------------

Some performance information (related to nmon):
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/nmon_FAQ

nmon reports, more than 100% CPU utilisation for a process:
Unlike AIX commands, nmon reports the CPU use of a process per CPU. If your process is, for example, taking 250% then it is using 2.5 CPUs and must be multiple threaded. This is far better than the AIX tools because the percentages on larger machines make it very hard to determine if a process is using a whole CPU. On a 64 CPU machine a single process uselessly spinning on the CPU takes up 1.56% of the total CPU - this makes it very unclear what is going on.


adapter busy goes over 100%:
there are no adapter stats in AIX. They are derived from the disk stats. The adapter busy% is simply the sum of the disk busy%.
So if the adapter busy% is, for example, 350% then you have 3.5 disks busy on that adapter. Or it could be 7 disks at 50% busy or 14 disks at 25% or ....
There is no way to determine the adapter busy, the adapter has a dedicated on-board CPU that is always busy and we don't run nmon of these adapter CPUs to find out what they are really doing!!


CPU wait is too high:
CPU "waiting for I/O" means the CPU is Idle but has a disk I/O outstanding. In history this was used to highlight that your application is being held up by slow disks or disks problems. In the Wait for I/O state the CPU is actually free to do other work and the CPU is NOT looping waiting for the disk - it in fact actioned the adapter to perform the disk I/O, put the calling process to sleep and carried on. If there is no other process it is in the same loop as in the Idle state i.e. it is available to do other things.

In benchmarks, Wait for I/O is seen positively as an opportunity - we can do throw in more work to boost throughput. In fact, faster CPUs would mean even high wait values.

free memory is near zero:
This is just how AIX works and is perfectly normal. All of memory will be soaked up with copies of filesystem blocks after a reasonable length of time and the free memory will be near zero. AIX will then use the lrud (least recently used daemon) process to keep the free list at a reasonable level. If you see the lrud process taking more than 30% of a CPU then you need to investigate and make memory parameter changes.

------------------------

PERF. - BASICS

Basics:

We can't tune what is not being taxed, we can't tune what can't be tracked.
(we tune the intensities not the sleepy times)
If it runs fast enough we are done (you can't tune it forever)


Hardware:
- date; uname -a; id; oslevel -s; lparstat -i
- check hardware (prtconf | more -- lsdev | grep vail -- lscfg | grep +)
- think to think or move the data workload
- check near-static structures (lvm, paging space, settings)
- check historical data (events, errorrs)

Memory - VMM
- shared memory segments: ipcs -bm (most of the time not all of the memory is allocated)
  the given values shows what is the maximum size that mem. segments can grow (it is a major component of the computational memory)
- uptime; vmstat -s  (it increments since boot)
- uptime; vmstat -v (I/O goes through fsbufs -> then pbufs (each of them can be exhausted))
- vmo -L; ioo -L

I/O - LVM:
- df -k (how much content is goverened by 1 inode)
- tech-stack map: RAIDset ->LUN->LVM(VG:lv:fs/with options) -> logical content
- iostat -a, iostat -D
- lvmo -a -v <vgname> (pbufs can be checked and increased if needed)
- iostat -AQ 2 (asynchronous I/O stats)

Processes:
- uptime; ps -ekf| egrep "syncd|lrud|nfsd|biod|wait
  match time of lrud with syncd: if lrud is greater, then it should grab your attention (if lower it is fine)
  if lrud is high it is scanning and freeing and scanning and freeing...
  lrud has high priority, so if it is running not much work can be done (reduce lrud to let other processing running)
- ps -kelmo THREAD (shows the threaded world)
- ps guww (shows in descending %CPU (RSS:in real memory SZ:in virtual memory, STIME: start time, TIME: accumulated system time))
- ps gvww (shows in ascending PID (PGIN:how many pages are moved))
- ps -ef | grep -v "Oct 20" (the day of boot has been grepped out and check what processes have been started from that time)
- ps -ef | grep -LOCAL=NO (for Oracle client sessions)


Network:
- netstat -ss (check non-zero values)
- netstat -v (queue overflow)
- nfsstat

6 in 1 tool:
- vmstat -Iwt 2

-------------------

if cpubound -> tprof is used to spot those processes which are using
if memory bound -> svmon is used to help to find what is using the most memory
if i/o bound -> filemon will help to find what is causing all of the disk activity

-------------------


CPU wait is too high, how can I reduce it?

CPU in waiting for I/O mode is not a problem. The CPU is actually in Idle mode but it has noted there is disk I/O outstanding and then it is reported as Wait instead of Idle. Lots of workloads that throw data away faster than it can be read will be seen as high Wait. In Wait for I/O mode it is fully available to run more application code.

In benchmarks, Wait for I/O is seen positively as an opportunity - we can do throw in more work to boost throughput.

Any workload in which the CPU does little work compared to the volume of disk I/O is going to give you high Wait for I/O.

If this high Wait for I/O is a sudden change from the normal pattern then it needs investigating and you should make sure as many disks as possible are involved in the disk I/O.

In fact, faster CPUs would mean even high wait values.


-------------------

Which process consumes most memory?

topas -P, you can tab to page space column to sort on that. It is called "page space" column, because it shows the memory usage which is backed by that amount of paging space (which is the size of the process in memory

-------------------

Which process has used the DISK I/O most frequently?

Start nmon --> t for top processes  -->Hit 5 to list them in I/O order, then look at the Char I/O column

-------------------

Free memory is near zero, how do I free more memory?

This is just how AIX works and is perfectly normal. All memory will be soaked up with copies of filesystem blocks after a reasonable length of time and the free memory will be near zero. If your file systems cache is a large percentage of memory then you are avoiding disk I/O. This is a good thing. You should NOT try to reduce it - this could damage performance.

AIX will then use the lrud process to keep the free list at a reasonable level. If you see the lrud process taking more than 30% of a CPU then you need to investigate and make memory parameter changes.

-------------------

20% paging space usage, how affect performance?

20% of paging space can be allocated but no actual I/O taking place. You need to look at the paging stats to determine, if paging I/O is actually happening. Allocating paging space would not have a performance impact.

-------------------

PERF. - CPU VIRTUALIZATION

VIO - POWERVM

VP: Virtual Processor
EC: Entitled Capacity

VP/Core ratio in the pool:
The sum of VPs in the pool / Cores in the pool --> should be around 2 (or less). Higher the ratio, less uncapped capacity is available.

EC/VP ratio of an LPAR:
If EC/VP ratio is below 0.6 you will not use the core for running programs, but mostly for dispatch cyles (overhead).
EC/VP for VIOS must be at least 0.6-0.8, in order to process incoming network and storage requests.
(You will not have any performance on an LPAR if VIOS is not able to process data.)

Some recommendations say to have VP=ROUNDUP (CE) to the next whole integer number.

-------------------------

Some considerations regarding Uncapped Pool, Hypervisor work...:

Performance on Uncapped Capacity is less predictable (We don't know what is happenning in Uncapped Area, so capacity from there is not guaranteed, additionally it creates additional work to Hypervisor and you may loose processor affinity.)

If you give work to the Hypervisor, the LPAR is paying it (For example too many VPs, or folding is deactivated)

Uncapped processor cycles mean work to the Hypervisor, and it is valuable dispatch time, what is payed by LPARs. (It is not good parctice to count on uncapped area fully, as it is extra work for Hypervisor, and it is not guaranteed.)

It is reasonable to use for a critical LPAR 80% of its own resources (entitled capacity) and 20% from the uncapped area (shared pool). For a test system you can use 90% from shared pool.

If a shared LPAR is using more than 80% of its resources (EC) 24/7 it is good to think about changing it to a dedicated LPAR.

Dedicated vs Shared processors: Wherever possible, using partitions in Dedicated processing mode instead of Sharing mode. (No other partions can use its resources. The partition’s memory remains in the cache for longer duration and thus resulting in faster memory access. )

We should consider rounding up some partitions to the nearest whole processors (for example 1.00, 2.00 etc) as it will reduce the effect of partitions sharing the same processor. If a partition is not sized in whole processor it will be sharing the L2 cache with at least one partition and there will be chances of L2 cache contents getting flushed and subsequently reading the data will take time and resulting in performance degrade.

-------------------------

Virtual Processor number by rungueue:

Check runqueue (processes that are currently running or waiting (queued) to run: topas or vmstat -Iwt 2)
    SMT off: runqueue should be less than Virtual Processors X 2
    SMT 2: runqueue should be less than Virtual Processors X 2 X 2
    SMT 4: runqueue should be less than Virtual Processors X 2 X 4

(The conventional rule of thumb of twice the number of CPUs (One process on CPU while second is waiting for some other factors like disk I/O, paging etc)

-------------------------

L2 Cache and performance:

L2 cache is a fast memory which stores copies of the data from the most frequently used  main memory locations.

First picture shows a Power5 system, second picture Power6 (or Power7) system.

In Power5 systems, there is a single L2 cache on a processor chip which is shared by  both the cores on the chip. In later servers (Power6 and Power7) they have separate L2 cache for each core on a chip. The partition’s performance depends on how efficiently it is using the processor cache.

If L2 cache interference with other partitions is minimal, performance is much better for the partition. Sharing L2 cache with other partitions means there is a chance of processor’s most frequently accessed data will be flushed out of the cache and accessing from L3 cache or from main memory will take more time.


Dedicated Procesor vs Shared Processor
Partitions with dedicated processors perform best. In dedicated mode, no other partitions can use a processor. This way the partition’s memory remains in the cache for longer duration and thus resulting in faster memory access.

Processors in multiple of 2
Power5 servers have a common L2 cache on a chip which is shared by two cores. If we assign processors as a multiple of 2, it will minimize the L2 cache contention with other partitions. (Otherwise each partition has to share its L2 cache with other partitions.) On Power6 and Power7 servers using multiple of two processors is not required as processors have private L2 caches.

Whole Processors in sharing mode
In sharing-processor mode consider rounding up some partitions to the nearest whole processors (for example 1.00, 2.00 etc) as it will reduce the effect of partitions sharing the same processor. If a partition is not sized in whole processor it will be sharing the L2 cache with at least one other partition and there will be chances of L2 cache contents getting flushed and reading the data again will take additional time.

Number of Virtual Processor
Number of virtual processors can be called as "spreading factor". It tells how many physical processors will be assigned on which work will be spread across.
(While spreading across multiple processors allow more work to be done in parallel; it can potentially reduce the effectiveness of L2 cache.)

Given a well-threaded application and 1.6 processing units total, four virtual processor each with 0.4 processing units is likely a better choice than two virtual processors each with 0.8 processing units. An application that isn't well-threaded would likely operate faster with the two 0.8 sized processors.

-------------------------

Some consideration about correct EC and VP ratio:
(EC: Entitiled Capacity, VP: Virtual Processor)

If EC=0.4 and VP=4 it means our EC can grow 10X bigger as we desired. This extra resource will be coming from uncapped area. Do we really need this huge amount spare capacity???? (If the whole machine is busy we could be forced down to 0.4 plus a little, based on our weight.)
This configuration is bad, because EC is lower than actual CPU usage.

The other drawback of this type of config, when LPAR is activated Hypervisor tries to assign memory to it from local chip where it's CPU resides (local memory). Later when thhis LPAR increases and it needs more CPU, probably CPU core will be found on other chip (or CEC or book), so reaching out for the memory from these new CPUs will be much longer.

Recommendation:
Monitor Physical Core usage of the LPAR, and if it is an important LPAR EC should be set up, to cover the CPU peaks. (For example if LPAR uses 1.2 cores most of the time EC should be around 1.5.) So, set EC (to normal peak) and VP correctly (a little higher) to have some spare resource.
Reducing VP number should be considered as well (too many VPs are "shredding" the cores.) Low utilized partitions should be configured with minimal number of virtual processors.


Dedicated and Shared Capped LPARs:
80% for normal work 20% extra capacity

Shared Uncapped LPARs:
EC: should cover regular busy peaks and we think should have that much guaranteed
VP: allows some extra space around 25-50% more

EC: 0.05 - 0.6 -->  VP=1
EC: 0.7 - 1.4  -->  VP=2
EC: 1.5 - 2.3  -->  VP=3

-------------------------

Finding spare capacity on a machine:

1.
- raise EC to peak use and lower VP (force higher SMT thread use)
- monitor CPU pool for unused CPU (lparstat)

2.
-for important LPARs do the same as above
-other LPARs have a standard for example VP=round (EC+1) CPU
-monitor and wait for complaints

3.
-fix few large LPARs (as above)
-remove 1 CPU from the pool each day until it hurts (add to a dedicated LPAR, which is doing nothing)
-when performance problems are arise DLPAR a CPU back to the pool

-----------------------

Spikey workload on a system with lots of busy LPARs not getting resources:
Sometimes disabling CPU folding (with schedo, vpm_xvcpus=-1) can help.

More info:
http://www.powershow.com/view4/4834f6-ZTU3Y/vpm_xvcpus_powerpoint_ppt_presentation

-----------------------

topas:

CPU  User%  Kern%  Wait%  Idle%  Physc   Entc
ALL   46.0   47.8    0.0    6.2   0.61  203.9

Physc: Physical consumption represents the amount of processing unit currently consumed. (The number of physical processors that are consumed.)
Entc: Entitled capacity shows the percentage of processing unit currently consumed compared to processing units allocated to the partition.     (Consequently, uncapped shared partitions can have an entitlement consumption that exceeds 100%.)

Runqueue    1.0
Waitqueue   0.0

Runqueue: On average over 2s, only one thread is ready to run.
Waitqueue: On average over 2s, no thread is waiting for paging to complete.


topas -L:
%usr %sys %wait %idle physc  %entc %lbusy    app    Vcsw    phint   %hypv   hcalls
  40   58     0     2   0.3 104.13   9.00   3.54     758       18    23.9    41667

%lbusy: how much percent of the logical processors are effectively in use

-----------------------

lparstat:

root@vios1: / # lparstat -h 1

%user  %sys  %wait  %idle physc %entc  lbusy   app  vcsw phint  %hypv hcalls
----- ----- ------ ------ ----- ----- ------   --- ----- ----- ------ ------
 48.9  15.2    0.0   35.9  1.96 178.3   14.0 19.88   869   159   60.5  10222
 48.8  12.4    0.0   38.8  2.73 247.9   15.2 19.65  6138   164   51.7  16838
 51.0  10.3    0.0   38.7  1.81 164.5   10.1 21.90  3128   121   45.5  15001 

lphysc (pc): Physical cores consumed by the partition.

%entc (ec): Percentage of entitled capacity consumed by the partition. (uncapped shared LPARS can exceed 100%)

lbusy: shows the percentage of logical processor utilization that occurs while executing in user and system mode. If this value approaches 100%, it may indicate that the partition could make use of additional virtual processors.

app: Indicates the available physical processors in the shared pool. (Shows if there are free CPUs)
if "app" shows 0.0, it can happen that only the 1st SMT thread is used, and actually there is space for many unused threads on the 2nd 3rd and 4th SMT thread.

vcsw: Indicates the number of virtual context switches

%hypv: Indicates the percentage of physical processor consumption spent making hypervisor calls.
       (Too many virt. cpus may result in high %hypv)

-------------------------

PERF. - VMM

VMM concepts
(In some AIX documentations Working vs Persistent memory segments are used, but sometimes persistent storage (like in svmon)  is referred to JFS file cache only. Because of the double meaning of the word "persistent", I prefer to use Working vs Permanent storage (like in numperm), and using the word persistent for JFS file cache.)

Virtual-memory segments are partitioned in units called pages; each page is either located in real physical memory (RAM) or stored on disk until it is needed. AIX uses virtual memory to address more memory than is physically available in the system. The management of memory pages in RAM or on disk is handled by the VMM.

A page is a fixed-size block of data (usually 4096 byte). A page might be resident in memory (that is, mapped into a location in physical memory), or a page might be resident on a disk (that is, paged out of physical memory into paging space or a file system).

The pagesize command shows the page size used by the system:
$ pagesize
4096

The VMM maintains a free list of available page frames. The VMM also uses a page-replacement algorithm to determine which virtual-memory pages currently in RAM will have their page frames reassigned to the free list.

AIX tries to use all of RAM all of the time, except for a small amount which it maintains on the free list. To maintain this small amount of unallocated pages the VMM uses page outs and page steals to free up space and reassign those page frames to the free list.

overhead             -- The load that AIX incurs while sharing resources between user processes and performing its internal accounting.
page                 -- A fixed-size (4KB) block of memory.
page fault           -- It occurs when a process tries to access an address in virt mem. that does not have a location in physical memory.
                        In response, the system tries to load the appropriate data from the hard disk
page stealing daemon -- The daemon responsible for releasing pages of memory for use by other processes
                        (It makes room for incoming pages, by swapping out mem. pages that are not the part of the working set of a process.)
paging in            -- Reading pages from swap.
paging out           -- Releasing pages of physical memory for use.

Kernel continuously checks to see if the number of pages on the free list is below a threshold. If so the page stealing daemon, becomes active and begins copying pages to the swap area, starting with least recently used pages. Each page placed on the free list then becomes available for use by other processes. Pages written out to swap must be read back into physical memory when the process needs them again.

The AIX VMM integrates cached file data with the management of other types of virtual memory (for example, process data, process stack, and so forth). It caches the file data as pages, just like virtual memory for processes. (In most modern computer systems, each thread has a reserved region of memory referred to as its stack.)

------------------

Working Storage

Working storage pages are pages that contain volatile data (in other words, data that is not preserved across a reboot). It can be like Process data, Stack, Shared memory, Kernel data

When modified working storage pages need to be paged out (moved from memory to the disk), they are written to paging space. Working storage pages are never written to a file system.

When a process exits, the system releases all of its private working storage pages. Thus, the system releases the working storage pages for the data of a process and stack when the process exits.


Permanent Storage

Permanent storage pages are pages that contain permanent data (that is, data that is preserved across a reboot). This permanent data is just file data. So, permanent storage pages are basically just pieces of files cached in memory.

When a modified permanent storage page needs to be paged out (moved from memory to disk), it is written to a file system.

You can divide permanent storage pages into two sub-types:

    - Non-client pages (aka persistent pages): these are pages containing cached Journaled File System (JFS) file data
    - Client pages: These are pages containing cached data for all other file systems (for example, JFS2 and Network File System (NFS)

------------------

In order to help optimize which pages are selected for replacement by the page replacement daemons, AIX classifies pages into one of two types:

    - Computational pages: pages used for the text, data, stack, and shared memory of a process
    - Non-computational pages: pages containing file data for files that are being read and written.

All working storage pages are computational. A working storage page is never marked as non-computational.

Depending on how you use the permanent storage pages, the pages can be computational or non-computational. If a file contains executable text for a process, the system treats the file as computational and marks all of the permanent storage pages in the file as computational. If the file does not contain executable text, the system treats the file as non-computational file and marks all of the pages in the file as non-computational. (Basically every file starts as being non-computational. When there is a reference to a memory block which contains instructions (not just data), then that block and all other blocks belonging to that file marked as computational.)

Once a file has been marked as computational, it remains marked as a computational file until the file is deleted (or the system is rebooted). Thus, a file remains marked as computational even after it is moved or renamed.


------------------

Page replacement

The AIX page replacement daemons scan memory a page at a time to find pages to evict in order to free up memory. The page replacement daemons must choose pages carefully to minimize the performance impact of paging on the system, and the page replacement daemons target pages of different classes based on tunable parameter settings and system conditions.

There are a number of tunable parameters that you can use to control how AIX selects pages to replace.

------------------

minperm and maxperm

These tunable parameters are used to indicate how much memory the AIX kernel should use to cache non-computational pages. The maxperm tunable parameter indicates the maximum amount of memory that should be used to cache non-computational pages. The minperm limit indicates the target minimum amount of memory that should be used for non-computational pages.

By default, maxperm is an "un-strict" limit, so it allows more non-computational files to be cached in memory when there is available free memory. The maxperm limit can be made a "strict" limit by setting the strict_maxperm tunable parameter to 1.
(The disadvantage of this is, that the number of non-computational pages cannot grow beyond maxperm and consume more memory when there is free memory on the system.)

numperm (lru_file_repage)

The number of non-computational pages is referred to as numperm: The vmstat -v command displays the numperm value for a system as a percentage of a system’s real memory.

When the number of non-computational pages (numperm) is greater than maxperm, the AIX page replacement daemons strictly target non-computational pages (for example, cached files that are not executables).

When the number of non-computational pages (numperm) is less than minperm, the AIX page replacement daemons target both computational and non-computational pages. In this case, AIX scans both classes of pages and evicts the least recently used pages.

When the number of non-computational pages (numperm) is between minperm and maxperm, the lru_file_repage (least recently used) tunable parameter controls what kind of pages the AIX page replacement daemons should steal. If lru_file_repage set to 0, AIX always targets non-computational pages when numperm is between minperm and maxperm.

In most customer environments, it is most optimal to just have the kernel always target non-computational pages, because paging computational pages (for example, a process’s stack, data, and so forth) usually has a much higher performance cost on a process than paging non-computational pages (that is, data file cache). Thus, the lru_file_repage tunable parameter can be set to 0. In this case, the AIX kernel always targets non-computational pages when numperm is between minperm and maxperm.

------
maxclient

maxclient specifies a limit on the maximum amount of memory that should be used to cache non-computational client pages. Because all non-computational client pages are a subset of the total number of non-computational permanent storage pages, the maxclient limit must always be less than or equal to the maxperm limit.

numclient

The number of non-computational client pages is referred to as numclient. The vmstat -v command displays the numclient value for a system as a percentage of a system’s real memory.

By default, the maxclient limit is a strict limit. This means that the AIX kernel does not allow the non-computational client file cache to exceed the maxclient limit (that is, the AIX kernel does not allow numclient to exceed maxclient). When numclient reaches the maxclient limit, the AIX page replacement daemons strictly target client pages.

------

minfree, maxfree

Two other important parameters are minfree and maxfree. If the number of pages on your free list (vmstat -v: free pages) falls below the minfree parameter, VMM starts to steal pages (just to add to the free list), which is not good. It continues to do this until the free list has at least the number of pages in the maxfree parameter.

------

# vmstat -v 
       4980736 memory pages
        739175 lruable pages
        432957 free pages                     <--6 digit generous, 5 digit ideal, 4 digits trouble, 3 digits big trouble
             1 memory pools
         84650 pinned pages
          80.0 maxpin percentage
          20.0 minperm percentage
          80.0 maxperm percentage
           2.2 numperm percentage             <--% of memory containing non-comp. pages (jfs, jfs2, nfs)
         16529 file pages                     <--# of non-comp. pages
           0.0 compressed percentage
             0 compressed pages
           2.2 numclient percentage           <--% of memory containing non-comp. client pages (jfs2, nfs)
          80.0 maxclient percentage
         16503 client pages                   <--# of non-comp client pages


So, in the above example, there are 16529 non-computational file pages mapped into memory. These non-computational pages consume 2.2 percent of memory. Of these 16529 non-computational file pages, 16503 of them are client pages.

The vmstat output does not provide information about computational file pages. Information about computational file pages can be gathered from the svmon command

# svmon -G                <--in memory pages of each type (work, pers., client)
               size      inuse       free        pin    virtual
memory       786432     209710     576722     133537     188426
pg space     131072       1121

               work       pers       clnt
pin          133537          0          0
in use       188426          0      21284

    - work: working storage
    - pers: persistent storage (persistent storage pages are non-client pages - that is, JFS pages.)
    - clnt: client storage (jfs2, nfs)

For each page type, svmon displays two rows:

    - in use: number of 4K pages mapped into memory
    - pin: number of 4K pages mapped into memory and pinned (pin is a subset of inuse)

So, in the above example, there are 188426 working storage pages mapped into memory. Of those 188426 working storage pages, 133537 of them are pinned (that is, can’t be paged out).

There are no persistent storage pages (because there are no JFS filesystems in use on the system). There are 21284 client storage pages, and none of them are pinned.

The svmon command does not display the number of permanent storage pages, but it can be calculated from the svmon output. As mentioned earlier, the number of permanent storage pages is the sum of the number of persistent storage pages and the number of client storage pages. So, in the above example, there are a total of 21284 permanent storage pages on the system:

0 persistent storage pages + 21284 client storage pages = 21284 permanent storage pages

The type of information reported by svmon is slightly different than vmstat. svmon  reports information about the number of in-memory pages of different types: working, persistent (that is, non-client), and client. svmon does not report information about computational versus non-computational. svmon just reports the total number of in-memory pages of each page type.

In contrast, vmstat reports information about non-computational versus computational pages.

To illustrate this difference, consider the above example of svmon output. Some of the 21284 client pages will be computational, and the rest of the 21284 client pages will be non-computational. To determine the breakdown of these client pages between computational and non-computational, use the vmstat command to determine how many of the 21284 client pages are non-computational.

-----------
suggested:

lru_file_repage = 0
maxperm = 90%
maxclient = 90%
minperm = 3%
strict_maxclient = 1 (default)
strict_maxperm = 0 (default)

# vmo -p -o lru_file_repage=0 -o maxclient%=90 -o maxperm%=90 -o minperm%=3
# vmo -p -o strict_maxclient=1 -o strict_maxperm=0

The above tunable parameters settings are the default settings for AIX Version 6.1.

With these settings computational memory pages will be NOT paged to disk until you do not exceed 97% computational memory (minperm=3%). If computational memory exceed 97% than the system does not care anymore what is in memory, the last recently used pages (can be comp. or non-comp.) will be paged out.

-----------------------------

An example:

topas:

 MEMORY
 Real,MB   26623
 % Comp     57          <--this is used for processes (OS+appl.), if you add nmon Process+System, for me it was the same (46+11)
 % Noncomp  22          <--fs cache
 % Client   22          <--fs cache (for jfs2)


nmon:

 FileSystemCache
 (numperm) 22.5%        <--this is for fs cache
 Process   46.0%        <--this is for appl. processes
 System    11.3%        <--this is for the OS
 Free      20.2%        <--free
           -----
 Total    100.0%

-----------------------------