dropdown menu

SR-IOV

SR-IOV

SR-IOV (Single Root IO Virtualization) is a network virtualization technology. The basic idea is, that a physical device can have multiple virtual instances of itself, and these can be assigned to any LPARs running on the managed system. We have only 1 physical adapter, but each LPAR will think it has its own dedicated adapter. This is achieved by creating logical ports on top of the physical ports on the adapter. These ports will be exist in the Hypervisor (Firmware), so no VIOS is necessary. In order to see new SR-IOV menu points in HMC GUI, Firmware and HMC should be in a correct level.

Dedicated Mode - Shared Mode:
An SR-IOV capable adapter is either in dedicated mode or shared mode.
In dedicated mode, the adapter is owned by one LPAR. Physical ports of the adapter are owned by that partition. (Usual old config.)
In shared mode, adapter is owned by the Hypervisor. In this mode logical ports can be created and these can be assigned to any LPAR.

Logical Port (LP) - Virtual Function (VF)
A Virtual Function (VF) is a general term used by PCI standards. We can think of it as a slice of the physical port on the adapter. On IBM Power Systems, SR-IOV implements VFs as logical ports. A Logical Port (LP) is an I/O device created for a partition to access the Virtual Function on the adapter. When a Logical Port is created, the hypervisor will configure a virtual function on the adapter, and maps it to the Logical Port. These are in 1 to 1 relationship. (In general documentations the term VF is used many times, but during SR-IOV configuration on HMC, the term LP is used. In these standards another term, the Physical Function (PF) also exists, which we can think as a reference to the Physical Port.)

------------------------------------

Capacity (SR-IOV desired bandwidth)

During Logical Port (Virtual Function) creation the desired capacity need to be configured. This is similar to the Entitled Capacity (CPU) setting, just here % is used. The configured value is the desired minimum bandwidth in percentage. It is not capped, which means, if there is additional bandwidth that is not being used currently, it will be shared equally among all logical ports. Assignments are made in increments of 2 %  and total assignments for a single port can not exceed 100%. Capacity cannot be changed dynamically. (It is possible to change in profile and after profile activation is needed.)

So, if an LPAR needs, it will have % desired outgoing bandwidth. If additional outgoing bandwidth is available, any partition can use it. If a partition doesn’t need its minimum, that bandwidth is available to other partitions until the owning partition needs it. Capacity settings don’t have any influence on the incoming bandwidth.

------------------------------------

SR-IOV and LPM:


Picture shows an adapter with 2 physical ports, and one of those ports, is virtualized into 3 Logical Ports or VFs (yellow squares).

From a virtualization perspective, SR-VIO logical ports are seen as physical adapters at OS level, therefore operations like Live Partition Mobility are not supported when an SR-IOV logical port is configured on the partition (LPAR B).

If a Logical Port is part of a SEA (which bridge traffic from client partitions to the physical network) then client LPAR has only a Virtual Ethernet Adapter (LPAR A), so it can continue using Live Partition Mobility.

------------------------------------

SR-IOV and Link Aggregation

In an LACP configuration multiple primary SR-IOV logical ports are allowed. When LACP (IEEE802.3ad,) configured with multiple main logical ports, only SR-IOV logical ports can be part of the link aggregation and only 1 single logical port can be configured per physical port.

So, with LACP only one logical port per physical port can be used.
(The second configuration, with more than one logical port assigned to a physical port, will not work.)


To prevent users from adding a logical port to the physical port when LACP being used, you can set the logical port capacity to 100%.

------------------------------------

SR-IOV and Etherchannel (NIB):

In an active-passive configuration (Network Interface Backup), SR-IOV logical port can be primary (active) or backup (passive), or both. If more than one primary adapter is configured in an Etherchannel, then SR-IOV logical port cannot be a primary adapter. When an SR-IOV logical port is configured in an active-passive configuration, it must be configured to detect when to fail over from primary to the backup adapter. This can be achieved by configuring an IP address to ping.

------------------------------------

SR-IOV Configuration:
(Here HMC classical GUI has been used, not enhanced HMC GUI.)


1. verify preprequisites: HMC level, Firmware level, compatible SR-IOV adapter, Man. Sys. capabilities (SR-IOV capable: True)

2. change adapter from Dedicated to Shared mode: Man. Sys. Properties --> IO tab --> choose adapter --> click on shared mode
(This happens at Man. Sys. level, so adapter should not be assigned to any LPARs. If you check again after modification, you will see adapter will be owned by Hypervisor. It is possible to switch back to dedicated mode, just already configured logical ports must be de-configured prior to that.)

3. Physical port config: Man. System --> Properties --> IO --> choose an SR-IOV adapter --> choose a port
Here you can configure speed (10Gb), MTU size, label...and Port Switch Mode:

VEB (Virtual Ethernet Bridge) (This is the default)
Bridging between logicalports (VFs) on the same physical port is done by the adapter.
(Logical port to logical port traffic is not exposed to an external switch, lower latency.)

VEPA (Virtual Ethernet Port Aggregator)
Bridging between logical ports on the same physical port is done by an external switch.
(Logical port to logical port traffic flows out to the external switch and then back. Switch can control traffic if needed.)


4.Logical Port config: Logical Port is mapped to an LPAR so an LPAR has to be chosen before config. It can be done as a DLPAR operation, or it can be done in LPAR profile.

Add a logical port to a Physical Port:
DLPAR: choose LPAR --> Dynamic part. --> SR-IOV Log. Ports --> Action --> Add Log. Port --> choose a phys. port to conf. additional Log. Port.
PROFILE: In LPAR profile, SR-IOV Log. Ports -->  SR-IOV Menu --> Add Log. Port -->  choose a physical port to configure additional Log. Port

After new window will pop up:


Capacity:     This is the desired minimum bandwidth of the port`s capacity (only even number can be used up to 100%)
              Logical ports can get more, if there is free bandwidth which is not being used. It is shared equally among all logical ports.
              Capacity settings do not apply to received traffic, only to transmitted traffic.

Promiscuous:  if the logical port will be assigned to a VIOS in a SEA, then enabling promiscuous mode is required.
              (Promiscuous mode can be enabled on only one logical port per physical port.)
              When promiscuous mode is enabled, Allow All VLAN IDs and  Allow all O/S Defined MAC Addresses are the only options available.

After finishing it, on OS level a new adapter will be configured with VF in the name (cfgmgr may needed)

# lsdev -Cc adapter
ent0    Available 02-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent1    Available 02-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent2    Available 02-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent3    Available 02-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent4    Available 0C-00 PCIe3 4-Port 10GbE SR Adapter VF(df1028e21410e304)

# entstat -d ent4
...
...
VF Minimum Bandwidth: 24%
VF Maximum Bandwidth: 100%

3 comments:

  1. Great article! Thanks for the post!

    ReplyDelete
  2. Hi, thank's for info.

    Is multiple logical ports per one physical port supported for Linux-bonding in active-backup mode ?

    ReplyDelete