POWERVM - SEA

Shared Ethernet Adapter (SEA)

A SEA can be used to connect a physical Ethernet network to a virtul Ethernet network. The SEA hosted in the Virtual I/O Server acts as a layer-2 bridge between the internal and external network. With Shared Ethernet Adapters on the Virtual I/O Server, virtual Ethernet adapters on client logical partitions can send and receive outside network traffic.

Shared Ethernet Adapter is a Virtual I/O Server component that bridges a physical Ethernet adapter and one or more virtual Ethernet adapters:
-The real adapter can be a physical Ethernet adapter, a Link Aggregation or EtherChannel device, or a Logical Host Ethernet Adapter . The real adapter cannot be another Shared Ethernet Adapter or a VLAN pseudo-device.
-The virtual Ethernet adapter (trunk adpater in the SEA) must be created with the following settings:

Adapter ID: Any ID for the Virtual ethernet adapter
Port Virtual Ethernet: PVID given to this adapter (usually a VLAN ID which is not used at any other adapter to avoid untagging packets)
IEE 802.1q: Additional VLAN IDs can be specified here
Ethernet bridging: This checkbox enables accessing external networks
Priority: For SEA Failover mode, you can specify which SEA should be the primary (here it is the secondary SEA)


--------------------------------------------------

A Shared Ethernet Adapter provides access by connecting the internal VLANs with the VLANs on the external switches. Using this connection, logical partitions without physical adapters can share the IP subnet with stand-alone systems and other external logical partitions. (A virtual Ethernet adapter connected to the SEA must have the Access External Networks check box enabled.)

The Shared Ethernet Adapter forwards outbound packets received from a virtual Ethernet adapter to the external network and forwards inbound packets to the appropriate client logical partition over the virtual Ethernet link to that logical partition.

IF SEA failover has been configured leave SEA without IP addresses. (It makes maintenance of SEA also easier)

Checking SEA on VIO server:
padmin@vios1: / # lsdev -dev ent* | grep Shared
ent8    Available       Shared Ethernet Adapter

padmin@vios1: / # lsdev -dev ent8 -attr | grep adapter
pvid_adapter  ent4     Default virtual adapter to use for non-VLAN-tagged packets         True
real_adapter  ent0     Physical adapter associated with the SEA                           True

---------------------------------------------------

Quality of Service

Quality of Service (QoS) is a Shared Ethernet Adapter feature which infulences bamdwidth. QoS allows the Virtual I/O Server to give a higher priority to some types of packets. Shared Ethernet Adapter on the VIO Server can inspect bridged VLAN-tagged traffic for the VLAN priority field in the VLAN header. The 3-bit VLAN priority field allows each individual packet to be prioritized with a value from 0 to 7 to distinguish more important traffic from less important traffic. More important traffic is sent preferentially and uses more Virtual I/O Server bandwidth than less important traffic.

---------------------------------------------------

PVID:

The SEA directs packets based on the VLAN ID tags. One of the virtual adapters in the SEA must be designated (at creation) as the default PVID adapter (ent1 on the below picture). Ethernet frames without any VLAN ID tags that the SEA receives from the external network are forwarded to this adapter and assigned the default PVID

---------------------------------------------------

SEA and VLAN traffic:

The VLAN tag information is referred to as VLAN ID (VID). Ports on a switch are configured as being members of a VLAN designated by the VID for that port. The default VID for a port is referred to as the Port VID (PVID). The VID can be added to an Ethernet packet either by a VLAN-aware host, or by the switch in the case of VLAN-unaware hosts.

For VLAN-unaware hosts, a port is set up as untagged and the switch will tag all packets entering through that port with the Port VLAN ID (PVID). The switch will also untag all packets exiting that port before delivery to the VLAN unaware host. A port used to connect VLAN-unaware hosts is called an untagged port, and it can be a member of only one VLAN identified by its PVID.

Hosts that are VLAN-aware can insert and remove their own tags and can be members of more than one VLAN. These hosts are typically attached to ports that do not remove the tags before delivering the packets to the host, but will insert the PVID tag when an untagged packet enters the port.

A port will only allow packets that are untagged or tagged with the tag of one of the VLANs that the port belongs to.

Based on the above image, incoming packets from external networks:
- SEA forwards untagged packets to ent1 and these are tagged with the default PVID=1
- SEA forwards packets with VID=1 or VID=10 to adapter ent1 as well
- before LPAR2 recieves packets Hypervisor will remove VLAN tag
- en0 on LPAR1 will receive untagged packets
- en1 on LPAR1 will receive only packets with VID=10

Outgoing packets to external networks:
- packets sent by LPAR2 will be tagged by Hypervisor, with PVID=1
- packets sent by LPAR1 through en1 are tagged with VID=10 by AIX, and en0 packets are tagged with PVID=1 by Hypervisor
- at VIOS: packets tagged with VID=10, are processed with the VLAN tag unmodified.
- at VIOS: packets with VID=1 (PVID of ent1 in SEA) are untagged before ent1 receives them, then bridged to ent0 and sent out.
 (VLAN-unaware destination devices on the external network will be able to receive these packets.)

(The virtual Ethernet adapter ent1 of the SEA also uses VID 10 and will receive the packet from the POWER Hypervisor with the VLAN tag unmodified. The packet will then be sent out through ent0 with the VLAN tag unmodified. So, only VLAN-capable destination devices will be able to receive these. )

---------------------------------------------------

Shared Ethernet Adapter Failover:

In a Shared Ethernet Adapter failover configuration there are two Virtual I/O Servers, each running a Shared Ethernet Adapter. The Shared Ethernet Adapters communicate with each other on a control channel using two virtual Ethernet adapters configured on a separate VLAN. The control channel is used to carry heartbeat packets between the two Shared Ethernet Adapters. When the primary Shared Ethernet Adapter loses connectivity the network traffic is automatically switched to the backup Shared Ethernet Adapter.

The trunk priority for the Virtual Ethernet adapters on VIO Server 1 (which has the Access external network flag set) is set to 1. This means that normally the network traffic will go through VIO Server 1. VIO Server 2 with trunk priority 2 is used as backup in case VIO Server 1 has no connectivity to the external network.

more info: https://www-304.ibm.com/support/docview.wss?uid=isg3T1011040

---------------------------------------------------

Shared Ethernet Adapter failover with Loadsharing

The Virtual I/O Server Version 2.2.1.0, or later, provides a load sharing function to enable to use the bandwidth of the backup Shared Ethernet Adapter (SEA).It makes an effective use of the backup SEA bandwidth.

In this example the packets of VLAN 10 will go through VIOS1 and packets of VLAN 20 will go through VIOS2.

Prerequisites:
- Both of primary and backup Virtual I/O Servers are at Version 2.2.1.0, or later.
- Two or more trunk adapters are configured for the primary and backup SEA pair.
- The virtual local area network (VLAN) definitions of the trunk adapters are identical between the primary and backup SEA pair.

To create or enable the SEA failover with Load Sharing, you have to enable the load sharing mode on the primary SEA first before enabling load sharing mode on the backup SEA. The load sharing algorithm automatically determines which trunk adapters will be activated and will treat network packets for VLANs in the SEA
pair. You can not specify the active trunk adapters of the SEAs manually in the load sharing mode.

Changing the SEA to Load Sharing mode:
$ chdev -dev ent6 -attr ha_mode=sharing

---------------------------------------------------

To reduce SEA failover time to minimum these can help:

- For all AIX client partitions, set up Dead Gateway Detection (DGD) on the default route:
1. route change default -active_dgd       <--Set up DGD on the default route
2. in etc/rc.tcpip add: route change default -active_dgd to the        <--it makes this change to permanent
3. no -p -o dgd_ping_time=2       <--set pings interval of a gateway by DGD to 2 seconds
       (default is 5s; 2s will allow faster recovery):
- On the network switch, enable PortFast if Spanning Tree is on or disable Spanning Tree.
- On the network switch, set the channel group for your ports to Active if they are currently set to Passive

---------------------------------------------------

Simplified SEA (without control channel):

SEA can implement a new method to discover SEA pair partners using the VLAN ID 4095 in its virtual switch. After partners are identified, a new SEA high availability (HA) protocol is used to communicate between them.

If the followings are met during SEA creation no control channel adapter is necessary:
-VIOS Version 2.2.3
-Hardware Management Console (HMC) 7.7.8
-Firmware Level 780 or higher

---------------------------------------------------

Good overview of SEA sharing mode + VLANs:
entstat -all entX | grep -e "  Priority" -e "Virtual Adapter" -e "  State:" -e "High Availability Mode" -e "  ent"

Good overview of SEA Link status + MAC address:
entstat -all entX | grep -e "(ent" -e "Type:" -e "Address:" -e "Link Status" -e "Link State:" -e "Switch "

---------------------------------------------------

Checking SEA Load sharing distribution:

# entstat -all ent8 | grep -e "  Priority" -e "Virtual Adapter" -e "  State:" -e "High Availability Mode"

ent8:       SEA adapter
ent4, ent5: Trunk virtual ethernet adapters in SEA

VIO1:
State: PRIMARY_SH                    <--shows it is in load sharing mode and it is the primary SEA adapter (if we were in failover mode)
High Availability Mode: Sharing
Priority: 1
...
Virtual Adapter: ent4
  Priority: 1  Active: False
Virtual Adapter: ent5
  Priority: 1  Active: True

VIO2:
State: BACKUP_SH                    <--shows it is in load sharing mode and it is the backup SEA adapter (if we were in failover mode)
High Availability Mode: Sharing
Priority: 2
...
Virtual Adapter: ent4
  Priority: 2  Active: True
Virtual Adapter: ent5
  Priority: 2  Active: False

---------------------------------------------------

SEA and SEA failover creation:

To create a Shared Ethernet Adapter (SEA) you need:
- <PHYS>: a physical adapter as backend
- <VIRT>: a virtual adapter
- <VLAN>: an internal VLAN ID
- default: specifies the default virtual adapter to be used for non-VLAN-tagged packets
- defaultid:  this VLAN ID used for untagged packets (the PVID used for the SEA device)

for SEA failover:
- <CONT>: a second virtual adapter for the control channel

+ optional settings:
-netaddr: SEA will periodically ping this IP address, so it can detect network failures
-largesend: enable TCP segmentation offload


 # simple SEA
 $ mkvdev -sea <PHYS> -vadapter <VIRT> -default <VIRT> -defaultid <VLAN>

 # Shared Ethernet Adapter Failover:
 $ mkvdev -sea <PHYS> -vadapter <VIRT> -default <VIRT> -defaultid <VLAN> -attr ha_mode=auto ctl_chan=<CONT>

 # Shared Ethernet Adapter Failover without control channel, example:
 $ mkvdev -sea ent14 -vadapter ent8 ent10 ent12 ent13 -default ent8 -defaultid 4000 -attr jumbo_frames=yes ha_mode=auto
 (After creation possible to change to sharing mode, 1st on primary VIO after backup VIO: chdev -dev ent15 -attr ha_mode=sharing

 (with optional settings)
 $ mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1 -attr ha_mode=auto ctl_chan=ent3 netaddr=9.3.4.1 largesend=1

(Any interface with an IP address on the adapters used when defining the SEA must be detached.)
(When you want to change something on SEA (enable/disable load sharing...), do the change on the primary SEA first, then set it on the backup SEA.)

---------------

adding a virtual adapter later to the SEA:

chdev -dev entx -attr virt_adapters=entY,entZ
(entX: SEA adapter; entY,entZ: virtual adapters - all virt. adapters has to be listed here, not just the new one)

---------------

Changing SEA online (without downtime):

SEA configured on VIO1 with priority 1 and on VIO2 on priority 2 (it is important when changing sharing mode)
SEA configured in load sharing mode, so first I change it to auto, and after to standby on each VIO where I work:

1.chdev -dev entX -attr ha_mode=auto                        <--1st on VIO1 after VIO2 change to auto mode, so both will have auto

2.chdev -dev entX -attr ha_mode=standby                     <--on VIO1: so network will go through on VIO2
3.rmvdev -sea entX                                        <--on VIO1: remove SEA
4.rmvdev -lnagg entY                                      <--on VIO1: remove Etherchannel
5.<<do any change/HW repair>>
6.mkvdev -lnagg ent0 ent1 -attr mode=8023ad...              <--on VIO1: recreate Etherchannel
7.mkvdev -sea ent2 -vadapter ent8 ent9 ... ha_mode=standby  <--on VIO1: recreate SEA
8.chdev -dev entX -attr ha_mode=auto                        <--on VIO1: set back ha_mode to auto, so traffic will go based on priority

do same tasks (from standby) on VIOS2...when finished:
chdev -dev entX -attr ha_mode=sharing                       <--1st on VIO1 after on VIO2


This works as well:
rmdev -l ent15
chdev -l ent15 -a jumbo_frames=yes
mkdev -l ent15

---------------

remove VLAN from SEA (HMC CLI):
lshwres -m <man.sys.> -r virtualio --rsubtype eth --level lpar --filter "lpar_names=<vio name>" -F lpar_name,port_vlan_id,addl_vlan_ids
chhwres -r virtualio --rsubtype eth -m <man.sys.> -o s --id <lpar id> -s <virt eth id> -a "addl_vlan_ids-=311"
(lpar id and virt. eth. adapter id on HMC is needed, for checking: entstat ..., and save vio profile may needed)


remove VLAN from SEA (HMC GUI):
choose Man. Sys. --> Virtual Networks --> choose VLAN --> Action:remove
chdev -dev entX -attr virt_adapters=entY,entZ
(chdev is needed only because on VIO entstat … still showed removed VLAN, so chdev did the trick.)
(Did not do anything else with SEA, no removal, reconfig is needed only chdev.)

---------------

SEA Failover testing:

On VIOS1 and VIOS2 virtual adapters have been created. At creation time trunk priority has been set:
VIOS1: 1
VIOS2: 2

With command 'mkvdev' SEAs (ent14) have been created on both VIO

1. check settings:

    VIOS1:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=auto
    netstat -v ent14 | grep Active             <--should show: Priority: 1  Active: True

    VIOS2:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=auto
    netstat -v ent14 | grep Active             <--should show: Priority: 2  Active: False

2. perform manual SEA failover:

    VIOS1:
    chdev -l ent14 -a ha_mode=standby

3. check settings:

    VIOS1:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=standby
    netstat -v ent14 | grep Active             <--should show: Priority: 1  Active: False
    errpt | head                               <--should show: BECOME BACKUP

    VIOS2:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=auto
    netstat -v ent14 | grep Active             <--should show: Priority: 2  Active: True
    errpt | head                               <--should show: BECOME PRIMARY

4. switching back:

    VIOS1:
    chdev -l ent14 -a ha_mode=auto

5. check settings:

    VIOS1:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=auto
    netstat -v ent14 | grep Active             <--should show: Priority: 1  Active: True
    errpt | head                               <--should show: BECOME PRIMARY

    VIOS2:
    lsattr -El ent14 | grep ha_mode            <--should show: ha_mode=auto
    netstat -v ent14 | grep Active             <--should show: Priority: 2  Active: False
    errpt | head                               <--should show: BECOME BACKUP

---------------

thread attribute:

Threading ensures that CPU resources are shared fairly when a Virtual I/O Server provides a mix of SEA and VSCSI services.
If it set to 1, it will equalize  the priority between virtual disk and SEA network I/O. This throttles Ethernet traffic to prevent it from consuming a higher percentage of CPU resources versus the virtual SCSI activity. This is a concern only when CPU resources are constrained resources.)

padmin@vios1 : /home/padmin # lsdev -dev ent14 -attr | grep thread
thread        1          Thread mode enabled (1) or disabled (0)                            True

Threading is enabled by default for shared Ethernet adapters.
Disable threading when a Virtual I/O Server is not used for VSCSI (chdev –dev entX –attr thread=0).

---------------

entstat -all ent4                                          shows if this adapter is active or not (entstat -all ent4 | grep Active)
netstat -cdlistats | grep -Ei "\(ent|media|link status"    this lists links on all physical adapter (good!!!)

---------------

Configuring the interface on SEA (adding ip...):

cfgassist or mktcpip command:
mktcpip -hostname VIO_Server1 -inetaddr 9.3.5.196 -interface en3 -netmask 255.255.254.0 -gateway 9.3.4.1

---------------

SEA load sharing mode error:

$ chdev -dev ent23 -attr ha _mode=sharing

Method error (/usr/lib/methods/chgsea):
        0514-018 The values specified for the following attributes
                 are not valid:
ha_mode. Insufficient no. of adapters.


This indicates that you have only 1 virtual adapter configured in the SEA, so load cannot be shared (that is why you cannot chage ha_mode attribute). Add additional Virtual Ethernet Adpater to the SEA for this sharing mode to activate.

---------------

Total Etherchannel failure and LIMBO state on SEA:

During dual VIOS install, when second SEA configured on VIOS2, network connection was lost and received this:

errpt:
CE9566DF   0719154713 P H ent9           TOTAL ETHERCHANNEL FAILURE

entstat for SEA:
    State: LIMBO
    High Availability Mode: Auto
    Priority: 1
...
Virtual Adapter: ent5
  Priority: 1  Active: False
Virtual Adapter: ent4
  Priority: 1  Active: False
Virtual Adapter: ent3
  Priority: 1  Active: False
Virtual Adapter: ent2
  Priority: 1  Active: False


Limbo state means:
The physical network is not operational or network state is unknown, or the Shared Ethernet Adapter cannot ping the specified remote host.

Limbo packets are sent by the primary Shared Ethernet Adapter when it detects that its physical network is not operational, or when it cannot ping the specified remote host (to inform the backup that it needs to become active).

After checking control channel on both SEA, found configuration problem. One of the control channel was in virtual switch ETHERNET0 the other on was in ETHERNET1, so control channel could not work properly. (You can check it in VIOS LPAR properties on HMC, or with entstat command.)

On the VIOS LPAR with wrong control channel:
1. remove SEA device: rmdev...
2. shutdown LPAR and change profile on HMC: control channel virt. adapter to the correct virtual switch
3. start LPAR and create SEA device again.

After this everything was OK.

---------------