dropdown menu

ETHERCHANNEL:

EtherChannel is a trademark registered by Cisco Systems and is generally called multi-port trunking or link aggregation. If your Ethernet switch device has this function, you can exploit the support provided in AIX. In this case, you must configure your Ethernet switch to create a channel by aggregating a series of Ethernet ports.

On AIX with "smitty etherchannel", in the same menu, depending how the fields are filled out, 2 distinctly separate functioning devices can be created. This can be seen:

ent2   Available       EtherChannel / IEEE 802.3ad Link Aggregation

1. EtherChannel (Network Interface Backup, NIB)
2. Link Aggregation (802.3ad)

1.Etherchannel (NIB):
Network Interface Backup (NIB) can be used to achieve network redundancy. Two adapters are used to create an EtherChannel that consists of one primary adapter and one backup adapter. The interface is defined on the EtherChannel. If the primary adapter becomes unavailable, the Network Interface Backup switches to the backup adapter.

2. Link Aggregation
Link Aggregation is a network port aggregation technology that allows several Ethernet adapters to be aggregated together to form a single pseudo Ethernet adapter. (There are for example 2 primary adapters).

The main benefit of a Link Aggregation is that it has the network bandwidth of all of its adapters. If an adapter fails, the packets are automatically sent on the next available adapter without disruption to existing user connections. A link or adapter failure will lead to a performance degradation, but not a disruption.


For me once in order to work correctly the Link Aggregation (with LACP) I had to set these:
smitty etherchannel --> Change/Show Characteristics...:
Mode          8023ad
Hash Mode     src_dst_port

(I used smitty, but chdev works as well)

------------------------

Etherchannel/Link Aggregation parameters:

Mode:
standard: this is good for default Etherchannel (primary and backup)
8023ad: this is the LACP (Link Aggregation Control Protocol) configuration (2 primary adapters).
(ad = Autodetect: if our server approaches switch with one adapter, switch sees one adapter,  if our server approaches switch with a Link Aggregation, switch auto detects that. For 10Gb, we should be LACP/8023ad.)

Hash Mode
default: is by IP address, good fan out for one server to many clients.  But will transmit to a given IP peer on only one adapter
src_dst_port: uses source and destination port numbers in hash.  Multiple connections between two peers likely hash over different adapters.
(src_dst_port is the best opportunity for multiadapter bandwidth between two peers.  IBM recommendation is hash_mode=src_dst_port, it should be verified at switch as well.)

In src_dst_port mode, the TCP/UDP port values are added, then divided by two. The resultant whole number (no decimals) is plugged into the standard algorithm, then TCP/UDP traffic is sent on the adapter selected by the standard algorithm. Non-TCP or UDP traffic will fall back to the default hash mode, using the last byte of either the destination IP address or MAC address.

The src_dst_port hash mode considers both the source and the destination TCP or UDP port values. In this mode, all of the packets in one TCP or UDP connection are sent over a single adapter (!), so they are guaranteed to arrive in order, but the traffic can be still spread out because additional connections (even to the same host) may be sent over on different adapters.

So, a 2x10Gb LACP channel, for 1 connection will utilize only 1 adapter at a time, and there is no guarantee that the next connection will be dispatched to the other adapter. (Theoretically it can happen that in an LACP channel, adapter1 is 100% utilized, adapter2 0% utilized, and next connection is dispatched again to adapter1, which will cause performance issues.)

Address to ping
Not typically used.  Aids detection for failover to backup adapter. Needs to be a reliable address, but perhaps not the default gateway.  Do not use this on the Link Aggregation, if SEA will be built on top of it.  Instead use netaddr attribute on SEA, and put VIO IP address on SEA interface.

------------------------

How to check which one is configured on the server:


entstat -d <etherchannel device> | head


Etherchannel for Network Interface Backup (primary - standby):
    ETHERNET STATISTICS (ent6) :
    Device Type: EtherChannel
    Hardware Address: 00:11:25:a5:9b:2a
    Elapsed Time: 0 days 12 hours 10 minutes 35 seconds


Etherchannel for Link Aggregation (primary - primary):
    ETHERNET STATISTICS (ent4) :
    Device Type: IEEE 802.3ad Link Aggregation
    Hardware Address: 00:11:25:c5:bf:1d
    Elapsed Time: 45 days 2 hours 37 minutes 55 seconds

------------------------

Link Aggregation with NIB:
By using a backup adapter you can add a single additional link to the Link Aggregation, which is connected to a different Ethernet switch. This single link will only be used as a backup.

------------------------

man ethchan_config            manual for configuring etherchannel with commands
smitty etherchannel           smitty menu for configuring etherchannel
lsattr -El entX               it will show the adapters belongin to the etherchannel (and other useful settings
entstat -d entX | grep Act    shows which adapter (Primary or Backup) is the active in the etherchannel (enX)

------------------------

Creating an EtherChannel:

1. before creating make sure no ip on either adapter
(easiest way: first remove enX/etX/entX before starting to configure, then cfgmgr)
2. when you have clean adapters: smitty etherchannel
    - choose the adapters which you want as primary adapter
    - add additional adapter if needed a backup adapter
    - set mode and hash mode if needed (for Link Aggreagtion once I needed  8023ad and src_dst_port)

------------------------

Add /Remove and adapter from EtherChannel:

Beginning with the 5200-03, Dynamic Adapter Membership functionality is available. This functionality allows you to add or remove adapters from an EtherChannel without having to disrupt any user connections.


Removing adapter from etherchannel without stopping etherchannel:
smitty etherchannel -> Delete Main/Backup Adapter
/usr/lib/methods/ethchan_config -d <etherchannel physical device> <physical device>
/usr/lib/methods/ethchan_config -d ent4 ent0

Adding adapter to etherchannel:
(make sure no ip is configured, if needed put to detach state)
smitty etherchannel -> Add Main/Backup Adapter
/usr/lib/methods/ethchan_config -a -b <etherchannel physical device> <physical device>         <--adding as backup adapter
/usr/lib/methods/ethchan_config -a <etherchannel physical device> <physical device>            <--adding as primary adapter
/usr/lib/methods/ethchan_config -a ent4 ent0
(For me once, only command line worked, when smitty has been used for removing, it always said: resource is busy...)

------------------------

Testing adapter failover:

Etherchannel: ent2
Primary Adapter: ent0 (connected on VLAN with VIOS1)
Backup Adapter: ent1 (connected on VLAN with VIOS2)

(IOCTL error message during creation of an etherchannel can be ignored)

1. check active channel:
    entstat -d ent2| grep Active                    <--Active channel: primary channel

2. unplug cable from primary channel or testing it with command:
    /usr/lib/methods/ethchan_config -f ent2         <--it will do a failover

3. checking:
    entstat -d ent2 | grep Active                   <--Active channel: backup adapter
    errpt | head                                    <--ETHERCHANNEL FAILOVER

4. switch back:   
    /usr/lib/methods/ethchan_config -f ent2         <--it will do a switch again

5. checking:
    entstat -d ent2 | grep Active                   <--Active channel: primary channel
    errpt | head                                    <--ETHERCHANNEL RECOVERY

38 comments:

  1. Wow! Great way to explain it. I was confused and didn't understand clearly the differences between link aggregation and etherchannel until now. Thanks.

    ReplyDelete
  2. Still understanding problem for me...
    ehterchannel & Link agg. are both same ?

    ReplyDelete
    Replies
    1. In a usual etherchannel configuration you have 1 primary and 1 standby (backup) adapter. When you have Link agg. you have 2 primary adapters, so both are functioning at the same time and your network bandwidth is doubled.

      Delete
    2. Thx a lot :-)
      Keep rocking ..If possible please upload screen shots or videos any other new Tech in power series :-)
      Once again Thx...

      Delete
  3. Thanks for providing ethrchannel details and commands.I was searching them and found all in one place !!!!

    ReplyDelete
  4. At first, congratulations for your blog. I visit it many times.
    Do you know what is the best option when creating an etherchannel in a VIOS/VIOC environment, to make the etherchannel in the VIOS or to do it in the VIOC? (only one VIOS, not dual)
    I think that is a cleaner way to do it in the VIOS and link it from the client partition/s, but if you want to do an etherchannel with a lhea and a virtual interface, for example, the only way is configuring it at the client partition, I think. I don't have an scenario to try so it's difficult for my to know the best option between them.

    Thank you very much.

    ReplyDelete
    Replies
    1. In our environment etherchannel is created in VIOS, but we don't use lhea in that. I think you are right in your description...and you should decide which one is fit to your environment.

      Delete
    2. Ok, both of them are running properly so, except the more resource consumption in VIOS or VIOC to mantain the etherchannel, the two options seem to be valid.
      Thanks.

      Delete
  5. We have an IBM p7 server running a full system partition, so all ethernet adapters are dedicated to the full system. We want to configure an etherchannel with four ethernet adapters. The server is a TSM backup server that receives a huge amount of in-bound data which the TSM application pushes out to tape or disk via fibre optic adapters.

    Given that we configure both the p7 hardware side and our network switch side correctly, we would expect to see greatly increased bandwidth to accommodate the inbound data from multiple TSM clients.

    Are we dreaming, or is this a realistic configuration idea that could improve the service between our TSM server and its 200 clients?

    Thanks for your consideration
    Jim T

    ReplyDelete
    Replies
    1. I think it should work. From AIX point of view you can have up to 8 primary Ethernet adapters per Etherchannel. I searched a little bit for this and found this forum, where soemeone wants to do the same: http://unix.ittoolbox.com/groups/technical-functional/ibm-aix-l/etherchannel-only-stays-on-1gb-4502568

      What is important, that a very close cooperation is necessary with network guys to make sure ports on the switch (speed, duplex, Link Aggregation with LACP (mode 8023ad)) are configured properly and exactly the same on AIX.

      I suggest also to read this, where are some tips how to increase bandwidth:
      https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

      Hope this helps,
      Balazs

      Delete
    2. Can I have a 4 nic primary channel and a 4 port backup channel?

      I have 2 4 port cards in my P7.

      I'm thinking 2 aggregations that fail between each other...

      Delete
    3. Hi, this is written by IBM: "You can have up to eight primary Ethernet adapters and only one backup Ethernet adapter per EtherChannel." Some other considerations can be found here: http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.commadmn%2Fdoc%2Fcommadmndita%2Fetherchannel_consider.htm

      Delete
  6. I just got done with making Link Aggregation work across our environment. As said by the Blog Admin you need to work very closely with the Networking Guy to make this work. The adapters making up the Primary Channel connect to the same switch and adapters making up Secondary Channel connect to another. We had these cris-crossed and it never was Aggregated.

    Another important thing that needs to be taken care of is flow_ctrl setting if you are using LHEA's. By default this is set to No and this needs to be YES for Aggregation to work properly. The Network guys at our place have this setting as Auto. So we had to enable flow_ctrl to Yes across LHEA's we had. On the contrary if using a PCI adapter flow_Ctrl is Yes by default.

    Once you have established the etherchannel and Aggregation dont forget to make sure they are indeed Aggregated :)

    entstat -d entX|grep -i Aggr

    And ofcourse this is a great Blog. Thanks for all the hard work. We Unix Admins appreciate it :)

    ReplyDelete
    Replies
    1. Thanks a lot for this feedback and especially for the details about flow_ctrl configuration...I learnt something new today :)

      Delete
  7. thanks for this amazing post.

    Rahul

    ReplyDelete
  8. Father of all AIX blogs...

    ReplyDelete
  9. Hello, I need your help with one of problems which occurs, regarding etherchannel:

    Description:
    Lpar's couldn't communicate or communication occurs but very very low, in the same VLAN's

    Lpar: [oslevel 5.3.0.0 !]
    2 etherchannels: 1 [front-end], 2 [back-end],


    1 etherchannel = VLAN 14 VIO1, VLAN24 VIO2,
    2 etherchannel = VLAN 20 VIO1, VLAN30 VIO2

    VLAN14/24 = 4 other's client
    VLAN20/30 = all client's in the frame.

    VIO [1.4.1.2-FP-9.2 !]
    basic SEA configured without faillover,

    Question:
    what is possibility that this problem occurs? and how can we handle it?

    Thanks in advance, for more details I can post here or we can discuss on private.



    ReplyDelete
    Replies
    1. Hi, first you need a good picture (chart/table...) on a paper about your network setup (LPARS connected to which network/switches/VLAN tags...) after that you need to separate if there is no communication at all or they can communicate but slowly. After this separation has been done check entstat for link status, speed, duplex, transmit errors, dropped packets, invalid vlan tags.... do ping and speed tests meanwhile checking these outputs and figure out where could be the problem.

      Delete
  10. Wonderful...srini

    ReplyDelete
  11. Hi, You said that "Link Aggregation is not a complete high-availability networking solution because all the aggregated links must connect to the same switch"
    but our network guys tell they can make config LACP both ports on differents switch. I am confused also I remember I did it before and it works.
    Thanks

    ReplyDelete
    Replies
    1. Hi Murat, thanks for the feedback! I did not find any reference document why I wrote that sentence, so I removed it from the blog.
      - Balazs

      Delete
  12. this is awesome.. was always confused with link aggregation and etherchannel.. now i understand more better.. thanks so much

    ReplyDelete
  13. I was wondering if there was a difference between the two modes "Standard" and "8023ad" when it comes to the load balancing algorithm used? If LACP is enabled, is it more efficient? thanks a lot

    ReplyDelete
  14. Great Blog!
    I have one question.
    Can we change link agg(active/active) to etherchannel(active/standby) without outage.
    Thanks in advance.

    ReplyDelete
  15. What if LACP is not enabled on core switch.

    ReplyDelete
  16. Hi we have configured etherchannel ent6 which of ent1/ent4 now we have a issue with one of the IO draw becuase of which ent4 is down, After a reboot the interface en6 (etherchannel ) shows defined. As per my understanding if one of the interfaces of etherchannel goes down the other one should be keep the interface up and available ! Please suggest me.

    ServerA# lsdev -l ent7
    ent7 Defined EtherChannel / IEEE 802.3ad Link Aggregation


    ServerA# lsdev -l en7
    en7 Stopped Standard Ethernet Network Interface


    ServerA# lsattr -El ent7|grep ent
    adapter_names ent0,ent3 EtherChannel Adapters True


    ServerA# entstat -d en0|grep -i link
    Link Status : Up


    ServerA# entstat -d en3|grep -i link

    entstat: 0909-003 Unable to connect to device en3, errno = 19



    ServerA# lsdev -Cc adapter
    ent0 Available 18-08 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
    ent1 Available 18-09 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
    ent2 Available 1M-08 10 Gigabit-SR Ethernet PCI-X Adapter (1410ba02)
    ent3 Defined 1N-08 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
    ent4 Defined 1N-09 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
    ent5 Defined 1P-08 10 Gigabit-SR Ethernet PCI-X Adapter (1410ba02)
    ent6 Defined EtherChannel / IEEE 802.3ad Link Aggregation
    ent7 Defined EtherChannel / IEEE 802.3ad Link Aggregation

    ServerA# lsattr -El ent7
    adapter_names ent0,ent3 EtherChannel Adapters True
    alt_addr 0x000000000000 Alternate EtherChannel Address True
    auto_recovery yes Enable automatic recovery after failover True
    backup_adapter NONE Adapter used when whole channel fails True
    hash_mode default Determines how outgoing adapter is chosen True
    mode standard EtherChannel mode of operation True
    netaddr Address to ping True
    noloss_failover yes Enable lossless failover after ping failure True
    num_retries 3 Times to retry ping before failing True
    retry_time 1 Wait time (in seconds) between pings True
    use_alt_addr no Enable Alternate EtherChannel Address True
    use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True



    ServerA# lsattr -El en7
    alias4 IPv4 Alias including Subnet Mask True
    alias6 IPv6 Alias including Prefix Length True
    arp on Address Resolution Protocol (ARP) True
    authority Authorized Users True
    broadcast Broadcast Address True
    mtu 1500 Maximum IP Packet Size for This Device True
    netaddr XX.XX.XXX.XX Internet Address True
    netaddr6 IPv6 Internet Address True
    netmask 255..XXX.XXX.X Subnet Mask True
    prefixlen Prefix Length for IPv6 Internet Address True
    remmtu 576 Maximum IP Packet Size for REMOTE Networks True
    rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True
    security none Security Level True
    state down Current Interface Status True
    tcp_mssdflt Set TCP Maximum Segment Size True
    tcp_nodelay Enable/Disable TCP_NODELAY Option True
    tcp_recvspace Set Socket Buffer Space for Receiving True
    tcp_sendspace Set Socket Buffer Space for Sending True
    ServerA#

    Thanks in Advance

    ReplyDelete
  17. we run ios 2.2.3.3,,with 10g optical cards channeled up; lsattr -EL on Link Aggr device works fine, reports mode (8023ad in our case),
    but entstat -d will not work, always returns entstat: 0909-003 Unable to connect to device ent10, errno = 19
    (or worse) regardless of padmin/oem/regular user. smitty etherchannel (as oem) seems to use lsattr. Any thoughts on an alternate way to list MAC for it?

    ReplyDelete
  18. Answering my own question re MAC of LA device, entstat -all on SEA device reported it, along with a zillion other stats.

    ReplyDelete
  19. need 2 DefaltGateway LPAR AIX 7.1 to 2 different enx, when I change the DG on a card makes me change also in the other adapter.

    as I can do to 2 DG?

    Thank you for your help

    ReplyDelete
  20. This is really good doco! Thanks for the great post!

    ReplyDelete