dropdown menu

NETWORK - ETHERCHANNEL

ETHERCHANNEL:

EtherChannel is a trademark registered by Cisco Systems and is generally called multi-port trunking or link aggregation. If your Ethernet switch device has this function, you can exploit the support provided in AIX. In this case, you must configure your Ethernet switch to create a channel by aggregating a series of Ethernet ports.

On AIX with "smitty etherchannel", in the same menu, depending how the fields are filled out, 2 distinctly separate functioning devices can be created. This can be seen:

ent2   Available       EtherChannel / IEEE 802.3ad Link Aggregation

1. EtherChannel (Network Interface Backup, NIB)
2. Link Aggregation (802.3ad)

1.Etherchannel (NIB):
Network Interface Backup (NIB) can be used to achieve network redundancy. Two adapters are used to create an EtherChannel that consists of one primary adapter and one backup adapter. The interface is defined on the EtherChannel. If the primary adapter becomes unavailable, the Network Interface Backup switches to the backup adapter.

2. Link Aggregation
Link Aggregation is a network port aggregation technology that allows several Ethernet adapters to be aggregated together to form a single pseudo Ethernet adapter. (There are for example 2 primary adapters).

The main benefit of a Link Aggregation is that it has the network bandwidth of all of its adapters. If an adapter fails, the packets are automatically sent on the next available adapter without disruption to existing user connections. A link or adapter failure will lead to a performance degradation, but not a disruption.


For me once in order to work correctly the Link Aggregation (with LACP) I had to set these:
smitty etherchannel --> Change/Show Characteristics...:
Mode          8023ad
Hash Mode     src_dst_port

(I used smitty, but chdev works as well)

SRIOV and LACP
IEEE802.3ad/802.1ax (LACP) is supported if only one SRIOV logical port is configured for the physical port. The logical port should be configured with a capacity value of 100% to prevent configuration of more than one logical port for the physical port.
------------------------

Etherchannel/Link Aggregation parameters:

Mode:
standard: this is good for default Etherchannel (primary and backup)
8023ad: this is the LACP (Link Aggregation Control Protocol) configuration (2 primary adapters).
(ad = Autodetect: if our server approaches switch with one adapter, switch sees one adapter,  if our server approaches switch with a Link Aggregation, switch auto detects that. For 10Gb, we should be LACP/8023ad.)

Hash Mode
default: is by IP address, good fan out for one server to many clients.  But will transmit to a given IP peer on only one adapter
src_dst_port: uses source and destination port numbers in hash.  Multiple connections between two peers likely hash over different adapters.
(src_dst_port is the best opportunity for multiadapter bandwidth between two peers.  IBM recommendation is hash_mode=src_dst_port, it should be verified at switch as well.)

In src_dst_port mode, the TCP/UDP port values are added, then divided by two. The resultant whole number (no decimals) is plugged into the standard algorithm, then TCP/UDP traffic is sent on the adapter selected by the standard algorithm. Non-TCP or UDP traffic will fall back to the default hash mode, using the last byte of either the destination IP address or MAC address.

The src_dst_port hash mode considers both the source and the destination TCP or UDP port values. In this mode, all of the packets in one TCP or UDP connection are sent over a single adapter (!), so they are guaranteed to arrive in order, but the traffic can be still spread out because additional connections (even to the same host) may be sent over on different adapters.

So, a 2x10Gb LACP channel, for 1 connection will utilize only 1 adapter at a time, and there is no guarantee that the next connection will be dispatched to the other adapter. (Theoretically it can happen that in an LACP channel, adapter1 is 100% utilized, adapter2 0% utilized, and next connection is dispatched again to adapter1, which will cause performance issues.)

Address to ping
Aids detection for failover to backup adapter. Needs to be a reliable address, but perhaps not the default gateway.  Do not use this on the Link Aggregation, if SEA will be built on top of it.  Instead use netaddr attribute on SEA, and put VIO IP address on SEA interface.

------------------------

How to check which one is configured on the server:


entstat -d <etherchannel device> | head


Etherchannel for Network Interface Backup (primary - standby):
    ETHERNET STATISTICS (ent6) :
    Device Type: EtherChannel
    Hardware Address: 00:11:25:a5:9b:2a
    Elapsed Time: 0 days 12 hours 10 minutes 35 seconds


Etherchannel for Link Aggregation (primary - primary):
    ETHERNET STATISTICS (ent4) :
    Device Type: IEEE 802.3ad Link Aggregation
    Hardware Address: 00:11:25:c5:bf:1d
    Elapsed Time: 45 days 2 hours 37 minutes 55 seconds

------------------------

Link Aggregation with NIB:
By using a backup adapter you can add a single additional link to the Link Aggregation, which is connected to a different Ethernet switch. This single link will only be used as a backup.

------------------------

Creating an EtherChannel:

1. before creating make sure no ip on either adapter
(easiest way: first remove enX/etX/entX before starting to configure, then cfgmgr)
2. when you have clean adapters: smitty etherchannel
    - choose the adapters which you want as primary adapter
    - add additional adapter if needed a backup adapter
    - set mode and hash mode if needed (for Link Aggreagtion once I needed  8023ad and src_dst_port)

Create etherchannel with command line (here as ent10 and netaddr is the address to ping):
mkdev -c adapter -s pseudo -t ibm_ech -l ent10 -a adapter_names='ent1' -a backup_adapter='ent3' -a netaddr='10.10.10.1'
/usr/lib/methods/defif                                          <---create the et and en devices (en10, et10)
chdev -l en10 -a netaddr=10.10.10.100 -a netmask=255.255.255.0 -a state=up <---add IP to the etherchannel

------------------------

Check/Add/Remove and adapter from EtherChannel:

Beginning with the 5200-03, Dynamic Adapter Membership functionality is available. This functionality allows you to add or remove adapters from an EtherChannel without having to disrupt any user connections.

man ethchan_config                    <--manual for configuring etherchannel with commands
smitty etherchannel                   <--smitty menu for configuring etherchannel
lsattr -El <eth.ch.>                  <--check eth.ch. config (primary/backup adapter and other parameters)
entstat -d <eth.ch.> | grep Link      <--check eth. ch. Links status (if the adapters are in UP/DOWN state)
entstat -d <eth.ch.> | grep Active    <-check which adapter is active in eth. ch. (primary or backup)


/usr/lib/methods/ethchan_config -a <eth.chan.> entx           <--add entX as primary adapter to eth.chan. (-a: add)
/usr/lib/methods/ethchan_config -a -b <eth.chan.> entx        <--add entX as backup adapter to eth. chan. (-a: add, -b: as backup)

/usr/lib/methods/ethchan_config -d <eth.chan.> entX           <--remove entX from eth.chan. (-d: delete)
/usr/lib/methods/ethchan_config -d ent4 ent0                  <--this will remove from the eth. ch. (which is ent4) the adapter: ent0    

/usr/lib/methods/ethchan_config -f <eth.chan.>                <--initiate a failover of the etherchannel (-f: failover)            
/usr/lib/methods/ethchan_config -c <eth.chan.> num_retries 2  <--changing num_retries parameter to 2 of etherchannel


smitty works as well:
smitty etherchannel -> Add Main/Backup Adapter (make sure no ip is configured, if needed put to detach state)
smitty etherchannel -> Delete Main/Backup Adapter

For me sometimes only command line worked, when smitty has been used for removing an adapter, it always said: resource is busy…

------------------------

Testing adapter failover:

Etherchannel: ent2
Primary Adapter: ent0 (connected on VLAN with VIOS1)
Backup Adapter: ent1 (connected on VLAN with VIOS2)

(IOCTL error message during creation of an etherchannel can be ignored)

1. check active channel:
    entstat -d ent2| grep Active                    <--Active channel: primary channel

2. unplug cable from primary channel or testing it with command:
    /usr/lib/methods/ethchan_config -f ent2         <--it will do a failover

3. checking:
    entstat -d ent2 | grep Active                   <--Active channel: backup adapter
    errpt | head                                    <--ETHERCHANNEL FAILOVER

4. switch back:   
    /usr/lib/methods/ethchan_config -f ent2         <--it will do a switch again

5. checking:
    entstat -d ent2 | grep Active                   <--Active channel: primary channel
    errpt | head                                    <--ETHERCHANNEL RECOVERY

------------------------

LACP check for problems

entstat command will make a best-effort determination of the status of LACP (based on the LACPDUs received from the switch). 
enstat -d entX                               <--check LACP at the device entX

Statistics for every adapter in the IEEE 802.3ad Link Aggregation:
------------------------------------------------------------------
Number of primary adapters: 4
Number of backup adapters: 0
Active channel: primary channel
Operating mode: Standard mode (IEE 802.3ad)
Primary Aggregation status: 
Aggregated     <--LACP has succeeded and the switch has aggregated the adapters together
Failed         <--LACP has failed (maybe adapters have different line speed or duplex mode, or connected to different switches)
                 (or at switch side are some limitations: LACP enabled, or number of adapters, or only contiguous ports are allowed)  
Negotiating    <--LACP is in progress, but the switch has not yet aggregated the adapters  (it should complete less than a minute) 
Inactive       <--LACP is not initiated (Link Aggr. has not yet been configured, no IP assigned or interface has been detached)               
...
Hash mode: Source and destination TCP/UDP ports

IEEE 802.3ad Port Statistics:              <--shows info/details for Actor (AIX side) and Partner (switch side)
-----------------------------
LACP activity: Active or Passive          <--sending LACPDUs always, or only in response to another LACPDU (IEEE 802.3ad Link Aggreg. operates in Active mode)
LACP timeout: Long or Short               <--time to wait before sending LACPDUs (IEEE 802.3ad Link Aggreg. uses the long timeout)
Aggregation: Individual or Aggregatable   <--this port can form an aggregation with other ports, or with itself only
Synchronization: IN_SYNC or OUT_OF_SYNC   <--showsif aggr. reached sync with the partner (IN_SYNC shows our Link Aggr. works good)
Collecting: Enabled or Disabled           <--show if Link Aggreg. is collecting (receiving) packets
Distributing: Enabled or Disabled         <--show if Link Aggreg. is distributing (sending) packets
Defaulted: True or False                  <--show if Link Aggreg. is using default values for the partner's information
Expired: True or False                    <--show if Link Aggregation is operated in expired mode

As an info: 
Once we tried to aggregate SRIOV Logigal Ports (shows VF on AIX) using LACP, but we could not reach the IN_SYNC status. We checked many things with Network team, and after found out we used VLAN tagging at the SRIOV port, and after removing VLAN tags LACP worked good. 

47 comments:

Anonymous said...

Wow! Great way to explain it. I was confused and didn't understand clearly the differences between link aggregation and etherchannel until now. Thanks.

aix said...

You're welcome :)

rAmpAge said...

Really nice.

Anonymous said...

Still understanding problem for me...
ehterchannel & Link agg. are both same ?

aix said...

In a usual etherchannel configuration you have 1 primary and 1 standby (backup) adapter. When you have Link agg. you have 2 primary adapters, so both are functioning at the same time and your network bandwidth is doubled.

Anonymous said...

Thx a lot :-)
Keep rocking ..If possible please upload screen shots or videos any other new Tech in power series :-)
Once again Thx...

Anonymous said...

Thanks for providing ethrchannel details and commands.I was searching them and found all in one place !!!!

aix said...

Welcome :-)

Anonymous said...

At first, congratulations for your blog. I visit it many times.
Do you know what is the best option when creating an etherchannel in a VIOS/VIOC environment, to make the etherchannel in the VIOS or to do it in the VIOC? (only one VIOS, not dual)
I think that is a cleaner way to do it in the VIOS and link it from the client partition/s, but if you want to do an etherchannel with a lhea and a virtual interface, for example, the only way is configuring it at the client partition, I think. I don't have an scenario to try so it's difficult for my to know the best option between them.

Thank you very much.

aix said...

In our environment etherchannel is created in VIOS, but we don't use lhea in that. I think you are right in your description...and you should decide which one is fit to your environment.

Anonymous said...

Ok, both of them are running properly so, except the more resource consumption in VIOS or VIOC to mantain the etherchannel, the two options seem to be valid.
Thanks.

Anonymous said...

We have an IBM p7 server running a full system partition, so all ethernet adapters are dedicated to the full system. We want to configure an etherchannel with four ethernet adapters. The server is a TSM backup server that receives a huge amount of in-bound data which the TSM application pushes out to tape or disk via fibre optic adapters.

Given that we configure both the p7 hardware side and our network switch side correctly, we would expect to see greatly increased bandwidth to accommodate the inbound data from multiple TSM clients.

Are we dreaming, or is this a realistic configuration idea that could improve the service between our TSM server and its 200 clients?

Thanks for your consideration
Jim T

aix said...

I think it should work. From AIX point of view you can have up to 8 primary Ethernet adapters per Etherchannel. I searched a little bit for this and found this forum, where soemeone wants to do the same: http://unix.ittoolbox.com/groups/technical-functional/ibm-aix-l/etherchannel-only-stays-on-1gb-4502568

What is important, that a very close cooperation is necessary with network guys to make sure ports on the switch (speed, duplex, Link Aggregation with LACP (mode 8023ad)) are configured properly and exactly the same on AIX.

I suggest also to read this, where are some tips how to increase bandwidth:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

Hope this helps,
Balazs

Fayez Mohammed said...

I just got done with making Link Aggregation work across our environment. As said by the Blog Admin you need to work very closely with the Networking Guy to make this work. The adapters making up the Primary Channel connect to the same switch and adapters making up Secondary Channel connect to another. We had these cris-crossed and it never was Aggregated.

Another important thing that needs to be taken care of is flow_ctrl setting if you are using LHEA's. By default this is set to No and this needs to be YES for Aggregation to work properly. The Network guys at our place have this setting as Auto. So we had to enable flow_ctrl to Yes across LHEA's we had. On the contrary if using a PCI adapter flow_Ctrl is Yes by default.

Once you have established the etherchannel and Aggregation dont forget to make sure they are indeed Aggregated :)

entstat -d entX|grep -i Aggr

And ofcourse this is a great Blog. Thanks for all the hard work. We Unix Admins appreciate it :)

aix said...

Thanks a lot for this feedback and especially for the details about flow_ctrl configuration...I learnt something new today :)

ra said...

thanks for this amazing post.

Rahul

Anonymous said...

Father of all AIX blogs...

aix said...

:-)

Ty Haller said...

Can I have a 4 nic primary channel and a 4 port backup channel?

I have 2 4 port cards in my P7.

I'm thinking 2 aggregations that fail between each other...

aix said...

Hi, this is written by IBM: "You can have up to eight primary Ethernet adapters and only one backup Ethernet adapter per EtherChannel." Some other considerations can be found here: http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.commadmn%2Fdoc%2Fcommadmndita%2Fetherchannel_consider.htm

Anonymous said...

Hello, I need your help with one of problems which occurs, regarding etherchannel:

Description:
Lpar's couldn't communicate or communication occurs but very very low, in the same VLAN's

Lpar: [oslevel 5.3.0.0 !]
2 etherchannels: 1 [front-end], 2 [back-end],


1 etherchannel = VLAN 14 VIO1, VLAN24 VIO2,
2 etherchannel = VLAN 20 VIO1, VLAN30 VIO2

VLAN14/24 = 4 other's client
VLAN20/30 = all client's in the frame.

VIO [1.4.1.2-FP-9.2 !]
basic SEA configured without faillover,

Question:
what is possibility that this problem occurs? and how can we handle it?

Thanks in advance, for more details I can post here or we can discuss on private.



aix said...

Hi, first you need a good picture (chart/table...) on a paper about your network setup (LPARS connected to which network/switches/VLAN tags...) after that you need to separate if there is no communication at all or they can communicate but slowly. After this separation has been done check entstat for link status, speed, duplex, transmit errors, dropped packets, invalid vlan tags.... do ping and speed tests meanwhile checking these outputs and figure out where could be the problem.

Anonymous said...

Wonderful...srini

Murat said...

Hi, You said that "Link Aggregation is not a complete high-availability networking solution because all the aggregated links must connect to the same switch"
but our network guys tell they can make config LACP both ports on differents switch. I am confused also I remember I did it before and it works.
Thanks

aix said...

Hi Murat, thanks for the feedback! I did not find any reference document why I wrote that sentence, so I removed it from the blog.
- Balazs

Anonymous said...

this is awesome.. was always confused with link aggregation and etherchannel.. now i understand more better.. thanks so much

Anonymous said...

I was wondering if there was a difference between the two modes "Standard" and "8023ad" when it comes to the load balancing algorithm used? If LACP is enabled, is it more efficient? thanks a lot

Unknown said...

Great Blog!
I have one question.
Can we change link agg(active/active) to etherchannel(active/standby) without outage.
Thanks in advance.

Abdul Rauf (Abidi) said...

What if LACP is not enabled on core switch.

Anonymous said...

Hi we have configured etherchannel ent6 which of ent1/ent4 now we have a issue with one of the IO draw becuase of which ent4 is down, After a reboot the interface en6 (etherchannel ) shows defined. As per my understanding if one of the interfaces of etherchannel goes down the other one should be keep the interface up and available ! Please suggest me.

ServerA# lsdev -l ent7
ent7 Defined EtherChannel / IEEE 802.3ad Link Aggregation


ServerA# lsdev -l en7
en7 Stopped Standard Ethernet Network Interface


ServerA# lsattr -El ent7|grep ent
adapter_names ent0,ent3 EtherChannel Adapters True


ServerA# entstat -d en0|grep -i link
Link Status : Up


ServerA# entstat -d en3|grep -i link

entstat: 0909-003 Unable to connect to device en3, errno = 19



ServerA# lsdev -Cc adapter
ent0 Available 18-08 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
ent1 Available 18-09 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
ent2 Available 1M-08 10 Gigabit-SR Ethernet PCI-X Adapter (1410ba02)
ent3 Defined 1N-08 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
ent4 Defined 1N-09 2-Port Gigabit Ethernet-SX PCI-X Adapter (14108802)
ent5 Defined 1P-08 10 Gigabit-SR Ethernet PCI-X Adapter (1410ba02)
ent6 Defined EtherChannel / IEEE 802.3ad Link Aggregation
ent7 Defined EtherChannel / IEEE 802.3ad Link Aggregation

ServerA# lsattr -El ent7
adapter_names ent0,ent3 EtherChannel Adapters True
alt_addr 0x000000000000 Alternate EtherChannel Address True
auto_recovery yes Enable automatic recovery after failover True
backup_adapter NONE Adapter used when whole channel fails True
hash_mode default Determines how outgoing adapter is chosen True
mode standard EtherChannel mode of operation True
netaddr Address to ping True
noloss_failover yes Enable lossless failover after ping failure True
num_retries 3 Times to retry ping before failing True
retry_time 1 Wait time (in seconds) between pings True
use_alt_addr no Enable Alternate EtherChannel Address True
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True



ServerA# lsattr -El en7
alias4 IPv4 Alias including Subnet Mask True
alias6 IPv6 Alias including Prefix Length True
arp on Address Resolution Protocol (ARP) True
authority Authorized Users True
broadcast Broadcast Address True
mtu 1500 Maximum IP Packet Size for This Device True
netaddr XX.XX.XXX.XX Internet Address True
netaddr6 IPv6 Internet Address True
netmask 255..XXX.XXX.X Subnet Mask True
prefixlen Prefix Length for IPv6 Internet Address True
remmtu 576 Maximum IP Packet Size for REMOTE Networks True
rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True
security none Security Level True
state down Current Interface Status True
tcp_mssdflt Set TCP Maximum Segment Size True
tcp_nodelay Enable/Disable TCP_NODELAY Option True
tcp_recvspace Set Socket Buffer Space for Receiving True
tcp_sendspace Set Socket Buffer Space for Sending True
ServerA#

Thanks in Advance

Anonymous said...

we run ios 2.2.3.3,,with 10g optical cards channeled up; lsattr -EL on Link Aggr device works fine, reports mode (8023ad in our case),
but entstat -d will not work, always returns entstat: 0909-003 Unable to connect to device ent10, errno = 19
(or worse) regardless of padmin/oem/regular user. smitty etherchannel (as oem) seems to use lsattr. Any thoughts on an alternate way to list MAC for it?

Anonymous said...

Answering my own question re MAC of LA device, entstat -all on SEA device reported it, along with a zillion other stats.

NETWORK_AIX said...

need 2 DefaltGateway LPAR AIX 7.1 to 2 different enx, when I change the DG on a card makes me change also in the other adapter.

as I can do to 2 DG?

Thank you for your help

Walangkopok said...

This is really good doco! Thanks for the great post!

Unknown said...

Thanks for this amazing post.

aix said...

:)

Anonymous said...

I have an issue where I created EtherChannel with - ent0 ent1 using mkvdev -lnagg ent0 ent1..... then Configure SEA on it. It was running fine for few years. Now the Network team got new CISCO switch, we are having all kind of problems. Most likely their switch is not configured for LACP, I think. So what I want to do is remove the one adapter port from the EtherChannel (ent0 or ent1) then add it back to EtherChannel as a backup adapter. Do think that is an option? I am thinking that way I will not have a problem with link aggregation of two adapters which probably will resolve the LACP issue on the Cisco side and I will still have some redundancy because other port will be the backup port, in case the something happens to switch or port. Could you please share your opinion? I am also looking some articles about settings hash_mode to src_dst_port and change it from standard to 8023AD. Can you suggest a solution ?

Unknown said...

My co worker Luis scripted this setup-lb ip ent1, ent2 primary.
It was also tcpdump used to get to make sure cards are connected to right subnet. You know network guys never plug the cables to the right correct vlan.
Great reading thanks .
Thanks, below also good reading.
http://www-01.ibm.com/support/docview.wss?uid=isg3T1011763

aix said...

Nice to hear :)

James said...

Struggling with etherchannel configuration, we have two AIX 7.1 servers (Primary and Backup), so want each to have an admin IP and both will have an application IP configured, but only running on one server at a time (other cables will be unplugged).
Primary server has en8 configured with primary en0 backup en4 (connected to seperate switches) on xx.xxx.xxx.51 (admin IP)
Primary server has en9 configured with primary en1 backup en5 (connected to seperate switches) on xx.xxx.xxx.53 (application IP)
Backup server has en8 configured with primary en0 backup en4 (connected to seperate switches) on xx.xxx.xxx.52 (admin IP)
Backup server has en9 configured with primary en1 backup en5 (connected to seperate switches) on xx.xxx.xxx.53 (application IP) with both network cables unplugged
I can ping/telnet to either server from other subnets, but from .51 server cannot ping .52 unless I ifconfig en9 down ?
I cannot find anything to say two etherchannels cannot have different IPs on the same subnet, and this seems a weird restriction, is there a way to configure the etherchannel to achieve this ?
All I want is to have two resilient network connections on two IPs, so if there is a different/better way to do this please advise.
Thanks.

James said...

Just to update, while trying crazy things, I've changed the IP of Backup server en9 to a spare address (xx.xxx.xxx.62) which made no difference, but plugging in the network cables to Backup server ent1/ent5 seems to fix this ???
It seems that even if etherchannel is set down, it still cannot cope without the cables connected ?

aix said...

Thanks James for this info :)

Unknown said...

Hello Team,
I have requirement of the new network design .. I have 2 ethernet cards with 10-GBPS speed and 4 ports each . 2 ports are for data and 2 for management links. I have designed initial diagram with my requirements

How can I create the aggregated link between 2 ethernet cards with dual VIOS?

aix said...

If you have 2 cards and 2 VIOS, and you will add 1 card to 1 VIOS and the other card to the other VIOS, you cannot create link aggregation between the cards (because the cards are not in the same OS). You can create link aggregation between ports in 1 card. (If you use SEA, this setup would be also redundant, because if 1 card or VIOS fails, the other VIO can provide network traffic.)

Unknown said...

Thank you for the reply , I have another followup question
1. If any port on one card fails , will the adapter will take the another port from another card?
2. Could you please provide the steps for creating the link aggreation and using it creating the SEA adapter?
3.Also we need to setup new VLAN , could you please provide that steps as well?

Anonymous said...

Looks like a typo enstat instead of entstat:

LACP check for problems

entstat command will make a best-effort determination of the status of LACP (based on the LACPDUs received from the switch).
enstat -d entX

Scott said...

If you've ever fallen victim to any crypto scam, you can hire a credible recovery agency to recover your assets. However, while trying to recover your stolen crypto, there is need to ensure that your security is paramount while dealing with the recovery company. Hence the need to consider an option that ensures your security, simultaneously recovering your money, and subsequently protect you from future encounters. You can reach out to easybinarysolutions@gmail .com