iSCSI
iSCSI (Internet SCSI) provides access to storage devices by carrying SCSI commands over a TCP/IP network. iSCSI was developed by IBM and Cisco in 1998 and submitted as a draft standard in March 2000.
NetApp is one of the leader company in storage hardware industry. In the early 1990s, NetApps's storage systems offered NFS and SMB protocols (based on TCP/IP) and in 2002 NetApp added Fibre Channel (FC) and iSCSI protocols. iSCSI protocol is configured to use TCP port number 3260.
The iSCSI protocol allows clients (called initiators) to send SCSI commands to storage devices (targets) on remote servers. It competes with Fibre Channel, but unlike Fibre Channel which usually requires dedicated cabling, iSCSI can be run over long distances using existing network infrastructure.
Differences between iSCSI, NAS and SAN
NAS and iSCSI are both using TCP/IP (LAN), but the protocols are different. NAS is using NFS and CIFS/SMB, but iSCSI is using a variant of the SCSI protocol, so iSCSI has a close relationship with SAN. Actually, only the transmission medium is different, because for SAN, the SCSI protocol is packaged in Fibre Channel, for iSCSI it is packaged in TCP/IP. In both cases, blocks are transferred, iSCSI is therefore not file-based (like NAS), but a block-based transmission, (so we will see hdisk devices in AIX). (Many NAS systems can offer file-based services such as SMB/CIFS and NFS, but also can offer block-based iSCSI.)
--------------------------------------------
Initiator / Target / IQN
To transport SCSI commands over the IP network, an iSCSI driver must be installed, which creates the initiator (the iscsi0 device). In AIX 7.2TL3, MPIO (Multipathing) support for iSCSI software initiator is introduced.
iSCSI targets are devices that respond to iSCSI commands. An iSCSI device can be a storage device, or it can be an intermediate device such as a bridge between IP and Fibre Channel devices.
iSCSI Initiator <--client (LUNs are mapped here), connected to a storage device (via Net. Switch), sends SCSI commands to SCSI target
iSCSI Target <--server (storage system), responds to iSCSI commands and exports local volumes (LUNs) to the initiator node.
Each initiator and target has a unique iSCSI name: iSCSI qualified name (IQN). AnIQN represents a worldwide unique name for each initiator or target in the same way that worldwide node names (WWNNs) are used to identify devices in a Fibre Channel fabric. IQN has a reversed hostname format like: iqn.2018-06.com.ibm.stglabs.aus.p9fwc-iscsi2.hostid.2
Some notes:
Performance considerations:
VG Considerations:
Configure volume groups on iSCSI devices, to be in an inactive state after reboot and manually activate the iSCSI-backed volume groups.Volume groups are activated during a different boot phase than the iSCSI software driver, for this reason, it is not possible to activate iSCSI volume groups during the boot process.
-----------------------------------------------
sanlun lun show short overview about the Netapp LUNs (sanlun command comes with Netapp Utility package)
sanlun lun show -v detailed overview about the Netapp LUNs
sanlun lun show -d hdisk81 -v detailed overview about 1 LUN
sanlun lun show -p shows multipathing info
lsmpio -l hdisk81 shows path details (shows which path using which Controller IP)
lsmpio -ql hdisk81 it is a very good command to check if LUN is really there or not !!!!!!!!!!!!
(it does not use ODM, it makes a real IO operation to reach the LUN and get some details)
lsmpio -Sl hdisk81 shows statistics (Adapter/SCSI errors....)
lsattr -El iscsi0 show settings of iscsi0
lsdev -p iscsi0 show LUNs of iscsi0
lsiscsi show target ip, iqn, port
iscsi_st -f -i 10.10.128.15 show target IQN info and tcp ports (IP is the target IP)
netstat -an | grep 3260 shows which IP is used locally and to which IP it is connected remote (target)
tcpdump -i en1 host 10.10.128.15 check if iscsi packates appear in output (IP is the target IP)
on VIOS check paths for all LUNs which are used as vsscsi devices:
for i in `/usr/ios/cli/ioscli lsmap -all | grep hdisk | awk '{ print $3 }'`; do lsmpio -l $i; done
lspath check of a LUN (use the whole "connection" column for the below command):
lspath -l hdisk81 -HF "name path_id parent connection path_status status"
more info about the path:
lspath -AHE -l hdisk81 -p iscsi0 -w "iqn.1782-08.com.netapp:sn.e1787cbba71411eab2b6d939eb1588a7:vs.29,10.10.128.15,0xcbb,0x50000000000000"
for enabling failed paths:
lspath | grep -v Ena | awk '{print "chpath -s enable -l " $2 " -p " $3 }'
-----------------------------------------------
iSCSI related filesets on AIX (with Netapp Storage):
# lslpp -l | grep -i iscsi
NetApp.MPIO_Host_Utilities_Kit.iscsi Kit iSCSI Disk ODM Stanzas
devices.common.IBM.iscsi.rte 7.2.5.200 APPLIED Common iSCSI Files
devices.iscsi.disk.rte 7.2.5.0 APPLIED iSCSI Disk Software
devices.iscsi.tape.rte 7.2.0.0 COMMITTED iSCSI Tape Software
devices.iscsi_sw.rte 7.2.5.200 APPLIED iSCSI Software Device Driver
devices.pci.14102203.diag 7.2.0.0 COMMITTED IBM 1 Gigabit-TX iSCSI TOE
devices.pci.14102203.rte 7.2.0.0 COMMITTED IBM 1 Gigabit-TX iSCSI TOE
devices.pci.1410cf02.diag 7.2.0.0 COMMITTED 1000 Base-SX PCI-X iSCSI TOE
devices.pci.1410cf02.rte 7.2.0.0 COMMITTED 1000 Base-SX PCI-X iSCSI TOE
devices.pci.1410d002.diag 7.2.4.0 COMMITTED 1000 Base-TX PCI-X iSCSI TOE
devices.pci.1410d002.rte 7.2.0.0 COMMITTED 1000 Base-TX PCI-X iSCSI TOE
devices.pci.1410e202.diag 7.2.0.0 COMMITTED IBM 1 Gigabit-SX iSCSI TOE
devices.pci.1410e202.rte 7.2.0.0 COMMITTED IBM 1 Gigabit-SX iSCSI TOE
devices.pci.77102e01.diag 7.2.0.0 COMMITTED 1000 Base-TX PCI-X iSCSI TOE
devices.pci.77102e01.rte 7.2.0.0 COMMITTED PCI-X 1000 Base-TX iSCSI TOE
-----------------------------------------------
iSCSI configuration
(there are 2 types of dicovery methods: file based or odm based. I used odm based below, for file based add the iscsi target info in ‘/etc/iscsi/targets’ file)
0. for MPIO (at least) 2 IPs should be configured in different subnets (these are our paths and 2 additional IPs are needed from Storage Team at Storage side too)
1. check iscsi drivers and ping target (by default port 3260 is used,not blocked by firewall)
# lslpp -l | grep -i iscsi ; ping <target>
2. set iqn for iscsi0, lsattr -El iscsi0 will show this value (storage team can give the initiator details)
# chdev -l iscsi0 -a initiator_name=iqn.2018-06.com.ibm.my_host1:0ab116bb
3. check discovery policy, I used odm (if needs to be changed: chdev -a disc_policy=odm -l iscsi0)
# lsattr -El iscsi0 | grep disc_policy
4. mkiscsi commands will add iSCSI target data to ODM, for each path needs an mkiscsi command
# mkiscsi -l iscsi0 -g static -t iqn.1986-03.com.ibm:2145.d59-v7k2.node1 -n 3260 -i 10.10.10.10
# mkiscsi -l iscsi0 -g static -t iqn.1986-03.com.ibm:2145.d59-v7k2.node2 -n 3260 -i 10.10.20.10
(the details should be given by storage team, -i iSCSI target IP -t iSCSI target name)
5. discover disks
cfgmgr -vl iscsi0
-----------------------------------------------
Netapp recommendations for iSCSI LUNs
-----------------------------------------------
Reset interface:
If there are problems, resetting the interface used for iscsi may help:
ifconfig en0 down
rmdev -l en0
cfgmgr
For info:
Once we had many disk/io/path errors in errpt, which came hourly/daily and we checked with storage team and they did not find any erros, we did reset but it also did not help.
We asked Network team, and they saw CRC errors on Network switch port, and changing cable/SFP at their side helped.
-----------------------------------------------
No comments:
Post a Comment