dropdown menu

Basics - SAN


The SAN fabric is built from interconnecting elements, such as FC hubs, Fc switches, routers, bridges. A SAN fabric connects workstations and computers to storage devices in a storage area network (SAN). SAN fabrics are a set of interconnected fibre channel switches, amongst which data can be physically transmitted. It is similar in concept to a network segment in a local area network. A typical Fibre Channel SAN fabric is made up of a number of Fibre Channel switches.

Storage routers differ from network routers in that the data being used uses storage protocols like FCP instead of TCP/IP.

Fibre Channel hubs are used to connect up to 126 nodes into a logical loop. All connected nodes share the bandwith of this one logical loop. Fibre Channel Arbitrated Loop (FC-AL)protocol is the most widely accepted, cost effective alternative.

Switches used for any-to-any connections: servers, storage systems, other switches, FC-ALs. Zoning for this can be:
    -hardware: port based
    -software: WWN based


Zoning:
Grouping together of multiple ports to form a virtual private storage network. Ports that are memebers of a group or zone can communicate with each other but are isolated from ports in other zones.

------------------------------
lsattr -El fscsi0
scsi_id      0x212500

21: Domain ID (switch id)
25: port id
00: lpar id of vio client (if we have vio environment)
------------------------------

(Fabric refers to a switched network (similar to ethernet) comparison to pt2pt (point to point) and al (arbitrared loop))

In the Fibre Channel switched fabric topology (called FC-SW), devices are connected to each other through one or more Fibre Channel switches. This topology allows the connection of up to the theoretical maximum of 16 million devices


FIBRE CHANNEL TOPOLOGIES:

There are three major Fibre Channel topologies, describing how a number of ports are connected together. A port in Fibre Channel terminology is any entity that actively communicates over the network. This port is usually implemented in a device such as disk storage, an HBA on a server or a Fibre Channel switch.

- Point-to-Point (FC-P2P): Two devices are connected directly to each other. This is the simplest topology, with limited connectivity.

- Arbitrated loop (FC-AL): In this design, all devices are in a loop or ring, similar to token ring networking.
  The failure of one device causes a break in the ring. FC hubs exist to connect devices together and may bypass failed ports.

- Switched fabric (FC-SW): All devices or loops of devices are connected to Fibre Channel switches.
  Similar conceptually to modern Ethernet implementations.

  Advantages of this topology over FC-P2P or FC-AL include:
    - The traffic between two ports flows through the switches only, it is not transmitted to any other port.
    - Failure of a port is isolated and should not affect operation of other ports.
    - Multiple pairs of ports may communicate simultaneously in a fabric.


LAYERS:

Fibre Channel does not follow the OSI model layering, but is split similarly into 5 layers, namely:

FC4 - Protocol Mapping layer, in which application protocols, such as SCSI or IP, are encapsulated into a PDU for delivery to FC2.
FC3 - Common Services layer, a thin layer that could eventually implement functions like encryption or RAID redundancy algorithms;
FC2 - Network layer, defined by the FC-PI-2 standard, consists of the core of Fibre Channel, and defines the main protocols;
FC1 - Data Link layer, which implements line coding of signals;
FC0 - PHY, includes cabling, connectors etc.;


PORTS:

N_port is a port on the node (e.g. host or storage device) used with both FC-P2P or FC-SW topologies. Also known as Node port.
F_port is a port on the switch that connects to a node point-to-point (i.e. connects to an N_port). Also known as Fabric port.
E_port is the connection between two fibre channel switches. Also known as an Expansion port.
EX_port is the connection between a fibre channel router and a fibre channel switch.
(On the side of the switch it looks like a normal E_port, but on the side of the router it is a EX_port.)


11 comments:

  1. Hi Balazs.

    I have a problem and I need ideas to resolve. That is the issue:

    I have this configurations:

    a) 1 power 7 rack server with 1 vios and 1 lpar connected to storage DS5020.
    b) 1 power 7 blade server with 1 vios and 1 lpar connected to storage DS5020.
    c) 1 power 5 rack server with 1 vios and 1 lpar connected to storage DS5020.

    This 3 servers (power7 rack, power 7 blade, power5 rack) are connected to the same storage IBM DS5020.


    Reviewing logs at October 12:

    The storage DS5020 has this message: Logical drive not on prefered path. 14 logical drives was moved to the second path.

    I have errors in 3 VIOS's servers.

    In October 12 at 1:59 am, I lost connection to the virtual disks and the lpar hangs located in the power 7 rack server.

    In October 12 at 1:59 am I lost connection to the virtual disks and the lpar hangs located in the power 5 rack server.

    In October 12 at 1:59 am I lost connection to the virtual disks and the lpar supports the change in the path located in the power 7 BLADE server. Only this lpar survive.


    Reviewing logs at October 15:

    The storage DS5020 has this message: Logical drive not on prefered path. 10 logical drives was moved to the second path.

    I have errors in 3 VIOS's servers.

    In October 15 at 1:59 am, lpar lost connection to the virtual disks and the lpar hangs located in the power 7 rack server.

    In October 15 at 1:59 am, lpar lost connection to the virtual disks and the lpar hangs located in the power 5 rack server.

    In October 15 at 1:59 am, lpar lost connection to the virtual disks and the lpar supports the change in the path located in the power 7 BLADE server. Only this lpar survive and continues working.

    I have the errpt logs if you want to see them.

    Regards.
    Paul.

    ReplyDelete
    Replies
    1. Hi Paul,

      it is the first time I see this kind of problem, so I can just guess that it is related to the SAN network or storage system. In Google I have found this page: http://www-01.ibm.com/support/docview.wss?uid=swg21412057
      To have a proper analysis, I would suggest to open an IBM call (unless someone has a better idea).

      Regards,
      Balazs

      Delete
  2. Hi Balazs

    Tks for your answer.

    I opened a IBM call 1 week ago, but still I don't have response. They only ask for the CASD.

    The only difference that I found is the Vios ioslevel. The ioslevel in the VIOS that continues working is 2.2.0.12-FP24 SP-02. The other 2 Vios that hangs are 2.1.3.10-FP23.

    Do you think that if I upgrade the vios version, I can workaround or resolve this problem?

    Regards
    Paul.



    ReplyDelete
    Replies
    1. Hi Paul,

      90% of the IBM calls, IBM suggests update to the latest level. In the past it happened to me, that strange errors disappeared after an OS update.
      I think, doing an upgrade could be a good approach in this case (and if we are lucky problem will be solved by upgrade.)

      Regards,
      Balazs

      Delete
  3. thank for sharing,
    any recommended redbooks for basics of san and aix ?

    ReplyDelete
  4. My ram is fail how will u replace it.........

    ReplyDelete
  5. FYI, in my environment (NPIV configuration to multiple Netapp SANs, Cisco FCP switches), I find that the third byte of the scsi_id does not correlate to the vio server as identified in the Zoning section. The first byte does correlate to the switch. I did not attempt to confirm the second byte matching to a port number.

    ReplyDelete