 A
Fibre Channel Primer: Part 2
W. Curtis Preston
In November's column, I discussed a lot of the basics of
Fibre Channel, including:
- Why Fibre Channel was developed
- Who makes up the Fibre Channel "industry"
- What is Fibre Channel?
- What are the different types of Fibre Channel ports?
- How are Fibre Channel devices addressed?
In this column, I will build on that information and explain the
different Fibre Channel architectures: point-to-point, fabric, and
arbitrated loop. In case you don't have access to last month's
column, let's review some important information.
There are three "basic" types of ports, the N_Port,
the F_Port, and the E_Port. As you add arbitrated loop capabilities
to these basic ports they take on the "combined" names
of: NL_Port, FL_Port, and G_Port, which are the most common types
of ports. An NL_Port is a node (device/host) port with arbitrated
loop capabilities. An FL_Port is a fabric port on a switch with
arbitrated loop capabilities. A G_Port is a multi-purpose port on
a switch that can do fabric, arbitrated loop, or expansion to another
switch.
Another term that will be used in this article is HBA, or Host
Bus Adaptor. This is the industry term for a Fibre Channel interface
card that plugs into a host, such as a PCI or SBUS card.
A unique, 64-bit address is assigned to each port, and is referred
to as its WorldWide Name (WWN). If a port connects to an arbitrated
loop, it will also be assigned a dynamic 8-bit address, referred
to as its arbitrated loop physical address, or AL_PA. If it connects
to a fabric, it will be assigned a dynamic 24-bit address, referred
to as its Native Address Identifier. When a port is connected to
both an arbitrated loop and a fabric, the lowest eight bits of the
Native Address Identifier become the AL_PA.
Now that I've reviewed the terms, I will explain the different
Fibre Channel topologies: point-to-point, fabric, and arbitrated
loop.
Point-to-Point
A point-to-point Fibre Channel network is the simplest and least
expensive of the three topologies; it is simply two N_Ports communicating
via a point-to-point connection. As seen in Figure 1, a Fibre Channel
array connected to a host is an example of a point-to-point connection.
Fabric
An illustration of a basic fiber network can be found in Figure
2. In a true fabric-only environment, each N_Port would plug into
one F_Port on the switch. Each port is assigned its Native Address
Identifier (S_ID) by the switch when it "logs into" the
fabric. This 24-bit address allows for up to 16 million unique addresses
within a single fabric, which should be more than enough for even
the biggest SANs. (At this point, I can't possibly imagine
a SAN with 16 million nodes on it, but then I keep thinking about
what the popularity of the Internet has done to the original IP
specification.) When an N_Port is connected to a switch, it and
other N_Ports connected to that switch can utilize the entire bandwidth
of the connection to which they are attached. Just as four nodes
in a 100-Mb switched Ethernet environment can hold two simultaneous
100-Mb/s "conversations," every port in a Fibre Channel
switch should be able to supply the connected node with as much
bandwidth as it needs, up to 100 Mb/s.
Of course, the switch in Figure 2 could connect to other switches
via its E_Port, allowing you to grow the network as much as you
needed to, up to 16 million nodes. Unfortunately, the per-port cost
of building a completely fabric-only environment would be quite
high.
Arbitrated Loop
The Fibre Channel arbitrated loop (FC-AL) was actually an add-on
to the original Fibre Channel specification, which included only
point-to-point and fabric topologies. However, the per-port cost
of the fabric topology was prohibitively high, so the arbitrated
loop topology was offered as a topology without the limits of point-to-point
or the cost of fabric. Now FC-AL is the most dominant Fibre Channel
topology, so much so that many people refer to all Fibre Channel
as "f-kal" (a common pronunciation of FC-AL). FC-AL's
dominance may disappear now that the per-port cost of fabric is
coming down.
Arbitrated loops can appear in two physical layouts. As seen in
Figure 3, a "true" arbitrated loop starts with several
NL_Ports (either hosts or storage devices) configured in an actual
loop, which requires splitting the transmit and receive wires of
the connecting cables. Most Fibre Channel equipment is not designed
to be used this way, but it can be done with fiber optic cables
that have splittable SC connectors, as illustrated in Figure 4.
True loops (as illustrated in Figures 3 and 4) typically work
only with short distances due to logistical problems, but can be
an inexpensive way to connect several Fibre Channel disk arrays
to a single host. Another major limitation of this physical layout
is that one bad HBA can take the entire network out, since each
HBA creates part of the loop. Therefore, the most common FC-AL physical
layout is the "star" layout seen in Figure 5.
An FC-AL star layout is accomplished by connecting each NL_Port
to a Fibre Channel hub, and is electrically and logically the same
as the loop layout in Figures 3 and 4, without forcing you to split
the transmit and receive wires as shown in Figure 4. As you can
see in Figure 6, the hub does the work of creating the loop.
FC-AL vs. Fabric
Now that I've explained why it is called a loop, why is it
called an arbitrated loop? The answer to that question is actually
one of two main differences between arbitrated loop and fabric.
First, nodes on a loop select their address, rather than having
it assigned by a switch. Second, FC-AL is a shared medium, in which
nodes that wish to transmit on the loop must arbitrate for the right
to do so. You will remember that when a node connects to a fabric,
its address is assigned by the switch. In an arbitrated loop, however,
the address is selected by each port from a list of available addresses.
This is done in several steps:
1. The first step in loop initialization is when an L_Port (either
an NL_Port or an FL_Port) transmits a Loop Initialization Primitive
(LIP) frame, which is a special type of frame transmitted during
loop failure or when a node powers on. This causes every other port
to also transmit an LIP frame, at which point the loop becomes unusable
until all frames have received the Close (CLS) frame, which will
be transmitted later.
2. Once they have transmitted the LIP frame, they need to select
a loop master, which will be very important in the next step of
initialization. This is done by continually transmitting the Loop
Initialization Select Master (LISM) frame. If the loop is connected
to a fabric (via an FL_Port, as discussed later in this article),
the fabric will become the loop master. If not, the port with the
lowest Port Name will be chosen as loop master.
3. The next step is that every node must select an AL_PA. The
loop master sends a Loop Initialization Select AL_PA (LISA) frame
"around the loop." Each L_Port on the loop selects a free
AL_PA, sets its AL_PA to that value, and changes that AL_PA's
"free" status in the LISA frame so that other L_Ports
will not select it. It then passes the LISA frame on to the next
L_Port in the loop. (In the case of a re-initialization of a previously
initialized loop, a particular port that already had an AL_PA assigned
will attempt to select the same AL_PA again. If that fails, it will
select a new one.)
4. Once the LISA frame has returned to the loop master, it sends
the Close (CLS) frame to notify all nodes that the initialization
process has completed. At this point, the loop is ready to be used.
The second main difference between FC-AL and fabric is that FC-AL
is a shared medium. Only one port may communicate at once. Ports
that wish to transmit data must arbitrate for the right to do so.
The arbitration process can be quite complex, so the following should
be considered a summary. A node begins arbitration by transmitting
the ARB (arbitrate) frame. If no other ports are communicating,
the ARB frame will be received by the port that transmitted it,
causing it to "win" arbitration and begin communicating.
If two or more ports are arbitrating simultaneously, the port with
the lowest AL_PA will win the arbitration.
Whenever a device wins arbitration, its "access variable"
is set to 0. As long as this access variable is set to 0, the device
cannot arbitrate, and if it can't arbitrate, it can't
win arbitration. This is what allows ports with lower priority (higher
AL_PAs) to eventually win arbitration and be able to transmit. This
and other types of logic built into FC-AL constitutes what is called
the fairness algorithm.
One of the reasons that the fairness algorithm is so important
is that once a port wins arbitration, it can transmit data until
it is finished. This is different from a token ring network, where
the "token" must be passed on after a set period of time.
While the "winning" port has control of the loop, obviously
other ports may wish to arbitrate, but will be told to "wait"
by the current arbitration winner until it is done. Once the winning
FC-AL port completes its "transaction," it yields control
of the loop to others. The port with the next highest priority that
has been trying to arbitrate while the previous device had control
of the loop will immediately win arbitration, set its access variable
to 0, and begin transmitting.
This process continues until all devices that were trying to talk
(while other devices had control of the loop) are allowed to talk.
Once all devices have been allowed to talk, the last one to talk
transmits an IDLEs frame, telling other ports that the loop is idle.
This causes them to set their access variable to 1, and the process
starts all over again. Without getting into detail on the different
types of frames that are sent, I'll give an example with a
loop of three devices, with AL_PAs of 1,2, and 3:
1. Ports 1 and 2 try to arbitrate at the same time by sending
the ARB frame. Port 1 wins, since its AL_PA is lowest.
2. Port 1 sets its access variable to 0 and begins transmitting.
3. While Port 1 is transmitting, Port 2 continues to try to arbitrate.
It is told by Port 1 to "wait."
4. While Port 2 is waiting, Port 3 also tries to transmit. It
is also told to wait.
5. Port 1 completes transmitting, at which point, Port 2 will
win arbitration since it is the port with the lowest AL_PA that
is awaiting arbitration.
6. While Port 2 is transmitting, Port 3 continues to wait.
7. Port 1 now has something else to transmit, but it is not allowed
to arbitrate because its access variable is set to 0. Therefore,
it must wait.
8. Port 2 completes transmitting, at which point, Port 3 wins
arbitration, since it is the only port that has requested arbitration
at this point.
9. Port 3 sets its access variable to 0 and begins its transmission.
10. When Port 3 completes its transmission, it will "notice"
that there are no other ports awaiting arbitration. It sends the
IDLEs frame, which notifies Port 1 that the loop is free.
11. Port 1 now begins and wins arbitration again.
Combining Fabric and Arbitrated Loop Topologies
Arbitrated loop is inexpensive, but has a physical limitation
of 126 devices and a practical limitation of even less than that.
Fabric can expand up to 16 million devices, but is extremely expensive.
Therefore, it is very common to combine several small arbitrated
loops via fabric. The network shown in Figure 7 gives you the best
of both worlds.
This is done by connecting an FL_Port or G_Port on a fabric switch
to an arbitrated loop, typically via a hub as shown in Figure 7.
Remember that all devices in a fabric are assigned a 24-bit address,
and the lowest 8 bits of that address become a device's AL_PA,
if it's a member of an arbitrated loop. One could think of
the AL_PA as the last octet of an IP address, where the higher 16
bits of the 24-bit address show which arbitrated loop a device is
on, and the lower 8 bits show its address within that loop.
W. Curtis Preston is a principal consultant at Collective Technologies
(http://www.colltech.com), and has specialized in designing
and implementing enterprise backup systems for Fortune 500 companies
for more than 7 years. Portions of this article are excerpted from
his O'Reilly book UNIX Backup & Recovery. Curtis
may be reached at: curtis@backupcentral.com.
| 
