This past month, the FC-BB-5 working group of the T11 Technical Committee completed its work on the FCoE standard and approved the final standard. This vote stems back to November when a letter ballot for approval resulted in more than two-thirds votes in favor of the standard. Four members, however, had cast opposing votes. The committee continued work on the standard, addressing comments from both dissenting and affirmative ballots. As a result, during its plenary meeting on June 4, the T11 unanimously approved to forward the standard to INCITS for further processing and public review with no dissenting votes.

The goal of FCoE is to enable LAN and SAN data convergence in data centers and simplify the network by bringing multiple technologies under the umbrella of a unified interface. Instead of separating over their differences or overriding the concerns of a minority of members, the drafting committee has worked towards a resolution that addresses the issues of all members concerned. The solidarity of the standard’s approval is a welcome portent of the success of FCoE.

After completing their work on converging LAN and SAN has been done, T11 is now focusing on IPC traffic for cluster computing. During the week of T11 June meeting, a new protocol proposal, RDMA over FCoE, was presented for discussion. The goal of this new protocol is to maintain high data transmission performance by taking advantage of low overhead structure of Fibre Channel. In the meantime, IO convergence is to be achieved by FCoE protocol. If this is successful, we may finally see the end of InfiniBand.

Certainly there is much work left to be done to finalize FCoE as a robust standard capable of enabling true network convergence. However, with this announcement, another milestone has been passed on the road to FCoE, and we are another step closer to achieving true network convergence in data centers.

The final of the four testing challenges developers of FCoE network equipment face when verifying equipment behavior, performance, and responsiveness is being able to not only test both the LAN and SAN components of the network simultaneously but to also combine test results and expertise onto a single, converged platform.

Currently, the LAN and SAN are separate networks which have their own tools and different testing focuses. For example, LAN testing focuses on the switch and network only. SAN testing, in contrast, requires network-to-end-device verification as all hosts, fabrics, and targets contribute to the overall performance of the network. Given that the LAN fabric is based on best effort delivery while the SAN has complete traffic control end-to-end governed by Fibre Channel’s protocol for zero frame loss, LAN QoS testing does not apply to storage networks. Rather, SAN testing measures the performance of Fibre Channel links through mechanisms such as buffer to buffer credit management from the host through the fabric to the storage. In addition, as SAN testing must verify the delivery of every single frame, it requires a wholly different dimension of performance and latency measurements compared to LAN testing.

Convergence of the LAN and SAN networks, however, needs to extend beyond simply carrying traffic out to all aspects of network management and testing. As the LAN and SAN become a single logical network under FCoE, analysis and monitoring tools need to accommodate this convergence as well, enabling developers to utilize a single test and development platform.

A common misconception is that as the network has converged on Enhanced Ethernet, converged testing should follow Ethernet’s traditional QoS-based structure. However, since all the new, enhanced features are designed to enable storage data to be converged with LAN data with performance comparable to a separate SAN, testing must actually focus on the SAN’s more stringent requirements. Thus, when designing a test setup for a converged Enhanced Ethernet network, developers must step away from QoS-based testing, which is a measure of lossy network performance, in order to focus on performance testing of a lossless network with zero tolerance for packet drop.

The challenge of testing Enhanced Ethernet is how to combine the expertise required for testing lossy traffic with that required for testing lossless traffic. As the testing focus of these two different types of traffic are difficult to reconcile, the analysis instrumentation itself must be integrated as a converged platform. In this way, it becomes possible to test the various convergence properties such as the priority groups and traffic classes as well as the influences these have on the converged network as a whole.

As I discussed previously, developers of FCoE network equipment face four key challenges when verifying equipment behavior, performance, and responsiveness.  The third testing challenge developers must address lies in testing long-term network congestion management mechanisms.

When performing conventional Ethernet LAN analysis, most tests involve only the switches as the QoS performance of the LAN network is mainly determined by them. FCoE developers may be tempted to employ fabric-only tests which utilize traffic generation tools to simulate the majority of nodes “operating” over the network.  Such fabric-only based testing, however, only exercises a subset of the Enhanced Ethernet capabilities that connect an FCoE network. 

For example, certain Data Center Bridging (DCB) protocols – such as congestion notification (CN) – require that all devices in a link participate in network management.  To comprehensively test CN mechanisms, then, developers must be able to create long-term congestion situations, monitor the resulting CN messages generated, and verify how end stations react to each CN message.

Useful inline test tools include protocol analyzers, error injectors (Jammers), and delay emulators. Delay Emulation allows developers to shape traffic bandwidth per priority, re-order frames, and emulate distance delays. For full control of traffic, Delay Emulation technology can be combined with Analyzer and Jammer functions to enable testing of extremely complex networks, including full CN message testing.

Continuing my discussion of the four key testing challenges developers of FCoE network equipment must face, the second challenge is the need for hardware-based protocol tools. With their high throughput and advanced capabilities, FCoE and Enhanced Ethernet technology have created the new need for hardware-based test tools.

Software-based analysis tools already struggling to evaluate Gigabit Ethernet will provide significantly degraded performance at 10 Gb/s.  Rather than providing 100% visibility into network interactions, software-based analyzers require developers to pare down what information is collected and made available for offline analysis.  Identifying problems is often the most difficult part of troubleshooting, especially when working with new protocols like FCoE with which developers are still becoming familiar.  Working with limited information obscures problems and complicates troubleshooting.  This is especially true for SAN analysis where it is almost impossible to debug issues using broken or incomplete traces. 

Another complication that arises is that all Data Center Bridging (DCB) protocols communicate at the lower data layers, making them inaccessible to software analysis tools.  To troubleshoot DCB issues requires hardware-based access in order to guarantee accurate, 100% capture at line-rate.  In addition, end node SPAN ports have a limitation of 10 GE because their aggregation mechanism requires higher bandwidth such as 40 GE, a technology which is still under development.  Only hardware-based protocol analyzers that guarantee100% line-rate analysis of everything on the wire can effectively capture all the necessary data to facilitate efficient network debugging and troubleshooting.

As I discussed last time, developers of FCoE network equipment face four key challenges when verifying equipment behavior, performance, and responsiveness. The first challenge is working with the multi-protocol nature of FCoE.

As a converged network technology, FCoE involves significantly more than simply implementing a new protocol. Because the FCoE network connects the FC SAN to the Ethernet network, the entire network is effectively involved in the mixed transportation of protocols. Developers must perform end-to-end analysis of mixed data across 10 Gigabit Ethernet and Fibre Channel links to ensure that these protocols are handled properly, accurately, quickly, and reliability.

To achieve this, developers must be able to capture and analyze multiple protocols at the same time. Note that the list of possible protocols involved may be quite large – FCoE, FC, Enhanced Ethernet, iSCSI, etc. – and that problems may arise at any transition, cross over, or end point. Test setups which can only monitor a few protocols due to the limitation of available interfaces prevent developers from comprehensively evaluating and testing FCoE capabilities on real-world FCoE network topologies.

Another critical element of multi-protocol analysis is the ability to accurately timestamp packets and frames. Unless traffic from different protocols can be correlated within the same time domain, developers will have great difficulty in tracking particular exchanges as they cross protocol domains. Without this capability, developers will be severely curtailed in their ability to perform multi-protocol FCoE analysis.

Designing for FCoE goes far beyond simply implementing new network protocols. As a converged network technology, FCoE brings together multiple protocols in addition to introducing new capabilities through Enhanced Ethernet. In order to accurately verify compliance, evaluate performance, and guarantee reliability, developers must create comprehensive test setups and scenarios that confirm traffic integrity as data crosses protocol domains. As a consequence, testing must encompass a variety of operating conditions that are more complex to verify than when working with the LAN and SAN solely. For example, FCoE packets possess two CRC values – one for the encapsulated FC frame and one for the Enhanced Ethernet packet.

Finisar has identified four specific testing challenges that developers must address when designing multi-protocol, FCoE-based equipment:

1) As an FCoE network is comprised of multiple protocols that must logically interoperate seamlessly across different physical networks, verifying equipment performance and reliability requires full access and visibility to network interactions at all layers.

2) Given the high data rates and extensive capabilities of FCoE, hardware-based testing tools are required that enable 100% line rate capture and achieve sufficient bandwidth saturation such as is required to effectively test Priority Flow Control (PFC) mechanisms.

3) Fabric-only tests using traffic generation tools may be unable to verify certain Data Center Bridging (DCB) protocols – such as congestion notification (CN) – that require that all devices in a link participate in network management.

4) Developers must be able to test both the LAN and SAN components of a network simultaneously.

Developers need to overcome each of these testing challenges in order to confidently release product to market. Over my next few posts, I’ll discuss each of these challenges in detail.

Recent news from Cisco has started what some experts foresee as a major shakeup that will significantly alter the landscape of network equipment vendors. The industry giant’s move to “Unify, simplify, amplify” communications networks is based on a strategic change of focus, with Cisco announcing that it will begin offering products that will directly compete with segments of its current customer base. To characterize this move solely as Cisco entering the server market, however, is to perhaps underestimate its near-term impact.

Consider that one of the primary value propositions of FCoE is unification of the network. Rather than have to maintain distinct storage and data network infrastructures, organizations will be able to deploy a single, logical network using virtualization and Ethernet as underlying technologies. However, while bridging networks across a single interface will simplify network access from an application standpoint, such unification is only possible through added complexity introduced into the network hardware itself.

One of the challenges for designers of next-generation network equipment will be in implementing this added complexity so as to be transparent to network administrators. If integrating equipment from different vendors presents too many problems, FCoE adoption will stall. From this perspective, Cisco’s announcement can be seen as an attempt, yes, to expand its market scope, but also to simplify deployment through unification at both the hardware and management software level. Put another way, by offering a system comprised of servers and fabrics, network administrators will not have to purchase various components and integrate them together as these issues will already have been taken care of. For example, Cisco’s new backplane consolidates myriad cables to a single cable. Additionally, a single system could house thousands of virtual servers. Without question, the drive to unify the network is changing how networks are being designed and deployed. FCoE is playing a key role in enabling this unification from a network perspective.

Certainly there is much that is controversial with this approach. There is also the question, as these systems come to market, of whether such systems will support existing infrastructure or interoperate only with a limited subset of available equipment that requires a more homogenous (i.e., single vendor) implementation.

Regardless of whether one views Cisco’s announcement as a prelude to a vendor shake down or as a welcome push to increase the momentum behind FCoE, it is clear that this industry leader is serious in its commitment to FCoE. For a technology that is already very promising given the benefits it brings to next-generation data centers, such strong support from the number one network equipment vendor can only mean accelerated adoption.

Migrating to FCoE sounds good on paper, but a key concern for network administrators is how to protect existing network performance when FCoE moves in.  In the past, network planners brought in expensive Fibre Channel equipment because of the guarantees it provided through its dedicated link architecture: guaranteed performance, guaranteed latency, and guaranteed robustness.  Ethernet is well-known for not providing these guarantees. 

In order for network administrators to feel confident enough to migrate to FCoE, they will need to be able to test and verify real-world performance against the marketing claims of FCoE equipment manufacturers.  For example, the IEEE FCoE standards being developed provide delivery guarantees, better latency performance, and higher bandwidth utilization.  While the standard drafts are as good as they sound, developers and administrators still need a way to verify that a particular implementation has been successful in meeting latency requirements. 

Likewise, administrators will need to be able to verify end-to-end performance.  An FCoE link is comprised of an Ethernet link accessing Fibre Channel storage.  Measuring performance up to the transition point fails to take into account all of the unexpected foibles of what happens when crossing protocol domains.  Developers and administrators need to be able to prove performance end-to-end; it’s simply too important a decision to leave to trust or assumption.  

Finally, key to generating sales of FCoE is the ability to compare native Fibre Channel to FCoE implementations.  Disruptive technologies need to not only reduce cost compared to incumbent technologies, they also need to provide greater functionality and/or benefit in order to overcome the inertia of the market buying what it is already used to.  Hard numbers will always sell more than wordy promises. 

Reaching the technological milestones of FCoE is important, but being able to prove value is what will carry FCoE to market successfully.

For FCoE to succeed in these economic conditions, it must bring down costs in the data center.  Much has been said about the benefits of consolidating Fibre Channel and Ethernet networks, in particular how FCoE consolidates hardware.  Rather than requiring two types of adapters for SAN and LAN – Fibre Channel HBAs and Ethernet NICs – FCoE allows SAN adapters to be collocated with LAN adapters.  Such an adapter is called a Converged Network Adapter (CNA). 

Converged networks integrate storage data and communication data on to the same pipeline with the FCoE technology  Network management, however, doesn’t change. Thus, FCoE technology promises to reduce the cost of hardware while requiring no additional spending for new management software, an overall cost-saving solution. 

Initial FCoE deployments, to the surprise of some, have not resulted in the substantial deployment and operational savings expected.  One reason for this is that the CNA and FCoE switch don’t have the similar cost structure compared to their corresponding peers.  These initial FCoE adapters are, in fact, more like two adapters crammed onto the same card.  As such, they cost almost as much as the two adapters they are replacing. 

Integration at the card level is not sufficient to reduce cost enough to entice network storage administrators to abandon the security and trust they have in their existing FC networks.  Equipment developers will need to leverage integration at the chip level – such as developing a single IC that integrates both FC HBA and Ethernet NIC ASIC functionality – to achieve the savings that will attract real attention. 

Another cost factor affecting FCoE technology is that FCoE is based mainly on 10 Gigabit Ethernet.  The cost savings of 10GE has not been sufficient compared to 1GE to really accellerate deployment.  Current efforts to further bring down the hardware cost of 10GE include higher level integration of discrete components such as LAN-On-Motherboard (LOM).  

Unless FCoE and Converged Ethernet can readily demonstrate significant cost savings in data centers, it seems they will still have a long way to go. 

Market predictors for the 2009 storage market suggest that the current economic downturn will result in significantly reduced activity. Specifically, companies will be tightening up expense budgets in the data center, which means there will be fewer dollars being spent on new equipment. As such, it appears any technological innovation may hit serious roadblocks trying to be adopted during this economic downturn.

The reality is that while spending may slow down, the demand for data will not. Bandwidth needs will continue to grow exponentially and, in order for storage networks to keep pace, new equipment acquisitions will be necessary. Rather than budget cuts reducing network capacity, network administrators will instead be challenged to do more with less. They will be more careful with their dollars and demand greater value when they do spend. Robustness, performance, bandwidth – these will all still be important considerations, but it is price that will determine sales. As a consequence, this economic downturn can be a golden opportunity for new technologies that successfully either reduce capital investment and/or eliminate operating expenses.

For FCoE to succeed in this climate, reducing cost is the key. While providing the advantages of a converged network bring many benefits to networks, if FCoE cannot seriously beat existing storage offerings on price, it will be slow to achieve traction.