RapidIO and the New Age of Fabrics

 

Sam Fuller

Executive Director – RapidIO Trade Association

In the late 90’s I had the privilege of working for Motorola and helping to plan the roadmap for PowerPC processors.  We were reviewing the interconnect technology requirements and recognized that the approach to buses that we were using were just not capable of scaling both bandwidth and connectivity concurrently.  The more devices you attached to a bus, in this case the 60x bus, the less performance the bus could provide.  The only way ahead seemed to be moving to a switched fabric as a replacement to the sorts of buses that had been used on processors for decades before.  We partnered with Mercury Computer on this effort as Mercury had significant experience in building fabric connected processing systems for the High Performance Computing (HPC) market for many years.  Because most of our customer base was in the embedded communications and networking space, we recognized the need for standards and interoperability among chips and systems provided by multiple vendors, so we embarked on a path of openness for our new interconnect and enlisted the support of leading equipment vendors in the effort. The RapidIO Interconnect and the RapidIO Trade Association were both formally launched in 2000.

In the 12 years since the launch of RapidIO a tremendous eco-system has developed around the RapidIO technology and it is widely deployed in a number of important markets with wireless base-stations being the most well known.  The technology has evolved from the initial Parallel –LVDS link layer specification through Serial RapidIO to RapidIO Gen 2, with products offering up to 40 Gbps bi-directional bandwidth to our new effort called 10xN, based on 10.312 GHz serial links which can provide up to 320 Gbps of bi-directional link bandwidth.

Over the years the RapidIO technology has also solidified and proven itself in hundreds of unique systems representing hundreds of thousands if not millions of shipped systems.  As one of the founders, I will say that there was a time when we weren’t sure if we were going to make it.  The competition seemed fierce and well funded.  There was Hypertransport and then NGIO and then Star Fabric all vying for the attention of customers.  All of these competitors have faded away.  Each of the competing technologies had its own value proposition, but there remained a unique value proposition associated with RapidIO that appealed to a very important customer base.  The same is true today, there are new competitors, each have their strengths and weaknesses with respect to RapidIO in any given application, but for an increasing number of very important applications, RapidIO continues to offer a compelling value proposition for systems that demand efficiency, reliability, low power, low latency and leading edge link and system bandwidth.

This RapidIO value proposition has been well known in the embedded market for many years.  It is now time to take this same value proposition to the mainstream data-processing markets, which are rapidly evolving to demand the same system attributes that communications networks have long required.

Investing in Fabrics

The last several months have seen a lot of M&A activity, where several important vendors of fabric technology have been acquired for nine figure sums.  It may not be as hot as photo-sharing (See #Instagram), but there are significant amounts of money  and investment flowing around fabric technology!

Both Intel and AMD have made sizeable recent investments to increase their technology portfolios and expertise as it relates to high speed interconnects.  Many other companies are re-examining their strategies of relying solely on PCI Express and 10G Ethernet for connectivity and looking for more optimal solutions for future datacenter connectivity needs.

AMD acquired SeaMicro for $334M based on their leadership approach to Micro-servers and their Freedom fabric technology and expressed plans to directly integrate fabric technology on to their future SOC devices.

Over the course of the last year, Intel has acquired Qlogic’s Infiniband business for $125M and Fulcrum Microsystems, a leader in 10G and 40G Ethernet switching for an undisclosed sum  (Fulcrum had raised $81M in venture funding prior to the acquisition) and recently agreed to acquired the high performance computing  (HPC) interconnect technology and team of Cray Computer for $140M, presumably to deploy even more tightly integrated systems for HPC and data center applications.

What does all this mean?

Opinions will vary of course, but what this represents in my view is the next stage of the evolution of computing.   One aspect of this evolution is towards mobile devices. The second is in the cloud data center,  here the evolution is towards more tightly coupled clusters of processors that represent processing farms built to host hundreds to thousands of virtual machines.  Or these servers will host pure application processing engines where all of the operating system and hardware constructs are completely abstracted away from the application software.  Google’s Application Engine is an example of this approach.  These processor clusters will be composed of up to thousands of multicore SOCs connected though high performance, low latency fabric interconnects.  The more efficient this fabric is, the better it will be for the performance and economics of the system.

Concepts that make a lot of sense when servers are strictly physical devices with internal hard-drives connected via SCSI and network interface cards plugged in to a PCI bus come into question when the server becomes a virtual computer running on a cluster of processors.  In this world the importance of evolutionary socket compatibility (think about the CPU sockets and the PCI bus evolution here) become less and less important and we can focus more and more on the core problem of efficiently connecting hundreds or thousands of processor SOC devices together.

The question becomes one of how do I build a system that can host hundreds or thousands of virtual servers as cost effectively and as power efficiently as possible.  There will remain, of course, an expectation of software compatibility that must be met, but virtualization and new APIs such as provided by OFED from the Open Fabrics Alliance greatly simplify this task.

Technologies like PCI Express and 10G Ethernet are certainly not going away any time soon, but they also will not be the foundation for these future tightly coupled computing systems.   PCI Express is not a fabric and can really only support the connectivity of small numbers of processors and/or peripherals.  It would only serve as a bridge to a fabric gateway device.  10G Ethernet can be used as a fabric, but has significant hardware and software protocol processing requirements.  Its widely variable frame sizes (46 Bytes to 9000 bytes for Jumbo frames) drive the need for both very fast processing logic to support lots of small packets and very large buffers to support very large packets in end-points and switches.  The use of PCI Express or 10G Ethernet will either restrict the topologies and connectivity available or add cost and overhead to the solution.  These drawbacks create opportunities for RapidIO in this very interesting new market.

Implementing servers, storage and high performance computing present an opportunity for OEMs to innovate with an interconnect paradigm that has the fundamental attributes of a low latency, and scalable switched fabric topology.  While there are challenges, they do not lie in the foundations of the RapidIO technology which has been successfully proven in wireless, industrial, imaging and aerospace markets.  As was the case for wireless where RapidIO went from early innovators to becoming the de facto base station interconnect standard, RapidIO’s biggest challenge in server, storage and HPC will be to cross the chasm from today’s innovators and early adopters markets to mass market proliferation.

It is an exciting time of change in the industry.  There will be new winners and losers and whole new multi-billion dollar opportunities appearing over the next five years.  We here at the RapidIO Trade Association believe that industry standard switch fabrics are a critical enabling technology for the next state of the evolution of computing.

 

Sam Fuller was a founder of the RapidIO Trade Association and new serves as Executive Director.  He has worked in all aspects of computing technology from Mainframes to desktops to SOCs and embedded systems from design to marketing and general management and startup CEO.  He has an MSEE in Computer Engineering from BYU and an MBA degree from the University of Texas.

 

Tags: