Systems Research

Software implementation and testing of iWarp protocol

Principal Investigators: Pete Wyckoff, OSC and Dennis cpu Dalessandro,
OSC-S and Ananth Devulapalli, OSC-S.
Funding Source: Sandia National Labs
Duration: 4/21/2005--9/30/2005

Description: The term iWarp indicates a set of published protocol specifications that provide remote read and write access to user applications, without operating system intervention or intermediate data copies. The iWarp protocol provides for higher bandwidth and lower latency transfers over existing, widely deployed TCP/IP networks. While hardware implementations of iWarp are starting to emerge, there is a need for software implementations to enable offload on servers as a transition mechanism, for protocol testing, and for future protocol research.

The work presented here allows a server with an iWarp network card to utilize it fully by implementing the iWarp protocol in software on the non-accelerated clients. While throughput does not improve, the true benefit of reduced load on the server machine is realized. Experiments show that sender system load is reduced from 35% to 5% and receiver load is reduced from 90% to under 5%. These gains allow a server to scale to handle many more simultaneous client connections.

Networking is one of the key aspects in high performance computing. As systems grow larger and more complex network technologies continue to evolve. Networking technology is driven by the need to move larger amounts of data in smaller amounts of time. From the dawn of computing until recently, the bottleneck has been the network infrastructure.

Today as network speeds reach 10Gbps and beyond the bottleneck has moved to the CPU. Basically the CPU spends so much time handling communication related processing that little time is left to do computational work. It is for this reason that we turn to advanced interconnect technologies like Infiniband and iWarp.

The High-Performance Networks and File Systems Research Group is active on the forefront of network technology. From investigating performance of a new technology to pushing the boundaries of what a new technology can do, the group aims to play a key role in how high-performance networks are used.

Ongoing work is centered around a technology known as RDMA (Remote Direct Memory Access). From vast experience with Infiniband to being an early adopter of iWarp, the group is active on all fronts. Current work includes on-going performance studies with Infiniband as well as new ways of utilizing Infiniband for the benefit of the system. The group is also concentrating a great deal of effort on iWarp. From utilizing iWarp and its RDMA in the wide area over the OSCnet, to interoperability with Infiniband, the group plans to make extensive use of its new network research cluster, located at OSC Springfield.

Currently the group is very busy with a software port of iWarp which will allow hosts without iWarp hardware to communicate with hosts that do have special hardware. While the hosts without the hardware will see no benefit, the hosts with the hardware will be able to benefit form the protocol offload and operating system bypass provided by the iWarp hardware. With the OSCnet on its way to Springfield, the group will have the unique opportunity to engage in Wide Area high-performance networking research, this is made possible with the iWarp hardware in the new research cluster.

Continue to the main iWarp project page.
Go straight to download the Source Code
NOTE* Please send an email to or just to let us know who's downloading the code.

Also please note that an email list has been established for public discussion of software iWARP see the Software iWARP Email list

Thanks to Sandia National Labs for supporting the initial work on the software iWarp project. For additional information contact