Systems Research
RDMA Enabled Apache (mod_rdma)
Principal Investigator: Dennis Dalessandro (OSC) & Pete Wyckoff (OSC)
1. Overview:
iWARP, RDMA over TCP/IP is emerging as an attractive solution for high
performance networking. One of the particularly attractive uses for iWARP is
to vastly improve performance of high end servers, that must handle many
connections. A prime example of this is a web server. With the emergence of
the OpenFabrics project, there is now a common API that any RDMA device can
support, allowing source code written for one device to work on another.
Software iWARP, emulating the iWARP protocols in software, allows
non-accelerated clients to communicate with hardware iWARP equipped
servers.
Definitions:
iWARP - is the term used to describe RDMA over TCP/IP. RDMA enables data
transfers with the lowest overhead and impact on a host. This is
not to be confused with the late 80's parallel computer project from
CMU/CS, of the same name.
OpenFabrics is the open source movement to provide support for RDMA devices
in the Linux Kernel. Currently InfiniBand and iWARP are supported.
With this project also comes a common API that can be used by all
supporting devices, allowing for code reuse.
Software iWARP as we use it refers to the capability to emulate the iWARP
protocols using only software. This allows a host equipped with a hardware
iWARP device to take advantage of its offload and OS bypass features, while
talking to a host that does not have such a device. The non-accelerated
device may experience a slight degradation in performance to benefit the
hardware equipped host.
Benefits
Server benefits
By utilizing the key features of the iWARP device; protocol offload, and OS bypass,
a web server is able to realize a drastic reduction in system load. This
requires that clients, speak the same language. That is, an iWARP device
needs to talk to an iWARP device. This can be accomplished in two ways.
Simply outfit all clients with a hardware iWARP device, which can be cost
prohibitive currently. The other option is to equip clients with software that
emulates hardware iWARP. While clients may see a slight higher overhead associated
with the extra processing of the iWARP stack, the server will be able to realize
the benefits of its iWARP device. The reduced load that the server will experience,
will allow for faster response times, and will enable to server to scale to handle
many more clients. Another benefit the server will see is more CPU to dedicate
to dynamic page creation, through PHP, or CGI. Having more CPU available for the
server is especially attractive for secure communication through SSL/TLS.
Client benefits
If clients are equipped with hardware iWARP then the client sees the same benefits
as the server, in reduced load. As stated above the clients may incur some extra
overhead where software iWARP is concerned. In general this overhead will not
even be noticed, particularly for web applications. The client will see benefits
though, in that the server is able to respond quicker to requests for data.
Clients on the whole will see a benefit, as the server will be able to handle a
larger volume of clients.
API Choice
There are numerous APIs we could have chosen for implementation. We have chosen
the OpenFabrics verbs API. It would be possible to create an Apache server module
in a native API for each iWARP device we wish to support, but it is far better to
write the software with an API that all devices can support. One such options for
this approach is OpenFabrics. OpenFabrics support is also included in the Linux
Kernel, which makes it likely that any existing iWARP device, and future devices
will make use of it. Other choices, such as DAPL, offer a single API for multiple
devices. Although DAPL is an API that is built on top of a lower level API, such
as an iWARP adapters native verbs. There is a DAPL implementation for OpenFabrics,
but we have chosen to use the lowest level API possible from userspace, that is
the OpenFabrics verbs. This should allow for slightly better performance than
DAPL and free us of any potential restrictions. DAPL, is also an optional
component to OpenFabrics, where as the OF verbs are guaranteed to be supported.
Design Overview
This implementation will follow the design of previous work in creating an iWARP
module for Apache. Our goal is to implement a more stable, production quality
software implementation, that supports features lacked in the previous implementation.
The previous software was written in the native verbs API of a particular iWARP
adapter. We are rewriting the software to use the OpenFabrics API. This will
prevent future rewrites of the code.
Our design includes two methods for retrieving data from the server. One known
as server-writes, the other as client-reads. In both cases, all request and r
response messages are sent using sockets() TCP. The default in Apache. This
frees us from vastly modifying the internals of Apache. By adding some
information to the HTTP request/response headers we are able to convey the
necessary information to do RDMA transfers between client and server.
In the server-writes case, the client makes the HTTP GET request. The server
uses information contained in the GET headers to make an RDMA connection back
to the client. The server then does an RDMA write to push data out to the
client. Once completed the server simply sends the usual HTTP 200 OK response,
just with out the requested data, since its already been written of course.
In the client-reads case, the client again makes the usual HTTP GET request,
and the server establishes the RDMA connection, but now the server must send
the HTTP response to tell the client where to RDMA read the data from. Once
the client receives the response it is able to pull the data from the sever.
The problem is that the server must wait until getting an acknowledgment from
the client before it can continue, or release resources associated with that
request. Due to this, once the client has read the data from the server, the
client sends an RDMA message back to the server to signal completion.
Client Applications
An RDMA enabled web server is no good if there are no clients to talk to.
We have implemented at least one client application to demonstrate what is
possible. The popular Linux command line tool, wget was our first choice.
Another potential client application, is a browser Plug-in, that would enable
web browsers such as Mozilla to talk to hardware iWARP equipped servers.
Yet there is another option for a client application, and that is a proxy
server. A proxy server could accept client connections in TCP and simply
tunnel requests through iWARP (hardware or software) to the web server.
HOME
NEXT
|
