Reflections Of The Void: How to install Soft-iWARP on Ubuntu 10.10, AKA how to have RDMA enabled system without the expensive hardware.

What is Soft-iWARP and why installing it :

Soft-iWARP [1] is a software-based iWARP stack that runs at reasonable performance levels and seamlessly fits into the OFA RDMA environment provides several benefits:

As a generic (RNIC-independent) iWARP device driver, it immediately enables RDMA services on all systems with conventional Ethernet adapters, which do not provide RDMA hardware support.
Soft-iWARP can be an intermediate step when migrating applications and systems to RDMA APIs and OpenFabrics.
Soft-iWARP can be a reasonable solution for client systems, allowing RNIC-equipped peers/servers to enjoy the full benefits of RDMA communication.
Soft-iWARP seamlessly supports direct as well as asynchronous transmission with multiple outstanding work requests and RDMA operations.
A software-based iWARP stack may flexibly employ any available hardware assists for performance-critical operations such as MPA CRC checksum calculation and direct data placement. The resulting performance levels may approach those of a fully offloaded iWARP stack.

How to install Soft-iWARP :

When I tried to install Soft-iWARP I ran into several issue and I am writing this installation manual so i don’t forget how to do it and other people can enjoy a semi seamless installation.

My setup:

Core 2 duo
e1000 nic
4 Gb of RAM
Ubuntu 10.10 server
Linux kernel : 2.6.35-28-generic

Pre-Install

Getting the Linux kernel dev environment:

sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge

```
sudo apt-get build-dep linux
```

sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev

Setting up the Infiniband / libibverbs environment

sudo apt-get install libibverbs1 libibcm1 libibcm-dev  libibverbs-dev  libibcommon1 ibverbs-utils

check if  “/etc/udev/rules.d/40-ib.rules” exist

1. If the file doesn’t exist , create it and populate it with (you need to reboot after):

#### /etc/udev/rules.d/40-ib.rules ####

KERNEL=="umad*", NAME="infiniband/%k"

KERNEL=="issm*", NAME="infiniband/%k"

KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"

KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"

KERNEL=="uat", NAME="infiniband/%k", MODE="0666"

KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"

KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

########

Verifying the Infiniband / libibverbs environment

· Run these command in the following order:

1. sudo modprobe rdma_cm

2. sudo modprobe ib_uverbs

3. sudo modprobe rdma_ucm

4. sudo lsmod

5.ls /dev/infiniband/

Getting the compile / dev environment for Soft-iWARP :

sudo apt-get install libtool autoconf

Installing Soft-iWARP :

Getting Soft-iWARP

Getting the source from the git repository :

You can get the source from : https://github.com/zrlio/softiwarp

Compiling and installing the Soft-iWARP kernel module :

Inside the soft-iwarp kernel module folder:

make
sudo make install

Compiling and installing the Soft-iWARP userlib:

Inside the soft-iwarp userlib folder:

./autogen.sh
Run AGAIN ./autogen.sh
./configure
make
sudo make install

· at that stage soft-iwrap install the infiniband stuff in “/usr/local/etc/libibverbs.d” we need to make asymbolic link t /etc/infiniband.d

o sudo ln -s /usr/local/etc/libibverbs.d /etc//libibverbs.d

Inserting the Soft-iWARP kernel module:

1. sudo modprobe rdma_cm

2. sudo modprobe ib_uverbs

3. sudo modprobe rdma_ucm

4. sudo insmod /lib/modules/"your_kernel"/extra/siw.ko

5. sudo lsmod

§ check if all the modules are correctly installed

6. ls /dev/infiniband/

Soft-iWARP kernel module parameters:

loopback_enabled: if set, attaches siw also to the looback device.

To be set only during module insertion.

mpa_crc_enabled: if set, the MPA CRC gets generated and checked

both in tx and rx path (kills all throughput).

To be set only during module insertion.

zcopy_tx: if set, payload of non signalled work requests

(such as non signalled WRITE or SEND as well as all READ

responses) are transferred using the TCP sockets

sendpage interface. This parameter can be switched on and

off dynamically (echo 1 >> /sys/module/siw/parameters/zcopy_tx

for enablement, 0 for disabling). System load benefits from

using 0copy data transmission: a servers load of

one 2.5 GHz CPU may drop from 55 percent down to 23 percent

if serving 100KByte READ responses at 10GbE's line speed.

Verifying the Soft-iWARP is correctly installed :

running :

ibv_devinfo
ibv_devices

· You should get :

if you get:

libibverbs: Warning: couldn't load driver 'siw': libsiw-rdmav2.so: cannot open shared object file: No such file or directory

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

o you need to run “sudo ldconfig” to update the library path

Running the simple test:

Install rdma-utils :

sudo apt-get install rdmacm-utils

Then you can test the soft-iwarp with :

rping : RDMA ping
udaddy : Establishes a set of unreliable RDMA datagram communication paths between two nodes using the librdmacm, optionally transfers datagrams between the nodes, then tears down the communication.
mckey : Establishes a set of RDMA multicast communication paths between nodes using the librdmacm, optionally transfers datagrams to receiving nodes, then tears down the communication.
ucmatose : Establishes a set of reliable RDMA connections between two nodes using the librdmacm, optionally transfers data between the nodes, then disconnects.

Note : the ibv_* utils don't work with iWARP device . you will get errors if you try to use them such as:

[root@iwarp1 ~]# ibv_rc_pingpong
Couldn't get local LID
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_uc_pingpong
Couldn't create QP
[root@iwarp1 ~]# ibv_ud_pingpong
Couldn't create QP

Note 2 : If you only have one machine and you want to use the loopback (optional you can still use your normal nic), to test if you have it enabled: