Luigi Rizzo, UNIPI, Aug. 2011 img/unipi.jpg

Netmap

netmap: a novel framework for high speed packet access
Luigi Rizzo, Università di Pisa
with help from
Gaetano Catalli, Marta Carbone, Matteo Landi
supported by EC Project CHANGE
http://info.iet.unipi.it/ luigi/netmap/

L < > T H local Netmap -- Page 1/36

# $Id: demo-plot.txt 8558 2011-04-23 22:10:21Z luigi $
[@background]
    {tr, Luigi Rizzo, UNIPI, Aug. 2011}
    {bl, @@sz5,img/unipi.jpg@@}
    {br, @@fr sz10,img/change-logo-small.jpg@@}

[Netmap]

{title, netmap: a novel framework for high speed packet access}

@@logo,img/bolt.jpg@@
{b author, Luigi Rizzo, Universita` di Pisa \\
{b fn60,with help from \\ Gaetano Catalli, Marta Carbone,
Matteo Landi}  }

{author, supported by EC Project CHANGE \\
[[http://info.iet.unipi.it/~luigi/netmap/]] }

[Motivations]

We have been doing packet processing in software on
general purpose OS for the past 30 years
+ routing/switching;
+ firewalls
+ traffic analysis;
+ network monitoring;
+ network testing.
Plenty of this software in use even in production systems

[Motivations]

Network speeds have increased by 3 orders of magnitude in the meantime\\
@@sz40,img/morse1.jpg@@
@@sz40,img/10g.jpg@@

But software architectures are still the same
- raw socket, BPF, libpcap
- mbuf/skbuf/NdisPacket encapsulation
- one system call per packet, poor parallelism
Even with faster clock speeds, there are some things that
do not scale:
- memory and bus latency, system calls

[Motivations]

At 10 Gbit/s the packet rate is 14.88 Mpps, or 67.2 ns/pkt.\\ \\

{h1 ctr red,How do we deal with such high rates ?}

<table class="fn80"><tr><td>
throw hardware at the problem\\
@@sz50,img/2403851423_eabf683a59_z.jpg@@ </td>

<td> use existing hardware the best we can\\
@@sz80,img/10g.jpg@@
</td></tr></table>

[So, what have we got ?]

{tt,netmap} is a novel framework (available on FreeBSD)
for fast packet I/O from userspace.
* 70..90 cycles per packet between the wire and userspace code
  (14.88 Mpps at 1.2 GHz)
* good scalability with CPU frequency and number of cores
* easy API
* {tt,libpcap} emulation library for easy porting of applications

@@ctr sz50,plot-tx-clock.jpg@@

[And how ?]

Summary of the talk

- existing options for packet I/O (and what's wrong with them)
- how {tt,netmap} addresses these problems
- the netmap API
- internals
- performance
- experience on porting applications
- future work

[Existing options for packet I/O]

{b,userspace:}
  + rich and friendly development/runtime environment
  + packet I/O uses various kinds of sockets, or BPF, or libpcap
  + {b red,expensive system calls and data copies} just to move packets
    from/to the kernel
{b, in-kernel:}
  + dangerous and constrained environment
  + modules using netfilter/pfilter/Ndis hooks
  + direct device management (Click)

{b, some experimental proposals:}
- various libpcap optimizations (mmap, batching...) \\
  only solve half of the problem.
- direct device access (UIO-IXGBE, DNA) \\
  as dangerous as in-kernel options
- custom hardware (NetFPGA, DAG, ...) \\
  hw-dependent

[In-kernel operation]

Even remaining in the kernel is expensive
- the NIC can handle lists of buffers without CPU intervention
- but the OS adds a {red b,costly encapsulation}
  (per-packet {b,malloc/free}, refcounted
  pointers, data sharing, ...);
@@ctr sz70,img/mbufs.jpg@@

[Netmap goals and approach]
{ctr red b, netmap focuses on packet I/O for userspace applications.}
Fix the weaknesses of existing APIs, taking good ideas from
existing proposals:
- amortize system call costs, reduce/remove data copies
- remove unnecessary allocations
- {red,don't be afraid of change (netmap could have been done 10 years ago!)}
Most of these ideas have been proposed and used before.

{ctr red b,Separately.}
But a collection of ideas does not make a system, and we want
to design and build one that is {red,usable} and {red,maintainable}:
- simple API (revised based on application requirements)
- device indepedent
- easy to integrate in existing OS
- compatible with existing applications

[Netmap data structures]

We need an efficient representation for batches of packets
- amortize syscall costs
- remove allocations
- reduce data copies
Hint: look at the NIC ring
  @@ctr sz50,img/mbufs.jpg@@

[The netmap ring]

@@my50,img/netmap-ring.jpg@@
A shadow copy of the NIC ring
- global fields:\\
  {tt b,cur}: current rx/tx position,\\
  {tt,avail}: available slots
- compact (8 bytes) slots:\\
  {tt, length, buffer index, flags}
- all addresses are relative (position independent).

Data structures are in {tt,mmap()}-ed memory.
{b, Ownership} is well defined:
- {red,global fields, and {tt,[cur..cur+avail-1]} :}
  owned by the application\\
  synchronized with the kernel during
  {tt,poll()} and {tt,ioctl(fd,NIOC*XSYNC)}
- {red,other slots/bufs:} owned by the kernel
these rules permit lock free access to the rings and buffers

[Netmap API]

@@my50,img/netmap-ring.jpg@@
{h1,Access}
- {tt,open("/dev/netmap")} returns a file descriptor
- {tt,ioctl(fd, NIOCREG, arg)} puts an interface
  in netmap mode
- {tt,mmap(..., fd, 0)} maps buffers and rings
{h1,Transmit}
- fill up to {tt,avail} buffers, starting from slot {tt,cur}.
- {tt,ioctl(fd,NIOCTXSYNC)} queues the packets
{h1,Receive}
- {tt,ioctl(fd,NIOCRXSYNC)} reports newly received packets
- process up to {tt,avail} buffers, starting from slot {tt,cur}.

[Netmap API - synchronization]

@@my50,img/netmap-ring.jpg@@
- {tt,ioctls(fd,NIOC*SYNC)} are non blocking
- {tt,poll()} and {tt,select()} work as expected,
  returning when {tt,avail>0}
- POLLIN and POLLOUT decide which sets of rings to work on
- interrupt mitigation delays are propagated up to the userspace process
- can use the {tt,ioctl()}s for busy-waiting

[Multiqueue/Multicore support]

@@my50,img/netmap-ring.jpg@@
Multiqueue cards and multicore systems are fully supported:

+ one netmap ring per physical ring on the NIC;
+ by default, the file descriptor operates on all rings
+ {tt,ioctl(fd, NIOCREG, arg)} can restrict the association
  to a single ring;
+ multiple {tt,open("/dev/netmap")} on the same card can
  be bound to different rings (and issued by different thread);
+ {tt,setaffinity()} can be used to map threads to cores

[Netmap and the host stack]

@@my30,img/host-stack.jpg@@
While in netmap mode, the control path remains unchanged:
- ifconfig, ioctl's, etc still work as usual
- the OS still believes the interface is there
The datapath is detached from the host stack:
- packets from the net end up in RX netmap rings
- packets from TX netmap rings are sent to the net

{red b,The host stack is attached to an extra pair of netmap rings:}
- packets from the host go to a {red,SW RX} netmap ring
- packets from a {red,SW TX} netmap ring are sent to the host
- the two extra rings are also managed using the netmap API:
- {tt,open("/dev/netmap")} and {tt,ioctl(fd, NIOCREG, arg)} to
  associate a descriptor to the {red,SW} rings;
- {tt,mmap()} to access the ring and buffers
- {tt,ioctl()} or {tt,poll()} to send and receive

[Example of use]

Using the native API is very simple.

Real code for a traffic generator (handles multiple queues,
does line rate at 10 Gbit/s)
@@fr sz40,img/netmap-ring.jpg@@
. <pre>
fds.fd = open("/dev/netmap", O_RDWR);
strcpy(nmr.nm_name, "ix0");
ioctl(fds.fd, NIOCREG, &nmr);
p = mmap(0, nmr.memsize, fds.fd);
nifp = NETMAP_IF(p, nmr.offset);
fds.events = POLLOUT;
for (;;) {
    poll(fds, 1, -1);
    for (r = 0; r < nmr.num_queues; r++) {
	ring = NETMAP_TXRING(nifp, r);
	while (ring->avail-- > 0) {
	    i = ring->cur;
	    buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
	    ... store the payload into buf ...
	    ring->slot[i].len =  ... \/\/ set packet length
	    ring->cur = NETMAP_NEXT(ring, i);
	}
    }
}
</pre>

[Multiple interfaces]

All rings and buffers for all interfaces in netmap mode
are in the {b,same} mmapped region:

- can do zero-copy forwarding by just swapping buffer indexes

@@fr sz40,img/netmap-ring.jpg@@
. <pre>
  src = &src_nifp->slot[i];
  dst = &dst_nifp->slot[j];

tmp = dst->buf_index;
  dst->buf_index = src->buf_index;
  src->buf_index = tmp;

dst->len = src->len;
  dst->flags = src->flags = BUF_CHANGED;
</pre>
Zero-copy also works with the rings from/to the host stack.
- firewalls, NAT boxes, IDS can use this mechanism

[Safety and isolation]

{ctr red b,netmap is meant to be used by trusted processes running
  with root privileges.}
The content of NIC rings is validated by the syscalls:
- no chance to crash the kernel or overwrite other memory
- protection at the page-level provided by the OS
Packet corruption still possible if a process is misbehaving
- but compare with the alternatives:\\
  in-kernel modules or direct NIC access are even more dangerous

{h1,Planned work: implement {em,virtual} per-process netmap rings}
- needs data copying for full isolation
- needs to design a mechanism to demux incoming packets
- still saves on the syscall and in-kernel representation
- can we do it as a userspace process ?

[The implementation]

Current prototype developed on FreeBSD (HEAD and RELENG_8)

{b,kernel side:}
+ 2000 lines of (heavily commented) common code and headers,
  small device-specific patches<pre>
$ wc -l net/* dev/netmap/* 
     283 net/netmap.h
      98 net/netmap_user.h
     393 dev/netmap/if_bge_netmap.h
     425 dev/netmap/if_em_netmap.h
     375 dev/netmap/if_lem_netmap.h
     436 dev/netmap/if_re_netmap.h
     425 dev/netmap/ixgbe_netmap.h
     306 dev/netmap/netmap.4
    1691 dev/netmap/netmap.c
     225 dev/netmap/netmap_kern.h
</pre>
{b,Userspace:}
+ just a few macros in netmap_user.h needed for the native API
+ a small libpcap emulation layer is enough to port many
  existing applications
  <pre>
     756 examples/pcap.c
</pre>

[Internals]

The netmap code is designed to minimize changes to the OS sources
- the common code is a completely independent module
- per-device code has all additional handlers in an external file
  <pre class="fn80">
        int (*nm_register)(struct ifnet *, int onoff);
        void (*nm_lock)(void *, int what, u_int ringid);
        int (*nm_txsync)(void *, u_int ring, int lock);
        int (*nm_rxsync)(void *, u_int ring, int lock);
</pre>
  plus very small patches to the ring initalization and
  interrupt service functions
  <pre class="fn80">
   ...
+#ifdef DEV_NETMAP
+#include \<dev/netmap/if_re_netmap.h>
+#endif
   ...
+#ifdef DEV_NETMAP
+       re_netmap_tx_init(sc);
+#endif
    ...
+#ifdef DEV_NETMAP
+       if (ifp->if_capenable & IFCAP_NETMAP) {
+               selwakeuppri(&NA(ifp)->rx_rings->si, PI_NET);
+               return 0;
+       }
+#endif
    ...
  </pre>

[Internals (2)]

Devices in netmap mode are essentially insensitive to livelock:
- the interrupt service routine does nothing but a {tt,selwakeup()}
- NIC rings are scanned by the upper half of the kernel in
  {tt,ioctl()} and {tt,poll()}
- the system remains perfectly controllable even under severe overload

{tt,poll()} and {tt,select()} are optimized 
for common usage patterns:
- unnecessary calls (those with {tt,avail \> 0}) are lock-free
- pending transmissions are sent out even if POLLOUT is not specified
- {tt,poll()} and {tt,select()} set a timestamp in the ring
These two features typically save two system calls
per iteration of eventloops.

[Libpcap emulation]

@@fr sz30,img/netmap-ring.jpg@@
Many applications use libpcap so we built a
simple emulation library (many stubs,
plus the two real I/O functions):
<pre>
int pcap_inject(pcap_t *p, void *buf, size_t size)
{
    int si, idx;

for (si = p->begin; si < p->end; si++) {
	struct netmap_ring *ring = NETMAP_TXRING(p->nifp, si);

if (ring->avail == 0)
	    continue;
	idx = ring->slot[ring->cur].buf_idx;
	bcopy(buf, NETMAP_BUF(ring, idx), size);
	ring->cur = NETMAP_RING_NEXT(ring, ring->cur);
	ring->avail--;
	return size;
    }
    return -1;
}
</pre>

[Libpcap emulation (2)]

@@fr sz30,img/netmap-ring.jpg@@
<pre>
int pcap_dispatch(pcap_t *p, int cnt,
    pcap_handler callback, u_char *user)
{
    int si, ret = 0;

for (si =p->begin; si < p->end; si++) {
	struct netmap_ring *ring = NETMAP_RXRING(p->nifp, si);

while ((cnt == -1 || ret != cnt) && ring->avail > 0) {
	    int i = ring->cur;
	    int idx = ring->slot[i].buf_idx;

p->hdr.len = p->hdr.caplen = ring->slot[i].len;
	    callback(user, &p->hdr, NETMAP_BUF(ring, idx));
	    ring->cur = NETMAP_RING_NEXT(ring, i);
	    ring->avail--;
	    ret++;
	}
    }
    return ret;
}
</pre>

[Performance]

Most of the cost is per-packet, so
the relevant metric is packets per second (pps), not bytes per second
- transmit and receive
- forwarding (native and with pcap emulation)
- ported applications (OpenvSwitch, Click userspace)

All experiments using intel 82599 dual-port card,
i7-870 CPU (2.93 GHz, 4 cores) if not specified otherwise.

{h1,Warning/disclaimer:}
- we try to measure only the packet I/O overhead, not the application
- CPU-intensive applications will benefit less from efficient I/O
- many cards (even 1 Gbit models) cannot do line rate irrespective
  of CPU speed
- buses, caches, CPU and memory speeds also affect your results
Anyways, the code is available, you can try it yourself.
[Tx throughput versus clock speed]

Tx performance with 1, 2, 4 cores at variable clock speeds (0.15..2.93 GHz).\\
10 Gbit/s interface, 64-byte packets ({tt,examples/pkt-gen.c})

@@ctr sz60,plot-tx-clock.jpg@@

[Tx throughput versus burst size]

Tx performance vs burst size. \\
10 Gbit/s interface, 64-byte packets, 2.93 GHz.
@@ctr sz60,plot-tx-burst.jpg@@

[Rx throughput versus packet size]

Receive does line rate at multiple of 64 bytes,
at other sizes the hardware limits the throughput
(reported by others as well).
Measured with  {tt,examples/pkt-gen.c} in receive mode.
@@ctr sz60,plot-rx-size.jpg@@

[Forwarding performance]

Forwarding speed between two interfaces, using 1 core.

Measured with {tt,examples/bridge.c} and {tt,examples/pcap.c}

@table l r
{b,Configuration} | {b,Mpps}
native API |  10.66
native API + timestamp | 9.42
libpcap emulation | 7.50
freebsd bridging | 0.69
@endtable

[Ported applications]

The libpcap emulation was used to port/test various applications.
- just run with LD_LIBRARY_PATH=path/to/new_libpcap

- many simple apps (such as trafshow) just run much faster
  out of the box
- others needed tweaks to exploit the faster library
{h1,Case studies}
- OpenvSwitch, userspace module (from 65 Kpps to 3 Mpps)
- Click userspace (from 400 Kpps to 3.95 Mpps)

[OpenvSwitch]

OpenvSwitch is a software implementation of an
OpenFlow controller plus an OpenFlow switch
- two forwarding engines: in-kernel and userspace (experimental)
- written in portable C, but uses Linux APIs

We ported the userspace module to FreeBSD using libpcap
- initial performance was poor (50..65 Kpps, both for
  Linux and FreeBSD) due to what
  we believe an oversight in the design
- fixing the problem brought the userspace performance to 780 Kpps
  (comparable with FreeBSD bridging, 690 Kpps)
- replacing libpcap with our version went up to 3 Mpps

{ctr red h1,Even the optimized version runs almost 4 times faster}

[Click]

Click is a popular packet processing platform.
Elements build packet processing graphs:

{ctr tt,FromDevice(ix0) -> Queue -> ToDevice(ix1)\\
  FromDevice(ix1) -> Queue -> ToDevice(ix0)}

- in kernel version uses modified device drivers.
  Widely used, high performance (Linux)

- userspace version: 400 Kpps with libpcap, 1.3 Mpps with netmap

- the C++ allocator was the bottleneck.
  Fixes contributed back to the Click project, now in 2.0

Forwarding performance with one core:

. @table l r r
  {b,Version}  | {ctr b,Kpps<br/>(original) } | {ctr b,Kpps<br/>(new)}
  kernel (estimated)   | 3-4000 | -
  libpcap | 400 | 495
  netmap  | 1300 | 3950
  @endtable

{ctr red h1,Click userspace is now
competitive or better than the in-kernel version}

[Related work]

Various mechanisms available for raw packet I/O
+ most of them retain the driver encapsulation (mbuf/skbuf):
  - netslice, PF_RING, libpcap variations with mmap support;
  - Click-based prototypes (including RouteBricks)
  - in-kernel modules: netfilter/pfilter/Ndis clients,
    netgraph.
  reduced syscall/copy overhead, still high per-packet overhead.
+ some control directly the hardware:\\
  - PF_RING-DNA, UIO-IXGBE
  need hardware-specific libraries;\\
  as dangerous as kernel applications (unless IO-MMU is used)

+ custom hardware and associated drivers
  - (DAG, NetFPGA)
  hardware specific

[Related work (2)]

The PacketShader Packet I/O engine
[[http://shader.kaist.edu/packetshader/io_engine/]]
is the closest to our work in terms of performance
and architecture
- reported performance similar to netmap
- exports data structures similar to netmap rings
- uses ioctl() to syncronize kernel and userspace

Limitations
- custom device driver, only supports one specific device
- does not seem to support {tt,select()/poll()}
- no pcap emulation
- no documentation

[Availability]

[[http://info.iet.unipi.it/~luigi/netmap/]]
is netmap's home, continuously updated with
- code snapshots (kernel, libraries, examples)
- bootable images (PicoBSD-based)
- papers and presentations

Currently works on
- FreeBSD HEAD (and RELENG_8 with minimal effort)
- ixgbe, e1000, re drivers
- user-contributed igb and bce drivers

[Future work]

Boring but useful tasks
- small performance optimizations
- support more cards
- port/fix more applications (either on libpcap or native API)
- linux port
Explore architectural issues
- per-user virtual rings
  - improved safety and isolation
- bring subsystems to userspace
  - firewalls (ipfw) and schedulers
  - routing tables

[Conclusions / Acks / Links]

{ctr red h1, Line-rate at 10 Gig in userspace is possible
without any magic.}

{h1,Thanks to:}
+ the Change Project @@sz10,img/change-logo-small.jpg@@
+ Intel Research Berkeley
+ my students Gaetano Catalli, Marta Carbone, Matteo Landi

{h1,Netmap home page}
+ [[http://info.iet.unipi.it/~luigi/netmap/]]