Four ways to do super fast packet processing in Go

Meet packetio, the Go packet I/O library I wish I'd had a years ago. AFXDP, DPDK, mlx5 Direct Verbs, and AFPACKET behind one API.

The last few months I've reignited my strange hobby of doing network packet processing in software. My language of choice is Go, and I'm always looking and learning how to do this as fast as possible.

A few weeks ago I released go-afxdp, a Go library for sending and receiving packets using AF_XDP. Then I built Wireblast on top of it and got that to 100G line rate.

I've also been playing quite a bit with DPDK, different NICs, and more recently NVIDIA/Mellanox Direct Verbs.

It's been a fun rabbit hole :) But while doing all of this, I kept running into the same thing.
The actual packet processing is usually the easy part. Receive some packets, inspect them, maybe change a few bytes, do a lookup, and send them back out.

Getting packets efficiently between the NIC and your Go program is where all the complexity lives. And depending on how you want to do that, you end up in completely different worlds.

AF_XDP has UMEM and rings. DPDK has mbufs, mempools, EAL, hugepages and PMDs. Mellanox ConnectX cards have yet another path using Direct Verbs. All complicated words most of us probably don't want to deal with.

That's what I've been working on. One API to rule them, It's called packetio.

Four ways to get at a packet

There are basically four ways I care about for getting raw packets in and out of a Go program on Linux.

There's AF_PACKET, which is the normal Linux packet socket. It's what tools like tcpdump use. It works almost anywhere and requires very little setup, but it's slow as packets still go through much of the normal kernel machinery.

Then there's AF_XDP, which I've written about quite a bit already. It lets you intercept packets very early in the Linux driver and move them into memory shared with your application. On supported drivers it can be zero-copy, so the NIC writes packets directly into memory your Go program can access.

The part I really like about AF_XDP is that Linux still owns the interface. You can send only the traffic you care about to your program and your SSH session, ARP, routing protocols and everything else keep working normally.

Then there's DPDK.
DPDK is still the obvious answer when someone says "I want to do packet processing as fast as possible in userspace." It supports a large number of NICs and has been optimized for this job for years.

It's also a bit of a world of its own. If you've ever started a DPDK application from scratch, you know what I mean. Hugepages, EAL arguments, mbufs, mempools, PCI devices, queue setup. None of it is particularly impossible, but there's a huge learning curve.

And then there is the fourth option, which is the one I only really started appreciating recently: mlx5 Direct Verbs. More on that one in a bit.

One API over all of them

The idea behind packetio is pretty simple. A device has RX queues and TX queues. Queues give you frames. You process those frames in batches and give them back.
That's about as much as I want the application to know.

For example:

dev, err := mlx5.Open("eno2", mlx5.WithQueues(4))

Or afxdp:

dev, err := afxdp.Open("eno2", afxdp.WithQueues(4))

Or DPDK:

dev, err := dpdk.Open("0000:c1:00.1", dpdk.WithQueues(4))

There's also:

dev, err := afpacket.Open("eno2", afpacket.WithQueues(4))

After that, the packet loop is the same. Something like:

rx := dev.RxQueue(0)
for {
    rx.Fill(rx.NumFreeFillSlots())
    rx.Poll(time.Second)

    descs := rx.Receive(256)

    for _, desc := range descs {
        pkt := rx.Region().Frame(desc)
        // Do something useful with the packet.
        process(pkt)
    }
    rx.Recycle(descs)
}

And transmitting is similarly boring:

tx := dev.TxQueue(0)

tx.SendFunc(256, func(_ int, frame []byte) int {
    return copy(frame, myPacket)
})

That's really all there's to it! You still have queues and batches because those things matter when you're doing high-speed packet processing. I'm not trying to hide reality behind some net.Conn abstraction.

But I also don't think you should need to understand an AF_XDP fill ring, an rte_mbuf, or an mlx5 WQE before you can get a packet onto the wire.
Changing the backend should mostly mean changing how you open the device.
That sounds like a relatively small thing, but in practice I think it's pretty powerful.

DPDK from Go, finally

One of the original motivations for this project was actually DPDK.
I've wanted a good way to use DPDK from Go for years.

There have been Go wrappers before, but I never found one that gave me the programming model I wanted. I don't want to write my packet processing in C, and I definitely don't want to cross from Go into C for every packet.

packetio crosses into DPDK once per batch, not once per packet. The packet memory stays in DPDK's mempool and Go works directly on those bytes.
So you get to write what looks like a pretty normal Go program while DPDK does all the ugly setup underneath.

No rte_eth_rx_burst scattered through your application. No mbuf ownership rules in your business logic. No giant EAL argument string you copied from Stack Overflow three years ago. And if you don't know what that means, congrats, you dont need to!

You open the device and start receiving packets, all in Go.That's the DPDK experience I've wanted for a long time.

Direct Verbs was the surprise

The most interesting part of this project for me has probably been learning about mlx5 backend and something called direct verbs.

Jérôme Tollet has been doing some great work in VPP around NVIDIA/Mellanox ConnectX NICs. He recently published a comparison of VPP's three high-performance datapaths: AF_XDP, DPDK mlx5 and native RDMA Direct Verbs.

That sent me down another rabbit hole. It turns out that with a ConnectX NIC you can talk to the hardware queues directly using mlx5 Direct Verbs.

So instead of:
Go > DPDK > mlx5 PMD > NIC
you can do:
Go > mlx5 queues > NIC

There is obviously quite a bit hiding behind that sentence, but from the application's perspective that's the useful mental model.

And it has another property I really like. ConnectX NICs support a bifurcated model. Linux can keep its normal interface while hardware steering sends selected traffic directly into queues owned by your application.

So eno2 doesn't disappear. SSH keeps working. Linux can still use the interface for ordinary traffic while your Go program handles the packets you've claimed.

ConnectX can do this with DPDK too, but Direct Verbs lets us skip the DPDK layer entirely. For what I've been building lately, that's a pretty compelling combination.
AF_XDP-like operational behavior, but with a very short userspace datapath.

How fast is it?

Yes, the big question! I benchmarked all of this! I actually spent quite a bit of time on these numbers, so I'm going to put them all here.

Hopefully they're useful to someone else comparing these different approaches. And if nothing else, I know I'll be coming back to this blog myself six months from now trying to remember how DPDK compared to AF_XDP :)

The setup is two AMD EPYC 9275F machines with NVIDIA ConnectX-6 Dx 100G NICs. I'm testing 64-byte packets, which is really the worst case for packet processing. At that size, 100G Ethernet is about 148.8 million packets per second.

That's one packet every 6.7 nanoseconds. That's very little time. So yeah, things like memory accesses and extra trips through the kernel start to matter.

The nice thing about this new packetio library is that all four backends are running essentially the same userspace benchmark program. Same receive loop, same transmit loop, same forwarding code. The backend is the only thing that changes.

I'm also measuring CPU across the whole machine, not just the Go process. That's particularly important for AF_XDP. Quite a bit of its work happens inside the kernel in IRQ/NAPI/softirq context. If you only measure your Go process, those cores magically disappear from the benchmark even though you're still paying for them.
Jérôme took the same approach in his VPP measurements, which is one of the things I liked about his comparison.

Receive

For receive I'm offering 148.6 Mpps across 4096 flows.

AF_PACKET is probably the most interesting row here, just for the wrong reason. We're throwing 148.6 million packets per second at the machine. The AF_PACKET program manages to receive only 2.3 million of them, while basically all of the CPU time is disappearing into kernel softirq processing.

That's the argument for bypassing the normal kernel packet path in one row, and why you can't really use it for anything that needs speed.

AF_XDP is a completely different story. It receives 146.5 Mpps, so essentially the whole 100G flood, with 11.4 cores of machine CPU.

But look at the softirq column. 7.5 of those cores are kernel work.
I think that's the really interesting comparison. If you only looked at CPU consumed by the Go program, AF_XDP would look dramatically cheaper than it actually is.

Direct Verbs and DPDK don't have that extra kernel datapath, and both receive the full 148.6 Mpps. DPDK does it with eight cores and mlx5 with ten.

Transmit

For transmit this is a single flow generated by the DUT itself.

This one still makes me smile. Three Go workers using DPDK can fill a 100G interface with minimum-sized Ethernet frames.
148.8 million packets per seconds, From Go! Pretty amazing.

The Direct Verbs backend gets there with four cores. And AF_XDP gets essentially there too, at 147.9 Mpps. But look at the CPU cost: 20 cores, with 14 cores worth of softirq.

Again, that's not necessarily bad. It's the tradeoff. With AF_XDP you keep the normal Linux driver and all the operational convenience that comes with it. DPDK and Direct Verbs have a much shorter path, so they need substantially less CPU to do the same amount of work.

Forwarding

For this test we receive a packet, do a L3 longest-prefix match, rewrite it and transmit it again.

So this is closer to the kind of thing I'm usually interested in: routers, NAT, firewalls, load balancers, etc.
The offered load is again 148.6 Mpps.

Direct Verbs gets to 147.4 Mpps, basically 100G line rate, using 12 cores.
DPDK is close behind at 143.5 Mpps using 16 cores.
AF_XDP gets to a very respectable 138.4 Mpps, but needs 28.7 cores of machine CPU to do it, 12.6 of which are softirq.

And then there's AF_PACKET :) Under the same flood it forwards essentially nothing, about 0.01 Mpps, while burning almost 50 cores in the kernel.
That's a pretty dramatic picture of why all these faster packet I/O mechanisms exist in the first place.

There are dozens of little things like queue counts, CPU placement, RSS configuration, descriptor formats and prefetching that can completely change your result.
I've spent way too much time figuring those things out. And that's another thing I want packetio to do for you: put good measured defaults in one place so you can mostly forget about them and get on with whatever you're actually trying to build.

It works on EC2 too

I also didn't want this to become one of those libraries that only works on the one server I happen to be testing on. So I've been testing packetio on EC2 as well.

The DPDK backend works with AWS ENA, and the same application code runs there unchanged. Networking in AWS is its own adventure, because the instances themselves have packet per second limits. Most of the instances I experimented with had either around 1 Mpps or 5 Mpps limits. These are AWS limits, not packetio limits. Generally the bigger the instance, the larger the networking allowance.

If you're experimenting with ENA yourself, the counter to keep an eye on is:

ethtool -S eth0 | grep pps_allowance_exceeded

If pps_allowance_exceeded starts increasing, you've hit the EC2 instance's packet-per-second allowance and AWS is queueing or dropping packets. When using the DPDK ENA driver, the same counter is available through the ENA PMD extended stats.

I also tested the DPDK backend on a 10G Intel X550. DPDK fills the wire in both directions on a single core. So we're starting to cover quite a large surface area: bare metal, EC2, ConnectX, Intel, ENA, AF_XDP and DPDK.
That's exactly what I wanted from this library.

So which one should you use?

well it depends... if I have a ConnectX card, I'm increasingly drawn to Direct Verbs. The performance is excellent and I really like keeping the normal Linux interface around.

For general modern Linux hardware, AF_XDP is a great place to start. It's fast, operationally convenient, and has much broader hardware support than the mlx5-specific backend.

DPDK is still the big hammer. It has a great driver ecosystem and decades of work behind it. There are plenty of environments where it's exactly the right choice.

And AF_PACKET is great when you just need to see some packets and don't care about moving millions of them per second.

But the thing I'm most excited about with packetio is that this choice doesn't have to dominate the design of your program anymore.

Write your packet processing once. Run it with AF_XDP. Try DPDK.
If there's a ConnectX in the server, try Direct Verbs. Then measure it.
Changing how packets reach your Go program should be a configuration decision, not a rewrite. That's really the whole idea.

I've already moved Wireblast onto packetio, and there are a few other things I'm building on top of it as well.

If you look carefully :) you'll even find a userspace TCP/IP stack based on gVisor that can run on top of these backends. I've spent waaay too much time trying to make that go fast as well. IMHO its defaults aren't exactly optimized for this kind of workload..but that's probably a blog post on its own.

If nothing else, I now finally have the packet I/O library I've wanted for the last few years. Super happy with it and also pretty proud!

It supports DPDK, which is what I originally really wanted. Along the way I got to go deep on AF_XDP and learned a whole lot about ConnectX NICs and Direct Verbs.

And it can push 148 million packets per second. Not bad for Go :)
The code is on GitHub, go give it a try, there's plenty of exampes. Also see this copy paste ready DPDK example.

That's it! Thanks for reading. Let's push some packets, and let me know what you're building.

Cheers,
Andree