If you've been following along, you know I've spent the last few months making packets go fast in Go. We looked at AF_XDP, DPDK, and most recently Mellanox Direct Verbs, all behind one library, packetio. On real hardware that ends the way you'd hope: 148 million packets per second, line rate on 100G, from a handful of Go workers.

This weekend I wondered how much of that survives on EC2. I already knew the DPDK backend worked on Amazon's ENA network interfaces, but I'd only ever tried it on small instances. Meanwhile, AWS sells network-optimized instances with numbers like 300 and 600 Gb/s on the spec sheet. So my question for today: how well do those machines actually hold up when you point a packet generator at them?

Two things I wanted out of this:

1. Take packetio for a ride on some seriously beefy machines.
2. And this is the bigger one: AWS publishes bandwidth for every instance type, but publishes nothing about packets per second. If you do anything with networking in software, you know packets per second is the number that matters. So let's go measure it.

The instances

I went up the c8gn ladder, Amazon's current Graviton network-optimized family, from the 100G size all the way to the 600G one:

These are not cheap machines, but the specs are pretty impressive! The 48xlarge pair costs about $23 an hour, so every test is designed to be short and to get the answer in one go.

How I tested

The setup was identical for every instance type:

• A pair of identical instances, same VPC, same subnet, same availability zone, talking to each other over a dedicated ENA NIC. Management traffic goes over a separate interface so nothing but test traffic touches the counters we care about.
• DPDK on both ends. The data interface is bound to vfio-pci and owned by a DPDK application; the kernel never sees a packet. I went with DPDK rather than the kernel or AF_XDP because I wanted the NIC to be the only variable.
• packetio's stock examples. The sender is blast, the receiver is drop, the same programs from the packetio repo that I used for the hardware numbers. No special EC2 build. Both are Go programs using the DPDK backend, so what you see below is what a Go application would see.
• Three packet sizes. 64 bytes for the packet rate, 1500 bytes for what a normal MTU gets you, and 9000 bytes to validate the published bandwidth.
• A queue ladder. For every instance I started with one queue on one core and doubled up until the number stopped moving. That's how you tell a per-core limit from an instance limit.
• Read the results at the receiver. The sender's transmit counter is not what gets delivered, as AWS's shaper sits after it. On one instance the sender happily reported 3.6 Mpps while 1.0 Mpps arrived. Every number here is what the other side actually received.

Two things kept me honest. Every transmit test was repeated with a ~120 line plain C DPDK sender that shares nothing with packetio, and the two always landed within a few percent of each other. And I kept a close eye on the ENA driver's allowance counters, pps_allowance_exceeded, bw_in_allowance_exceeded and friends, which the Nitro card exposes to tell you which limit you just hit. This gives me confidence the numbers aren't implementation specific. I added support for reading those from a DPDK-owned interface to packetio along the way, because without them you're guessing.

Packets per second is everything

The thing about networking in software: every packet costs roughly the same amount of work regardless of how big it is: a lookup, a header rewrite, a descriptor. A router, a firewall, a VPN gateway, a load balancer, all of them are packet per second machines first and bandwidth machines second. Bandwidth is just packets times size. So when a spec sheet says 100 Gbps, the question I actually need answered is: at what packet size?

At 64 bytes, 100 Gbps is 148.8 million packets per second. That's what the ConnectX-6 in my lab does, and it's our benchmark.

One queue, one core

Let's start small. One transmit queue, driven by one core, 64 byte packets:

About 1.5 to 2.2 million packets per second per core. On the ConnectX-6 in my lab one core does around 60 million. That's a 30x gap, and it's not packetio: the plain C sender lands in the same place. Bigger batches, deeper rings, prebuilt frames, checksum offload: I tried them all and none of it moves the number.

I don't have a definitive answer for why, but I have a suspect. ENA runs its transmit queues in what the driver docs call Low Latency Queue mode: instead of the NIC fetching your packet from memory, the driver pushes the descriptor and the first 96 bytes of every packet straight into the device over PCIe. For a 64-byte packet that's the whole thing, written by the CPU, one packet at a time. At 2.8 GHz, 2 Mpps works out to roughly 1,450 cycles per packet, against about 50 on the Mellanox card, and that's the kind of gap a PCIe write per packet would explain. You can't test the theory by turning it off: with enable_llq=0 the queue refuses to start, and DPDK's own docs say disabling it is "highly not recommended" anyway.

So on EC2, a core buys you about two million packets per second. The interesting question is what happens when you add more of them.

Adding queues

Same test, now with N transmit queues on N cores, receiver side:

It scales linearly, about 1.6 to 2.2 Mpps per queue, right up until it doesn't. The 8xlarge peaks at 16 queues and 21 Mpps and then actually gets worse with more. And the 12xlarge, 16xlarge and 24xlarge all stop at the same place: 36.6 million packets per second, at 24 queues, flat as a table from there to 64. Adding 40 more cores changes nothing. That's not a CPU limit, that's a wall, and the ENA counters confirm it: push the offer past ~36 M and pps_allowance_exceeded starts ticking on the sender.

The takeaway: on a c8gn.12xlarge or anything bigger, one network interface tops out at roughly 36 million packets per second. It doesn't matter how many queues you use, how many cores you throw at it, or whether the spec sheet says 150, 300 or 600 Gbps. 36M pps, that's the number I was looking for!

The results

Here's the complete table I wish AWS would publish. Everything measured at the receiver, 64 byte packets for the packet rate, 1500 and 9000 byte packets for bandwidth:

A few things jump out.
The bandwidth numbers are honest. With 9000 byte packets every instance hit its published number, usually a few percent over. If you move big packets, you get what you paid for.

Packets per second are a different story. 36.6 million packets per second is about a quarter of what a single 100G NIC does in my lab, on an instance sold as 150, 200 or 300 Gbps. At 64 bytes you're only using between 8 and 16 percent of the bandwidth you're paying for. That's not a complaint, it's how the platform works, but it is the number to design around, and it's not on any spec sheet.

There's one packet rate from the 12xlarge up. 150G, 200G, 300G, one network card of the 600G: all ~36 Mpps. The extra bandwidth and the extra cores buy you nothing for small packets. Per gigabit that's 244 kpps on the 12xlarge, 183 on the 16xlarge and 122 on the 24xlarge. If your workload is small packets, the 12xlarge is the sweet spot of this family, and the bigger sizes only make sense when the packets get big too.

Where does bandwidth take over? Since the packet rate is flat, the bandwidth you actually get is simply the packet rate times the packet size, until you hit the published number. I ran a size ladder to find where that happens. The 8xlarge fills its 100G at about 640 bytes. The 12xlarge fills 150G already at 512 bytes. The 16xlarge needs about 700 bytes. The 24xlarge doesn't get there at 1500 bytes at all: 282 Gbps, and that's the sender's NIC topping out, not AWS. Same for the 48xlarge. Above 200G, you need jumbo frames to see the bandwidth you're paying for.

600 Gbps needs two network cards

The 48xlarge was the one I was most curious about, and it came in at exactly the same numbers as the 24xlarge: 35 Mpps, 282 Gbps at 1500 bytes, 304 Gbps with jumbo frames. Half the published bandwidth, and this time with bw_out_allowance_exceeded firing on the sender. AWS was shaping me at 300G on a 600G instance.

The reason: to get 600 Gbps on the 48xlarge you need to use two network cards, each with its own 300 Gbps allowance. Normally the published bandwidth is per instance and every ENI shares it; here it's per card. To get 600 you need two interfaces, one attached to each card (--network-card-index 1 for the second), and something spreading traffic over both. There's no bonding layer doing that for you. Attach one interface, run iperf, and you'll measure a 300G machine.

In hindsight this makes sense. The fastest NICs you can commonly buy today are 400G; anything beyond that is two of them. It would have been odd for AWS to have a single 600G device.

It is documented, sort of. Someone on Twitter kindly pointed me at the network specifications table for compute optimized instances, which has a "Network cards" column, and the c8gn.48xlarge says 2. Not a page you naturally end up on, and nothing there says one interface gets half.

It probably also means the packet rate in the table is per card, so both cards together would be good for ~70 Mpps. I didn't test that (the pair was expensive enough for one weekend), so treat it as a well-founded guess.

What I didn't test

AWS documents a burst and baseline mechanism for bandwidth on the smaller instance sizes. The c8gn sizes I tested all have a fixed bandwidth, and my runs were short by design, a few minutes each, so I have nothing to say about credit buckets here. I did keep the allowance counters in view the whole time and saw no sign of a packet-rate bucket in any run. If you run these instances at full blast for an hour, tell me what you see.

Things fixed along the way

As usual, running on new hardware finds bugs. packetio's DPDK backend now builds on arm64 (all of these are Graviton machines, and the mempool header alignment differs from x86), it can read the ENA allowance counters through DPDK's xstats, and the CPU pinning code learned about hyperthreads: on an Intel instance it used to place one worker per physical core and leave half the vCPUs idle. All of that is in the repo.

Wrap-up

So, how fast can you go on AWS?

• Bandwidth: as fast as it says on the box. Every published number held, as long as the packets are big enough, and above 200G that means jumbo frames. On the 48xlarge it also means two network interfaces.
• Packets per second: about 36 million per network interface, from the 12xlarge up. That's the number AWS doesn't publish, and it's about a quarter of what one 100G NIC does on bare metal.
• Per core: about 2 million pps. Roughly 30x more expensive per packet than a ConnectX. You need 24 cores just to push enough packets to find the instance limit.

So we got the numbers we were after, and packetio got better along the way. If you're building network functions on EC2, you now have the figure that was missing, and it's the one to pick your instance by: know your packet size first. For small packets the budget is one flat number across a 4x range of prices, and a single 100G ConnectX in a bare-metal box does four times the packets of a c8gn.24xlarge for a lot less money.

One more catch, and it has nothing to do with packets. Everything here ran inside one availability zone, where traffic is free. Cross an AZ and it's $0.01 per GB each direction. 300 Gbps is 135 TB an hour, so a 48xlarge at its rated bandwidth costs about $1,350 an hour in transfer, on top of $11 for the instance. Serious networking on AWS needs a serious credit card, and it's the traffic that runs it up. That's a post of its own.

The tooling is in the packetio repo; the blast and drop examples are all you need to reproduce any row in the table. If you run this on an instance family I didn't, I'd love to see the numbers.

That's it, thanks for reading! Pushing packets on AWS is now a little clearer :)

Cheers
-Andree

Meet packetio, the Go packet I/O library I wish I'd had a years ago. AFXDP, DPDK, mlx5 Direct Verbs, and AFPACKET behind one API.

The last few months I've reignited my strange hobby of doing network packet processing in software. My language of choice is Go, and I'm always looking and learning how to do this as fast as possible.

A few weeks ago I released go-afxdp, a Go library for sending and receiving packets using AF_XDP. Then I built Wireblast on top of it and got that to 100G line rate.

I've also been playing quite a bit with DPDK, different NICs, and more recently NVIDIA/Mellanox Direct Verbs.

It's been a fun rabbit hole :) But while doing all of this, I kept running into the same thing.
The actual packet processing is usually the easy part. Receive some packets, inspect them, maybe change a few bytes, do a lookup, and send them back out.

Getting packets efficiently between the NIC and your Go program is where all the complexity lives. And depending on how you want to do that, you end up in completely different worlds.

AF_XDP has UMEM and rings. DPDK has mbufs, mempools, EAL, hugepages and PMDs. Mellanox ConnectX cards have yet another path using Direct Verbs. All complicated words most of us probably don't want to deal with.

That's what I've been working on. One API to rule them, It's called packetio.

Four ways to get at a packet

There are basically four ways I care about for getting raw packets in and out of a Go program on Linux.

There's AF_PACKET, which is the normal Linux packet socket. It's what tools like tcpdump use. It works almost anywhere and requires very little setup, but it's slow as packets still go through much of the normal kernel machinery.

Then there's AF_XDP, which I've written about quite a bit already. It lets you intercept packets very early in the Linux driver and move them into memory shared with your application. On supported drivers it can be zero-copy, so the NIC writes packets directly into memory your Go program can access.

The part I really like about AF_XDP is that Linux still owns the interface. You can send only the traffic you care about to your program and your SSH session, ARP, routing protocols and everything else keep working normally.

Then there's DPDK.
DPDK is still the obvious answer when someone says "I want to do packet processing as fast as possible in userspace." It supports a large number of NICs and has been optimized for this job for years.

It's also a bit of a world of its own. If you've ever started a DPDK application from scratch, you know what I mean. Hugepages, EAL arguments, mbufs, mempools, PCI devices, queue setup. None of it is particularly impossible, but there's a huge learning curve.

And then there is the fourth option, which is the one I only really started appreciating recently: mlx5 Direct Verbs. More on that one in a bit.

One API over all of them

The idea behind packetio is pretty simple. A device has RX queues and TX queues. Queues give you frames. You process those frames in batches and give them back.
That's about as much as I want the application to know.

For example:

dev, err := mlx5.Open("eno2", mlx5.WithQueues(4))

Or afxdp:

dev, err := afxdp.Open("eno2", afxdp.WithQueues(4))

Or DPDK:

dev, err := dpdk.Open("0000:c1:00.1", dpdk.WithQueues(4))

There's also:

dev, err := afpacket.Open("eno2", afpacket.WithQueues(4))

After that, the packet loop is the same. Something like:

rx := dev.RxQueue(0)
for {
    rx.Fill(rx.NumFreeFillSlots())
    rx.Poll(time.Second)

    descs := rx.Receive(256)

    for _, desc := range descs {
        pkt := rx.Region().Frame(desc)
        // Do something useful with the packet.
        process(pkt)
    }
    rx.Recycle(descs)
}

And transmitting is similarly boring:

tx := dev.TxQueue(0)

tx.SendFunc(256, func(_ int, frame []byte) int {
    return copy(frame, myPacket)
})

That's really all there's to it! You still have queues and batches because those things matter when you're doing high-speed packet processing. I'm not trying to hide reality behind some net.Conn abstraction.

But I also don't think you should need to understand an AF_XDP fill ring, an rte_mbuf, or an mlx5 WQE before you can get a packet onto the wire.
Changing the backend should mostly mean changing how you open the device.
That sounds like a relatively small thing, but in practice I think it's pretty powerful.

DPDK from Go, finally

One of the original motivations for this project was actually DPDK.
I've wanted a good way to use DPDK from Go for years.

There have been Go wrappers before, but I never found one that gave me the programming model I wanted. I don't want to write my packet processing in C, and I definitely don't want to cross from Go into C for every packet.

packetio crosses into DPDK once per batch, not once per packet. The packet memory stays in DPDK's mempool and Go works directly on those bytes.
So you get to write what looks like a pretty normal Go program while DPDK does all the ugly setup underneath.

No rte_eth_rx_burst scattered through your application. No mbuf ownership rules in your business logic. No giant EAL argument string you copied from Stack Overflow three years ago. And if you don't know what that means, congrats, you dont need to!

You open the device and start receiving packets, all in Go.That's the DPDK experience I've wanted for a long time.

Direct Verbs was the surprise

The most interesting part of this project for me has probably been learning about mlx5 backend and something called direct verbs.

Jérôme Tollet has been doing some great work in VPP around NVIDIA/Mellanox ConnectX NICs. He recently published a comparison of VPP's three high-performance datapaths: AF_XDP, DPDK mlx5 and native RDMA Direct Verbs.

That sent me down another rabbit hole. It turns out that with a ConnectX NIC you can talk to the hardware queues directly using mlx5 Direct Verbs.

So instead of:
Go > DPDK > mlx5 PMD > NIC
you can do:
Go > mlx5 queues > NIC

There is obviously quite a bit hiding behind that sentence, but from the application's perspective that's the useful mental model.

And it has another property I really like. ConnectX NICs support a bifurcated model. Linux can keep its normal interface while hardware steering sends selected traffic directly into queues owned by your application.

So eno2 doesn't disappear. SSH keeps working. Linux can still use the interface for ordinary traffic while your Go program handles the packets you've claimed.

ConnectX can do this with DPDK too, but Direct Verbs lets us skip the DPDK layer entirely. For what I've been building lately, that's a pretty compelling combination.
AF_XDP-like operational behavior, but with a very short userspace datapath.

How fast is it?

Yes, the big question! I benchmarked all of this! I actually spent quite a bit of time on these numbers, so I'm going to put them all here.

Hopefully they're useful to someone else comparing these different approaches. And if nothing else, I know I'll be coming back to this blog myself six months from now trying to remember how DPDK compared to AF_XDP :)

The setup is two AMD EPYC 9275F machines with NVIDIA ConnectX-6 Dx 100G NICs. I'm testing 64-byte packets, which is really the worst case for packet processing. At that size, 100G Ethernet is about 148.8 million packets per second.

That's one packet every 6.7 nanoseconds. That's very little time. So yeah, things like memory accesses and extra trips through the kernel start to matter.

The nice thing about this new packetio library is that all four backends are running essentially the same userspace benchmark program. Same receive loop, same transmit loop, same forwarding code. The backend is the only thing that changes.

I'm also measuring CPU across the whole machine, not just the Go process. That's particularly important for AF_XDP. Quite a bit of its work happens inside the kernel in IRQ/NAPI/softirq context. If you only measure your Go process, those cores magically disappear from the benchmark even though you're still paying for them.
Jérôme took the same approach in his VPP measurements, which is one of the things I liked about his comparison.

Receive

For receive I'm offering 148.6 Mpps across 4096 flows.

AF_PACKET is probably the most interesting row here, just for the wrong reason. We're throwing 148.6 million packets per second at the machine. The AF_PACKET program manages to receive only 2.3 million of them, while basically all of the CPU time is disappearing into kernel softirq processing.

That's the argument for bypassing the normal kernel packet path in one row, and why you can't really use it for anything that needs speed.

AF_XDP is a completely different story. It receives 146.5 Mpps, so essentially the whole 100G flood, with 11.4 cores of machine CPU.

But look at the softirq column. 7.5 of those cores are kernel work.
I think that's the really interesting comparison. If you only looked at CPU consumed by the Go program, AF_XDP would look dramatically cheaper than it actually is.

Direct Verbs and DPDK don't have that extra kernel datapath, and both receive the full 148.6 Mpps. DPDK does it with eight cores and mlx5 with ten.

Transmit

For transmit, the DUT (machine under test) sends as fast as it can from N queues. Every packet carries the same 5-tuple, since varying it only matters when something is receiving.

This one still makes me smile. Three Go workers using DPDK can fill a 100G interface with minimum-sized Ethernet frames.
148.8 million packets per seconds, From Go! Pretty amazing.

The Direct Verbs backend gets there with four cores. And AF_XDP gets essentially there too, at 147.9 Mpps. But look at the CPU cost: 20 cores, with 14 cores worth of softirq.

Again, that's not necessarily bad. It's the tradeoff. With AF_XDP you keep the normal Linux driver and all the operational convenience that comes with it. DPDK and Direct Verbs have a much shorter path, so they need substantially less CPU to do the same amount of work.

Forwarding

For this test we receive a packet, do a L3 longest-prefix match, rewrite it and transmit it again.

So this is closer to the kind of thing I'm usually interested in: routers, NAT, firewalls, load balancers, etc.
The offered load is again 148.6 Mpps.

Direct Verbs gets to 147.4 Mpps, basically 100G line rate, using 12 cores.
DPDK is close behind at 143.5 Mpps using 16 cores.
AF_XDP gets to a very respectable 138.4 Mpps, but needs 28.7 cores of machine CPU to do it, 12.6 of which are softirq.

And then there's AF_PACKET :) Under the same flood it forwards essentially nothing, about 0.01 Mpps, while burning almost 50 cores in the kernel.
That's a pretty dramatic picture of why all these faster packet I/O mechanisms exist in the first place.

There are dozens of little things like queue counts, CPU placement, RSS configuration, descriptor formats and prefetching that can completely change your result.
I've spent way too much time figuring those things out. And that's another thing I want packetio to do for you: put good measured defaults in one place so you can mostly forget about them and get on with whatever you're actually trying to build.

It works on EC2 too

I also didn't want this to become one of those libraries that only works on the one server I happen to be testing on. So I've been testing packetio on EC2 as well.

The DPDK backend works with AWS ENA, and the same application code runs there unchanged. Networking in AWS is its own adventure, because the instances themselves have packet per second limits. Most of the instances I experimented with had either around 1 Mpps or 5 Mpps limits. These are AWS limits, not packetio limits. Generally the bigger the instance, the larger the networking allowance.

If you're experimenting with ENA yourself, the counter to keep an eye on is:

ethtool -S eth0 | grep pps_allowance_exceeded

If pps_allowance_exceeded starts increasing, you've hit the EC2 instance's packet-per-second allowance and AWS is queueing or dropping packets. When using the DPDK ENA driver, the same counter is available through the ENA PMD extended stats.

I also tested the DPDK backend on a 10G Intel X550. DPDK fills the wire in both directions on a single core. So we're starting to cover quite a large surface area: bare metal, EC2, ConnectX, Intel, ENA, AF_XDP and DPDK.
That's exactly what I wanted from this library.

So which one should you use?

well it depends... if I have a ConnectX card, I'm increasingly drawn to Direct Verbs. The performance is excellent and I really like keeping the normal Linux interface around.

For general modern Linux hardware, AF_XDP is a great place to start. It's fast, operationally convenient, and has much broader hardware support than the mlx5-specific backend.

DPDK is still the big hammer. It has a great driver ecosystem and decades of work behind it. There are plenty of environments where it's exactly the right choice.

And AF_PACKET is great when you just need to see some packets and don't care about moving millions of them per second.

But the thing I'm most excited about with packetio is that this choice doesn't have to dominate the design of your program anymore.

Write your packet processing once. Run it with AF_XDP. Try DPDK.
If there's a ConnectX in the server, try Direct Verbs. Then measure it.
Changing how packets reach your Go program should be a configuration decision, not a rewrite. That's really the whole idea.

I've already moved Wireblast onto packetio, and there are a few other things I'm building on top of it as well.

If you look carefully :) you'll even find a userspace TCP/IP stack based on gVisor that can run on top of these backends. I've spent waaay too much time trying to make that go fast as well. IMHO its defaults aren't exactly optimized for this kind of workload..but that's probably a blog post on its own.

If nothing else, I now finally have the packet I/O library I've wanted for the last few years. Super happy with it and also pretty proud!

It supports DPDK, which is what I originally really wanted. Along the way I got to go deep on AF_XDP and learned a whole lot about ConnectX NICs and Direct Verbs.

And it can push 148 million packets per second. Not bad for Go :)
The code is on GitHub, go give it a try, there's plenty of exampes. Also see this copy paste ready DPDK example.

That's it! Thanks for reading. Let's push some packets, and let me know what you're building.

Cheers,
Andree

If you’ve followed my blog for a while, you know I love tinkering with packets and making them go fast in software. Today I'm releasing a new tool called wireblast, a blazingly fast network packet generator.

A few weeks ago, I wrote about go-afxdp, a small Go library I've been working on for sending and receiving Ethernet frames at line rate using AF_XDP, without DPDK and without leaving Go.

I was pretty happy with how it turned out. But a library is just a library; the real value is in how you use it. So the next question is, what can we build with it?

I have a few things cooking, all about fast packet forwarding on Linux. But to know whether a fast forwarding path is actually fast, you need to throw traffic at it. A lot of traffic! And you need to trust the numbers. So I built a packet generator. It’s called Wireblast.

The gap iperf leaves

For network performance testing, most of us reach for iperf3. I do too. It’s easy, widely available, and perfect for answering, “How many gigabits per second can these two hosts push?”

But bandwidth is only part of the story. Packets per second matter just as much when testing routers, firewalls, load balancers, or anything else that does work for every packet.

That’s where iperf3 (or really the kernel) hits a wall. Its traffic still passes through the Linux networking stack, which becomes the bottleneck long before the wire does. At 10G, line rate for 64-byte frames is 14.88 Mpps. At 100G, it’s 148.8 Mpps. You cannot reach those numbers with Linux kernel, in fact you may only get to 10% of the 100G capability and small packets sizes.

For small packets, high packet rates, or lots of flows, you traditionally need something like T-Rex, DPDK Pktgen, or Ixia. Those tools are powerful, but they also bring DPDK, hugepages, dedicated interfaces, extra setup, and sometimes a purchase order. It's a lot of work and complicated.

That middle row is the idea behind Wireblast: nearly as easy to use as iperf3, but built for the point where packets per second not just bits per second, become the limiting factor.

Meet Wireblast

Wireblast is a single, static, 14 MB Go binary. There’s no DPDK and no complicated setup. You download it, run it, and start sending (or receiving) packets.

Run it without arguments and you get a small interactive wizard. Or provide everything as command-line flags, which makes it easy to use from scripts and automated tests.

The code is available on GitHub, and the documentation is at wireblast.mintlify.site.

Under the hood, Wireblast uses AF_XDP to bypass most of the normal kernel networking path when transmitting packets. That makes it possible to reach very high packet rates: 14M pps on 10G hardware and close to line rate, 138M pps packets per second on 100G NICs.

Let’s send some packets

Grab the latest binary:

curl -sSL https://github.com/atoonk/wireblast/releases/latest/download/wireblast_linux_amd64.tar.gz \
  | tar xz

sudo install -m 0755 wireblast /usr/local/bin/
wireblast --version

There’s an arm64 build too, or you can install it using Go:

go install github.com/atoonk/wireblast/cmd/wireblast@latest

Now let’s start with the hardest thing you can ask a NIC to do: tiny packets.

Small frames are the classic stress test because you’re bottlenecked on packets per second rather than bits per second.

sudo wireblast --start --yes -i eno2 --vlan 2043 \
   --src-ip 192.168.43.1 \
   --dst-ip 192.168.43.2 \
   --dst-mac 7c:c2:55:be:f4:1b \
   --packet-size 64 \
   --flows 100 \
   --pps unlimited -d 30s

UDP is the default, so that’s the whole command. Of course you only need the vlan flag when running on tagged vlans.

On a 10G capable nic, that's 14.88 million packets per second: line rate for 10G, from a Go program using only a few CPU cores. I still think that’s kind of amazing.

IMIX and PCAP replay

Fixed-size packets are great for finding the ceiling, but real traffic contains a mix of packet sizes.

Wireblast includes the classic IMIX profile: a 7:4:1 ratio of 64-, 594-, and 1518-byte frames.

sudo wireblast -i eth1 \
  --dst-ip 192.0.2.10 \
  --mode imix \
  --flows 64 \
  --pps 200k \
  --start

The 64 flows vary the source ports, which is useful when testing anything involving RSS, ECMP, link aggregation, load balancers, or connection tables.

You can also replay PCAP and PCAPNG files:

sudo wireblast -i eth1 \
  --pcap capture.pcap \
  --pps 100k \
  --start

By default, Wireblast loops the capture at your requested rate. It can also reproduce the original packet timing:

sudo wireblast -i eth1 \
  --pcap capture.pcap \
  --pcap-timing original \
  --pcap-loop=false

For automation, --no-tui skips the wizard and dashboard and prints plain-text statistics once per second:

sudo wireblast --no-tui \
  -i eth1 \
  --dst-ip 192.0.2.10 \
  --pps unlimited \
  -d 30s \
  -y

[0:05] tx 74.4 M pkts  14.88 Mpps  L1 10 Gbit/s  L2 7.62 Gbit/s  avg 64B

Everything is scriptable, and there isn’t much to learn.

Going big: 100G

10G is fun, but I had to know how far this thing would go.
So I put Wireblast on a pair of machines with 100G Mellanox ConnectX NICs using the mlx5 driver and let it rip. (By the way, I’ve been spending waaay too much money on Latitude.sh for this project, but I highly recommend them for renting bare-metal machines with a second NIC for network testing.)

Below is a test with 80 bytes packets.  98Gb/s at 122Mpps! nice!

That’s 99% of tagged line rate for minimum-size frames on a 100G link, from one Go binary. And it wasn’t limited to tiny packets:

The queue-scaling results surprised me even more. With large frames, one queue already pushed 83 Gbit/s. Two queues nearly saturated the link.

You don’t need a monster machine. You need a decent NIC and a couple of CPU cores, which is a wild thing to be able to say about a software packet generator.

Where Wireblast fits

Wireblast is meant to cover the easy 80%. It is not trying to replace an Ixia.

It counts packets and bytes, but it does not measure latency or jitter, and it does not support hardware timestamping. Its TCP mode generates stateless SYN packets rather than complete TCP sessions.

That makes it useful for testing:

• Packet-forwarding performance
• Firewalls and ACLs
• Connection tables
• RSS and queue distribution
• ECMP Link aggregation
• Load balancers
• Packet-processing software

When I want to know, “Can this path do line rate, and does the far end see the packets?”, Wireblast answers that question in about 30 seconds.

Wrap-up

Building Wireblast was fun. More importantly, building it on top of go-afxdp was easy.

Claude Code also made building the TUI surprisingly quick. That’s a nice thing to be able to say about both the library and the current state of software development.

For years, software traffic generation meant choosing between an easy tool limited by the normal kernel path and a powerful tool that pulled you into DPDK, out dated howto's and specialized test setups.

Wireblast is the middle ground I wanted: download one binary, point it at an interface, and start sending packets at line rate.

No hugepages. No NIC surgery. No DPDK expertise required. And let’s be real: what’s more fun than sending packets at line rate?

Fourteen million packets per second on a normal 10G box. More than 100 million packets per second on a 100G NIC. All from one command, with almost no setup required. Pretty amazing!

Next up, now that I have a reliable traffic generator, I want to get back to the project that started all of this: a completely Go-based dataplane that bypasses the kernel for routing, switching, and firewalling.

Now I finally have a good way to measure it. Stay tuned!

The code is on GitHub, the documentation is at wireblast.mintlify.site, and it’s all built on go-afxdp.

That’s it :) you made it to the end. Thanks for reading!

Cheers,
Andree

If you've been following my blog for a while, you know I have a soft spot for making packets go fast in software. I'm a network guy at heart, raised on routing and switching, but these days most of that love is written in Go.
Every few years I circle back to the same question: how fast can we send and receive packets on a plain Linux box? We've explored this with DPDK, with XDP-based routers, and with the kernel itself. Today we're going a level deeper into my favorite of the bunch: AF_XDP, using a Go library I wrote to make it easy: go-afxdp .

If you want to skip ahead and see it in action, the blast and drop examples in the repo are a great place to start.

Some background

A while ago I wrote this article: High-Speed Packet Transmission in Go: from net.Dial to AF_XDP. In that article we worked through every way Linux gives you to send a packet from Go: net.Dial, raw sockets, AF_PACKET, pcap. At the end of that journey sat AF_XDP, and it beat everything else by a mile.

That was my first hands-on experience with AF_XDP, through a third-party library asavie/xdp. It showed off the potential, but it was quite low-level and, honestly, a bit fragile; some of you who tried it on real hardware ran into panics I couldn't fix from my side. So, building on that code, I wrote my own library. The goal: keep the raw speed, make it higher level and more reliable, and have it do the right thing for whatever hardware or cloud you're on, without you needing to understand every knob under the covers.

So, what is AF_XDP?

AF_XDP is a socket type in Linux, like the ones you already know. The mental model is close to AF_PACKET (the thing tcpdump is built on): instead of asking the kernel to "send this payload to this IP and port," you hand it complete Ethernet frames, MAC addresses and all, and you receive complete Ethernet frames back. You're operating at layer 2.

The difference is how much of the kernel you skip on the way to the wire. AF_PACKET still takes a fairly scenic route through the network stack. AF_XDP takes the express lane: your program and the kernel share a chunk of memory (the UMEM), carved into packet buffers, and packets move through lock-free rings in that shared memory. On a modern NIC and driver it goes one step further with zero-copy mode: the NIC's DMA engine writes incoming packets directly into your program's memory and reads outgoing packets straight from it. No copies, no socket buffers, no per-packet syscalls.

How fast is that in practice? On a bare-metal machine with a 10G Intel NIC, I measured three methods, all sending small 64-byte packets. Plain kernel UDP sockets, one sendto() per packet, topped out around 3 million packets per second no matter how many workers I threw at it. Adding UDP GSO trickery (UDP_SEGMENT, where the kernel slices one big send into many packets) more than doubled that to almost 8 million. And go-afxdp? ~13.5 million packets per second, which is virtually 10G line rate, with the receiving side keeping up without dropping a packet. At that point the wire is the bottleneck, not the software. I still think that's kind of amazing for a Go program you can go build like any other.

These numbers come from go-pktgen, my packet generator that benchmarks all the send methods side by side; I've updated it so its af_xdp mode now uses this new library. Check that repo if you want the full comparison table. Btw, AF_XDP shines even more on the RX path than TX.

The best part: you don't lose your NIC

When you open an AF_XDP socket with go-afxdp, the library compiles a small eBPF program and attaches it to the interface as an XDP program. XDP programs run inside the driver, for every incoming packet, before the kernel stack ever sees it (the same technology behind my XDP-based BGP peering router  from a few years back). The program's job here is simple: if a packet matches the filter you asked for, redirect it into your socket's ring. That's it.

The key word is if. Everything that doesn't match is handed to the kernel stack, completely unaffected. Say you only want UDP port 9999: those packets land straight in your Go program, while your SSH session, DNS lookups, and everything else on the interface keep flowing like nothing happened. You can be surgical about exactly which traffic takes the fast path. This is exactly where DPDK hurts: it takes over the entire NIC, and the interface disappears from the kernel, SSH session and all. With AF_XDP the NIC stays a normal Linux interface.

What it looks like in Go

This is what receiving packets looks like. Open the interface with a filter, and you get one socket per NIC queue:

fleet, err := afxdp.Open("eth0", afxdp.WithUDPPorts(9999))
if err != nil {
	log.Fatal(err)
}
// Info() tells you what you actually got: queues, XDP mode, zero-copy, driver.
// e.g. "eth0: 12 queues, zero-copy, native XDP, 8192x2048B frames, driver ixgbe"
info, _ := fleet.Info()
log.Printf("started: %s", info)
// One receive loop per queue: Fill -> Poll -> Receive -> Recycle.
for _, xsk := range fleet.Sockets() {
	go func(xsk *afxdp.Socket) {
		for {
			xsk.Fill(xsk.NumFreeFillSlots()) // hand the kernel empty buffers
			n, err := xsk.Poll(-1)           // block until packets arrive
			if err != nil {
				return
			}
			descs := xsk.Receive(n)
			for _, d := range descs {
				packet := xsk.GetFrame(d) // the raw Ethernet frame ([]byte)
				// ... do something with it ...
			}
			xsk.Recycle(descs) // give the buffers back for reuse
		}
	}(xsk)
}

Pretty simple, right? No BPF to write, no UMEM math, no ring management. The Fill -> Poll -> Receive -> Recycle loop is the whole programming model, and transmit is the same idea in reverse. This snippet is essentially the drop example, a UDP sink that counts how fast it can receive; its counterpart blast is the transmit side, a UDP flooder that fills every TX queue. Those two together are how I got the numbers in this article.

Notice the Info() line. AF_XDP can run in very different modes: native (in the driver) or generic (a slower fallback that works anywhere, even on veth), with zero-copy or copy. The difference between them is huge. The library picks the fastest mode your driver supports and then tells you what you got.

It should just do the right thing

That mode business is where my old frustrations came from, and where I spent the most effort this time. Whether you get the fast path depends on your driver, your queue configuration, even your MTU. That's a lot of tribal knowledge to expect from someone who just wants to receive or send packets quickly.

A concrete example: I recently tested on AWS, on a pair of c7gn.xlarge instances. Out of the box, AF_XDP on the ENA driver quietly falls back to generic mode, and in generic mode my receiver silently lost about 25% of the packets, with no counter anywhere admitting to it. Getting the fast path on ENA requires some unobvious things, including page-sized (4096-byte) packet buffers instead of the usual 2048. Miss that one and zero-copy silently doesn't engage. The library now detects the ENA driver and handles it automatically, so the stock examples come up as "zero-copy, native XDP" on EC2 with no code changes. The same test then ran at a clean, steady 5 million packets per second, completely lossless, at which point you hit AWS's per-instance packets-per-second allowance. But hey, that means the software is no longer the bottleneck.

That's the philosophy of the library, you shouldn't need to know that one particular AWS driver wants page-aligned frame sizes. It should just do the right thing.

How does it compare to DPDK?

Honestly, the performance is in the same ballpark, and that surprised me. DPDK is still the gold standard for absolute numbers, but the gap has become small and the price difference is enormous. With DPDK you lose the whole NIC, you're tied to specific drivers and a heavy build-and-tuning universe, and there's effectively no good Go story. AF_XDP works on any modern Linux kernel, keeps the NIC as a normal interface, lets you pick which traffic to bypass, and is one go get away. For me it's become the sweet spot: 95%+ of the performance at 10% of the operational pain.

Where this fits (and where it doesn't)

The cases where AF_XDP shines is anything that's packet-in, packet-out at high packet rates:

• Packet generators and performance testing. the blast and drop examples together are a poor man's IXIA, the same itch I used to scratch with DPDK's pktgen, now in a couple hundred lines of Go.
• UDP-based servers and protocols.  There's a DNS server example in the repo that answers queries straight from the AF_XDP path. Tunnels, VXLAN, QUIC-adjacent experiments: anything UDP-shaped fits naturally.
• Network scanners, IDS-style monitoring, DDoS scrubbing. Anywhere you want to see a lot of packets cheaply, for a specific slice of traffic, without taking the box off the network.
  

And where it doesn't fit: TCP. With AF_XDP you're getting raw Ethernet frames, so TCP means bringing your own TCP stack in userland, retransmissions and congestion control included. The kernel gives you a world-class TCP implementation for free, and my strong suspicion is that a userland stack bolted onto AF_XDP would give back most of the performance while adding a pile of complexity. If your workload is TCP-shaped, use the kernel. AF_XDP is for when your workload is packets.

Wrap-up

To me, a reliable easy to use AF_XDP library felt like a missing piece in Go for years: kernel sockets are easy but top out in the low millions of packets per second, and DPDK is fast but heavy, using C, and takes your NIC away. AF_XDP sits right in between: a Go program, built with the normal toolchain, sending and receiving at 10G line rate with a couple of CPU cores, while the rest of the box carries on like normal.

The code is at github.com/atoonk/go-afxdp. If you try it on hardware I haven't, I'd love to hear about it, especially the failure cases. That's how the ENA quirks got found and fixed.

That's it; you made it to the end! Thanks for reading! Let's push some packets :)
Cheers
-Andree

One of my great passions is infrastructure operations. My first few jobs were in network engineering, and later on, as the cloud became more prevalent, that turned into what we now call infrastructure or cloud operations. Probably the most stressful times were during outages. Over the last two decades, I've been part of many outages, some worse than others, but I wear my operational battle scars proudly. In this blog, I'll share some learnings and observations from these incidents.

What triggered me to write this article

I could write many articles about the "epic" incidents I've been part of. While stressful at the time, they make for good war stories afterward. In recent years, companies like Facebook and Cloudflare have started publicly sharing their outage retrospectives (timelines, root causes, and lessons learned), which I enjoy reading. It's free learning!

A few days ago, Rogers posted its public incident summary, covering the typical aspects of an outage retrospective. The outage itself was significant. Rogers is Canada's second-largest ISP and cell phone carrier. Two years ago, they experienced a country-wide 24-hour outage that impacted all their phone (including 911) and internet customers. Yes, hard down for a full day! So naturally, as both a Rogers customer and internet operations enthusiast, when they posted their report (2 years later!), I had to read it.

The Rogers outage

Though not the primary focus of this article, let's briefly review the outage. During planned maintenance, Rogers staff removed a network routing policy, which caused (BGP, or OSPF/ISIS) routes to be flooded internally. This overwhelmed the control plane, causing all kinds of challenges, eventually causing it to crash or otherwise render its core routers inoperable. A logical next step would be to try and undo that change, however, the routers themselves were no longer manageable due to the control plane issues, and to make matters worse, the out-of-band (OOB) management network was also down. As it turns out, the out-of-band network somehow depended on the production data network, so when that went down, so did the out-of-band network, making troubleshooting and remediation impossible. Eventually, engineers had to be dispatched to manually get console access.

Thoughts on the Rogers outage

Outages suck. They’re super stressful and, as the Rogers incident shows, can have significant real-world impacts. Having these retrospectives after the fact, and getting to the root cause, lessons learned, etc., is mandatory for any organization that wants to continuously improve itself.

One thing that stood out for me was the news article from CBC.ca (a major news outlet in Canada), the headline of which was: “Human error triggered massive 2022 Rogers service outage, report finds.” While it's arguably true that the change made kicked off a sequence of events, declaring that as the root cause is short-sighted. Many of these outages are like airplane crash investigations, where multiple factors contribute to the failure, sometimes starting years ahead or being poorly designed to start with.

Reflecting on Outages: Can We Eliminate Them? Should We Want To?

It’s always easy to dunk on an outage from the sidelines and to be clear, that’s not my intent. I’ve learned firsthand that Shit happen! So, with the Rogers outage in mind, but also from past experience, let’s extract some generic lessons.

First off, I’d say that the goal of eliminating outages altogether is likely too ambitious for most of us. Yes, part of an outage retrospective process should be to prevent the exact same outage from happening in the future, but architectures evolve, technologies change, and even a slight change in parameters can lead to a similar outage again.

So, although we can limit the chance of outages, you won’t be able to eliminate them 100%. In fact, given the cost of complete outage elimination and depending on your industry (say Facebook vs. flying a passenger airplane), you may not necessarily want to aim for that. There are real financial costs, process costs, agility costs, or time-to-value costs by trying to completely achieve outage elimination.

So, let’s accept that outages will happen and instead use that insight to focus on limiting their damage and impact! This approach allows us to prioritize effective strategies, which I'll discuss next.

Time to Detection - aka Be The First To Know!

Sufficient monitoring helps to reduce time to detect and pinpoint root causes. Sounds obvious, right? Yet, almost every outage retrospective will have an improvement action item called “more monitoring.” That’s an indication that we’re always forgetting some metric for some reason we never appear to have all the checks and metrics. That’s why I’m a big fan of synthetic monitoring for availability.

This check should mimic a typical user experience. So even if you swap out a core component, you’ll still be able to see the impact without adding more metrics. Because you can’t easily miss something, it’s the first check you should add, and most importantly, should alert on. So whether it’s a ping or a simple HTTP test, this will give you instant confidence your service is working. I call this the “be the first to know” principle. You don’t want to hear from your customers that your service is broken; you’re responsible for it, and you should be the first to know!

Limit blast radius.

Can we contain the impact when something breaks? The Rogers outage is a prime example of the worst case: one change broke the entire network. Logical separation, whether geographical or functional, could have prevented this. Modern-day cloud providers like AWS actively encourage us to think about this. With AWS, you’re always thinking about Multi-AZ or Multi-region. Some go further by going multi-cloud (think multi-vendor in network land).

However, transitioning from a single region to multi-region or multi-cloud can become exponentially complex and costly. The same is true in network land, where the path from a single vendor to multi-vendor can become quite complex easily.

Certain services are easier to make multi-region than others (I’m not even thinking about multi-cloud yet). Take your core database, for example. This comes with real complexity costs, and it's understandable why this might not always happen.

I get it, infrastructure teams are often considered cost centers. While high availability and reliability are touted as priorities, urgent needs, feature development, and cost pressures often take precedence. However, never let a good disaster go to waste! After a significant outage, budgets and priorities often shift. This is your opportunity to advocate for investments in resilience. The value proposition is strongest in the immediate aftermath, so seize the moment.

Management access

The outages that still haunt me and gave me my “Ops PTSD” are those where we lost management access during the outage. This is what happened to Rogers, and a similar incident happened to one of Facebook’s largest outages. It’s a real nightmare scenario: you can’t troubleshoot or remediate the issue since you’ve lost the ability to do anything to the device. Often even the ability to just shut it down or otherwise take it out of rotation. Until you restore some form of management access to the device, you’re stuck.

This problem is more present for folks running physical hardware than those running, say, on AWS. So, if you run your own data centers, never compromise on out-of-band network access, without thinking about the worst case. Make sure you have OOB access and that it is isolated from your in-band network access. It's not always easy, but if you don't have this, your outage time will go from minutes to many hours (see Rogers and Facebook's example). The feeling of helplessness, even if you know what the issue is, and it only takes one command to fix the issue, is awful.

Rollback scenarios

Another valuable tool in your toolbox for reducing the impact of an outage is to make sure its duration is limited. You made the change, you quickly see things are bad, and you simply roll back your change. You can see that for this to work, you need management access though (see above).

In the early days, with Cisco IOS, we’d do it like this:

# reload in 5
# conf t
# < do your change >
# no reload

This meant that unless you executed the last command, the device would reload 5 minutes after your change. Assuming you wouldn't lose access, you would execute no reload, which would cancel the reload, since it wasn’t necessary. Juniper later introduced a more elegant solution with "commit confirmed," which auto-rolls back changes if not confirmed within a set time frame.

[edit]
user@host# commit confirmed 
commit confirmed will be automatically rolled back in 10 minutes unless confirmed
commit complete
#commit confirmed will be rolled back in 10 minutes
[edit]
# < On-call blows Up! >
user@host# rollback 1
user@host# commit

If your environment is more of a software development team, then look at using blue-green deployments. Whatever your choice of making changes or deploying new versions of your software, it’s essential to have the ability to quickly roll back to the latest, known working version. Knowing that you can quickly roll back is great for confidence. Obviously, this depends on our earlier topics: “be the first to know” and management access.

Team culture - Don’t be a hero

Last but not least, we have to talk about team culture. One of the good things about moving from the traditional Ops model to DevOps (you build it, you own it) is that you no longer throw a new version of the software over the wall and ask your ops team to deploy it sight unseen.

Inevitably, at some point, you ship a broken version and your Ops person (this was me) has no clue what changed or how to troubleshoot it. This is extremely stressful and, frankly, unfair.

Nowadays, most development teams own and deploy their own changes and have modern deployment pipelines to deploy them. But even then, the software is complex, and not everyone in the team understands all components. So, when there’s an outage, it should be easy to bring in the rest of the team. Don’t be shy about calling your team members, even in the middle of the night. In a healthy organization, no one should want to wake up the next morning hearing there was a 4-hour outage and someone on your team tried to troubleshoot the bug you introduced. So have a WhatsApp group, or whatever tool you use, just for your team, your own out-of-band channel for emergency help.

Outages are overwhelming. You get a ton of alerts that all need to be acknowledged, and you get questions from support teams, customers, senior management, etc. And then you’re also expected to troubleshoot. No one person can do this. You need a few folks to troubleshoot, manage the alerts, and coordinate the whole process, including communication. So, don’t be a hero. Call in your teammates; together you’ll be able to deal with this. Remember, one team, one dream.

Wrap up

Outages will inevitably occur. Let's acknowledge this reality and take proactive steps to limit their impact and duration.

Be mindful of the knee-jerk management response: “We need more change management process!” Unless you’re a real YOLO shop, this is rarely the answer.

Instead, use the “be the first to know” principle, ensure access to your gear at all times, and have tools ready to swiftly roll back changes to a last-known-good state. Last but not least, embrace the opportunity to learn with your team through retrospectives and emerge even stronger. With a healthy team dynamic, outages can surprisingly become valuable bonding experiences. By being prepared and working together, you'll turn outages into mere blips on the radar.

Recently, I developed a Go program that sends ICMP ping messages to millions of IP addresses. Obviously I wanted this to be done as fast and efficiently as possible. So this prompted me to look into the various methods of interfacing with the network stack and sending packets, fast! It was a fun journey, so in this article, I’ll share some of my learnings and document them for my future self :)  You'll see how we get to 18.8Mpps with just 8 cores. There’s also this Github repo that has the example code, making it easy to follow along.

The use case

Let’s start with a quick background of the problem statement. I want to be able to send as many packets per second from a Linux machine. There are a few use cases, for example, the Ping example I mentioned earlier, but also maybe something more generic like dpdk-pktgen or even something Iperf. I guess you could summarize it as a packet generator.

I’m using the Go programming language to explore the various options. In general, the explored methods could be used in any programming language since these are mostly Go-specific interfaces around what the Linux Kernel provides. However,  you may be limited by the libraries or support that exist in your favorite programming language.

Let’s start our adventure and explore the various ways to generate network packets in Go. I’ll go over the options, and we’ll end with a benchmark, showing us which method is the best for our use case. I’ve included examples of the various methods in a Go package; you can find the code here. We’ll use the same code to run a benchmark and see how the various methods compare.

The net.Dial method

The net.Dial method is the most likely candidate for working with network connections in Go. It’s a high-level abstraction provided by the standard library's net package, designed to establish network connections in an easy-to-use and straightforward manner.  You would use this for bi-directional communication where you can simply read and write to a Net.Conn (socket) without having to worry about the details.

In our case, we’re primarily interested in sending traffic, using the net.Dial method that looks like this:

conn, err := net.Dial("udp", fmt.Sprintf("%s:%d", s.dstIP, s.dstPort))
if err != nil {
	return fmt.Errorf("failed to dial UDP: %w", err)
}
defer conn.Close()

After that, you can simply write bytes to your conn like this

conn.Write(payload)

You can find our code for this in the file af_inet.go

That’s it! Pretty simple, right? As we’ll see, however, when we get to the benchmark, this is the slowest method and not the best for sending packets quickly. Using this method, we can get to about 697,277 pps

Raw Socket

Moving deeper into the network stack, I decided to use raw sockets to send packets in Go. Unlike the more abstract net.Dial method, raw sockets provide a lower-level interface with the network stack, offering granular control over packet headers and content. This method allows us to craft entire packets, including the IP header, manually.

To create a raw socket, we’ll have to make our own syscall, give it the correct parameters, and provide the type of traffic we’re going to send. We’ll then get back a file descriptor. We can then read and write to this file descriptor. This is what it looks like at the high level; see rawsocket.go for the complete code.

fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_RAW)
if err != nil {
	log.Fatalf("Failed to create raw socket: %v", err)
}
defer syscall.Close(fd)

// Set options: here, we enable IP_HDRINCL to manually include the IP header
if err := syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_HDRINCL, 1); err != nil {
	log.Fatalf("Failed to set IP_HDRINCL: %v", err)
}

That’s it, and now we can read and write our raw packet to file descriptor like this

err := syscall.Sendto(fd, packet, 0, dstAddr)

Since I’m using IPPROTO_RAW, we’re bypassing the transport layer of the kernel's network stack, and the kernel expects us to provide a complete IP packet. We do that using the BuildPacket function. It’s slightly more work, but the neat thing about raw sockets is that you can construct whatever packet you want.

We’re telling the kernel just to take our packet, it has to do less work, and thus, this process is faster. All we’re really asking from the network stack is to take this IP packet, add the ethernet headers, and hand it to the network card for sending. It comes as no surprise, then, that this option is indeed faster than the Net.Dial option. Using this method, we can reach about 793,781 pps, about 100k PPS more than the net.Dial method.

The AF_INET Syscall Method

Now that we’re used to using syscalls directly, we have another option. In this example, we create a UDP socket directly like below

fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_DGRAM, syscall.IPPROTO_UDP)

After that we can simply write our payload to it using the Sendto method like before.

err = syscall.Sendto(fd, payload, 0, dstAddr)

It looks similar to the raw socket example, but a few differences exist. The key difference is that in this case we’ve created a socket of type UDP, which means we don’t need to construct the complete packet (IP and UDP header) like before.  When using this method, the kernel manages the construction of the UDP header based on the destination IP and port we specify and handles the encapsulation process into an IP packet.

In this case, the payload is just the UDP payload. In fact, this method is similar to the Net.Dial method before, but with fewer abstractions.

Compared to the raw socket method before, I’m now seeing 861,372 pps—that’s a 70k jump. We’re getting faster each step of the way. I’m guessing we get the benefit of some UDP optimizations in the kernel.

The Pcap Method

It may be surprising to see Pcap here for sending packets. Most folks know pcap from things like tcpdump or Wireshark to capture packets. But it’s also a fairly common way to send packets. In fact, if you look at many of the Go-packet or Python Scappy examples, this is typically the method listed to send custom packets. So, I figured I should include it and see its performance. I was skeptical, but was pleasantly surprised when I saw the pps numbers!

First, let’s take a look at what this looks like in Go; again, for the complete example, see my implementation in pcap.go here

We start by creating a Pcap handle like this:

handle, err := pcap.OpenLive(s.iface, 1500, false, pcap.BlockForever)
if err != nil {
	return fmt.Errorf("could not open device: %w", err)
}
defer handle.Close()

Then we create the packet manually, similar to the Raw socket method earlier, but in this case, we include the Ethernet headers.
After that, we can write the packet to the pcap handle, and we’re done!

err := handle.WritePacketData(packet)

To my surprise, this method resulted in quite a performance win. We surpassed the one million packets per second mark by quite a margin: 1,354,087 pps—almost a 500k pps jump!

Note that, towards the end of this article, we’ll look at  a caveat, but good to know that this method stops working well when sending multiple streams (go routines).

The af_packet method

As we explore the layers of network packet crafting and transmission in Go, we next find the AF_PACKET method. This method is popular for IDS systems on Linux, and for good reasons!

It gives us direct access to the network device layer, allowing for the transmission of packets at the link layer. This means we can craft packets, including the Ethernet header, and send them directly to the network interface, bypassing the higher networking layers. We can create a socket of type AF_PACKET using a syscall. In Go this will look like this:

fd, err := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, int(htons(syscall.ETH_P_IP)))

This line of code creates a raw socket that can send packets at the Ethernet layer. With AF_PACKET, we specify SOCK_RAW to indicate that we are interested in raw network protocol access. By setting the protocol to ETH_P_IP, we tell the kernel that we’ll be dealing with IP packets.

After obtaining a socket descriptor, we must bind it to a network interface. This step ensures that our crafted packets are sent out through the correct network device:

addr := &syscall.SockaddrLinklayer{
	Protocol: htons(syscall.ETH_P_IP),
	Ifindex:  ifi.Index,
}

Crafting packets with AF_PACKET involves manually creating the Ethernet frame. This includes setting both source and destination MAC addresses and the EtherType to indicate what type of payload the frame is carrying (in our case, IP). We’re using the same BuildPacket function as we used for the Pcap method earlier.

The packet is then ready to be sent directly onto the wire:

syscall.Sendto(fd, packet, 0, addr)

The performance of the AF_PACKET method turns out to be almost identical to that achieved with the pcap method earlier. A quick Google, shows that libpcap, the library underlying tools like tcpdump and the Go pcap bindings, uses AF_PACKET for packet capture and injection on Linux platforms. So that explains the performance similarities.

Using the AF_XDP Socket

We have one more option to try. AF_XDP is a relatively recent development and promises impressive numbers! It is designed to dramatically increase the speed at which applications can send and receive packets directly from and to the network interface card (NIC) by utilizing a fast path through the traditional Linux network stack. Also see my earlier blog on XDP here.

AF_XDP leverages the XDP (eXpress Data Path) framework. This capability not only provides minimal latency by avoiding kernel overhead but also maximizes throughput by enabling packet processing at the earliest possible point in the software stack.

The Go standard library doesn’t natively support AF_XDP sockets,  and I was only able to find one library to help with this. So it’s all relatively new still.

I’m using this library github.com/asavie/xdp and this is how you can initiate an AF_XDP socket.

xsk, err := xdp.NewSocket(link.Attrs().Index, s.queueID, nil)

Note that we need to provide a NIC queue; this is a clear indicator that we’re working at a lower level than before. The complete code is a bit more complicated than the other options, partially because we need to work with a user-space memory buffer (UMEM) for packet data. This method reduces the kernel's involvement in packet processing, cutting down the time packets spend traversing system layers. By crafting and injecting packets directly at the driver level. So instead of pasting the code, please look at my code here.

The results look great; using this method, I’m now able to generate 2,647,936 pps. That’s double the performance we saw with AF_PACKET! Whoohoo!

Wrap-up and some takeaways

First off, this was fun to do and learn! We looked at the various options to generate packets from the traditional net.Dial method, to raw sockets, pcap, AF_PACKET and finally AF_XDP. The graph below shows the numbers per method (all using one CPU and one NIC queue). AF_XDP is the big winner!

If interested, you can run the benchmarks yourself on a Linux system like below:

./go-pktgen --dstip 192.168.64.2 --method benchmark \
 --duration 5 --payloadsize 64 --iface veth0

+-------------+-----------+------+
|   Method    | Packets/s | Mb/s |
+-------------+-----------+------+
| af_xdp      |   2647936 | 1355 |
| af_packet   |   1368070 |  700 |
| af_pcap     |   1354087 |  693 |
| udp_syscall |    861372 |  441 |
| raw_socket  |    793781 |  406 |
| net_conn    |    697277 |  357 |
+-------------+-----------+------+

The important number to look at is packets per second as that is the limitation on software network stacks. The Mb/s number is simply the packet size x the PPS number you can generate. It’s interesting to see the easy 2x jump from the traditional net.Dial approach to using AF_PACKET. And then another 2x jump when using AF_XDP. Certainly good to know if you’re interested in sending packets fast!

The benchmark tool above uses one CPU and, thus, one NIC queue by default. The user can, however, elect to use more CPUs, which will start multiple go routines to do the same tests in parallel. The screenshot below shows the tool running with 8 streams (and 8 CPUs) using AF_XDP,  generating 186Gb/s with 1200 byte packets (18.8Mpps)! That’s really quite impressive for a Linux box (and not using DPDK). Faster than what you can do with Iperf3 for example.

Some caveats and things I’d like to look at in the future

Running multiple streams (go routines) using the PCAP method doesn’t work well. The performance degrades significantly. The comparable AF_PACKET method, on the other hand, works well with multiple streams and go routines.

The AF_XDP library I’m using doesn’t seem to work well on most hardware NICs. I opened a GitHub issue for this and hope it will be resolved. It would be great to see this be more reliable as it kind of limits more real-world AF_XDP Go applications. I did most of my testing using veth interfaces; i’d love to see how it works on a physical NIC and a driver with XDP support.

It turns out that for AF_PACKET, there's a zero-copy mode facilitated by the use of memory-mapped (mmap) ring buffers. This feature allows user-space applications to directly access packet data in kernel space without the need for copying data between the kernel and user space, effectively reducing CPU usage and increasing packet processing speed. This means that, in theory, the performance of AF_PACKET and AF_XDP could be very similar.  However, it appears the Go implementations of AF_PACKET do not support zero-copy mode or only for RX and not TX. So I wasn’t able to use that. I found this patch but unfortunately couldn’t get it to work within an hour or so, so I moved on. If this works, this will likely be the preferred approach as you don’t have to rely on AF_XDP support.

Finally, I’d love to include DPDK support in this pktgen library. It’s the last one missing. But that’s a whole beast on its own, and I need to rely on good Go DPDK libraries. Perhaps in the future!

That’s it; you made it to the end! Thanks for reading!

Cheers
-Andree

Three years ago, I wrote a blog titled “AWS and their Billions in IPv4 addresses “in which I estimated AWS owned about $2.5 billion worth of IPv4 addresses. AWS has continued to grow incredibly, and so has its IPv4 usage. In fact, it’s grown so much that it will soon start to charge customers for IPv4 addresses! Enough reason to check in again, three years later, to see what AWS’ IPv4 estate looks like today.

A quick 2020 recap

Let’s first quickly summarize what we learned when looking at AWS’s IPv4 usage in 2020. First, in 2020, we observed that the total number of IPv4 addresses we could attribute to AWS was just over 100 Million (100,750,168). That’s the equivalent of just over six /8 blocks.

Second, for fun, we tried to put a number on it; back then, I used $25 per IP, bringing the estimated value of their IPv4 estate to Just over $2.5 billion.

Third, AWS publishes their actively used IPv4 addresses in a JSON file. The JSON file contained references to roughly 53 Million IPv4 addresses. That meant they still had ~47 Million IPv4 addresses, or 47%, available for future allocations. That’s pretty healthy!

The 2023 numbers

Okay, let’s look at the current data. Now, three years later, what does the IPv4 estate for AWS look like? I used the same scripts and methods as three years ago and found the following.

First, we observe that AWS currently owns 127,972,688 IPv4 addresses. Ie. almost 128 million IPv4 addresses. That’s an increase of 27 million IPv4 addresses. In other words, AWS added the equivalent of 1.6 /8’s or 415 /16’s in three years!

Second, what’s it worth? This is always tricky and just for fun. Let’s first assume the same $25 per IPv4 address we used in 2020.

127,972,688 ipv4 addresses x $25 per IP = $3,199,317,200.

So, with the increase of IPv4 addresses, the value went up to ~$3.2 Billion. That’s a $700 million increase since 2020.

However, if we consider the increase in IPv4 prices over the last few years, this number will be higher. Below is the total value of 127M IPv4 addresses at different market prices.

Total number of IPv4 addresses: 127,972,688
value at $20 per IP: $2,559,453,760
value at $25 per IP: $3,199,317,200
value at $30 per IP: $3,839,180,640
value at $35 per IP: $4,479,044,080
value at $40 per IP: $5,118,907,520
value at $50 per IP: $6,398,634,400

Based on this data from ipv4.global, the average price for an IPv4 address is currently ~$35 dollars. With that estimate, we can determine the value of AWS’s IPv4 estate today to about 4.5 Billion dollars. An increase of 2 Billion compared to three years ago!

Thirdly, let’s compare the difference between the IPv4 data we found and what’s published in the JSON file AWS makes available. In the JSON today, we count about 73 million IPv4 addresses (72,817,397); three years ago, that was 53 Million. So, an increase of 20 million in IPv4 addresses allocated to AWS services.

Finally, when we compare the ratio between what Amazon owns and what is allocated to AWS according to the JSON data, we observe that about 57% (72817397 / 127972688) of the IPv4 addresses have been (publicly) allocated to AWS service. They may still have 43% available for future use. That’s almost the same as three years ago when it was 47%.
(Note: this is an outsider’s perspective; we should likely assume not everything is used for AWS).

Where did the growth come from

A quick comparison between the results from three years ago and now shows the following significant new additions to AWS’ IPv4 estate,

• Two new /11 allocations: 13.32.0.0/11 and 13.192.0.0/11. This whole 13/8 block was formerly owned by Xerox.
  (Note: it appears AWS owned 13.32.0.0/12 already in 2020).
• Two new /12 allocations: 13.224.0.0/12 (see above as well). It appears they continued purchasing from that 13/8 block.
• I’m also seeing more consolidation in the 16.0.0.0/8 block. AWS used to have quite a few /16 allocations from that block, which are now consolidated into three /12 allocations: 16.16.0.0/12 16.48.0.0/12, and 16.112.0.0/12
• Finally, the 63.176.0.0/12 allocation is new.

AWS is starting to charge for IPv4 addresses

In August of this year, AWS announced they will start charging their customers for IPv4 addresses as of 2024.

Effective February 1, 2024 there will be a charge of $0.005 per IP per hour for all public IPv4 addresses, whether attached to a service or not (there is already a charge for public IPv4 addresses you allocate in your account but don’t attach to an EC2 instance).

That’s a total of $43.80 per year per IPv4 address; that’s a pretty hefty number! The reason for this is outlined in the same AWS blog:

As you may know, IPv4 addresses are an increasingly scarce resource and the cost to acquire a single public IPv4 address has risen more than 300% over the past 5 years. This change reflects our own costs and is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses

The 300% cost increase to acquire an IPv4 address is interesting and is somewhat reflected in our valuation calculation above (though we used a more conservative number).

So, how much money will AWS make from this new IPv4 charge? The significant variable here is how many IP addresses are used at any given time by AWS customers. Let’s explore a few scenarios, starting with a very conservative estimate, say 10% of what is published in their IPv4 JSON is in use for a year. That’s 7.3 Million IPv4 addresses x $43.80, almost $320 Million a year. At 25% usage, that’s nearly $800 Million a year. And at 31% usage, that’s a billion dollars!

Notice that I’m using a fairly conservative number here, so it’s not unlikely for AWS to make between $500 Million to a Billion dollars a year with this new charge!

The data

You can find the data I used for this analysis on the link below. There, you’ll also find all the IPv4 prefixes and a brief summary. https://gist.github.com/atoonk/d8bded9d1137b26b3c615ab614222afd
Similar data from 2020 can be found here.
PS. Let me know if someone knows the LACNIC or AFRINIC AWS resources, as those are not included in this data set.

Wrap up

In this article, we saw how, over the last three years, AWS grew its IPv4 estate with an additional 27 million IP addresses to now owning 128 Million IPv4 addresses. At a value of $35 per IPv4 address, the total value of AWS’ IPv4 estate is ~4.5 Billion dollars. An increase of $2 billion compared to what we looked at three years ago!

Regarding IPv4 capacity planning, it seems like the unallocated IPv4 address pool (defined as not being in the AWS JSON) has remained stable, and quite a bit of IPv4 addresses are available for future use.

All this buying of IPv4 addresses is expensive, and in response to the increase in IPv4 prices, AWS will soon start to charge its customers for IPv4 usage. Based on my estimates, It’s not unlikely that AWS will generate between $500 million and $1 billion in additional revenue with this new charge. Long live IPv4!

Cheers
Andree

With the help of ChatGTP, AI has officially changed everything we do. It’s only been a few months since chatGPT was released, and like many, I’ve been exploring how to best use it. I used it as a coding buddy, to brainstorm, to help with writing, etc.

The potential of this new technology blows me away. But as a technology enthusiast, I’d love to understand better how it works, or at least better understand some of the common terms and underlying technology. I keep hearing about Large Language Models (LLMs), vector databases, training, models, etc. I’d love to learn more about it, and what better way to just dive in, get my hands dirty and experiment with it?

So in today’s blog, I’m sharing some learnings on one of these building blocks, called embeddings. Why embeddings? Well, originally, I planned to learn more about Vector databases, but I quickly learned that in order to understand these better, I should start with vectors and embeddings.

What is an embedding

This is the definition from the OpenAI website

An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness, and large distances suggest low relatedness.

Hhm, ok, but what does that really mean? Imagine you have a word, say hamburger. In order to use this in an LLM (large language model) like GPT, the LLM needs to know what it means. To do that, we can turn the word hamburger into an embedding. An embedding is essentially a (set of) numerical representations of a word, indicating its meaning. We call this the Vector.

With embeddings, we can now represent the words as vectors:

• dog: [0.2, -0.1, 0.5, …]
• cat: [0.1, -0.3, 0.4, …]
• fish: [-0.3, 0.6, -0.1, …]

Notice that it’s a representation of the meaning (semantics) of the word. For example, the word embeddings for “dog” and “puppy” would be close together in the vector space because they share a similar meaning and often appear in similar contexts. In contrast, the embeddings for “dog” and “car” would be farther apart because their meanings and contexts are quite different.

It is this “Word embeddings” technology that enables semantic search, which goes beyond simple keyword matching to understand the meaning and context behind a query. “Semantic” refers to the similarity in meaning between words or phrases.

For example, traditional string matching would fail to connect the “searching for something to eat” query with the sentence “the mouse is looking for food.” However, with semantic search powered by word embeddings, a search engine recognizes that both phrases share a similar meaning, and it would successfully find the sentence.

Ok, great. Now that we somewhat understand how this works, how does it really work?

Turning words or sentences into embeddings

A word or sentence can be turned into an embedding (a vector representation) using the OpenAI API. To get an embedding, send your text string to the embeddings API endpoint along with a choice of embedding model ID (e.g., text-embedding-ada-002). The response will contain an embedding you can extract, save, and use.

In my case, I’m using the Python API. Using this API, you can simply use the code below to turn the word hamburger into an embedding.

from openai.embeddings_utils import get_embedding
hamburger_embedding = get_embedding("hamburger", engine='text-embedding-ada-002')

# will look like something like [-0.01317964494228363, -0.001876765862107277, …

If you have a text document, you would turn all the words or sentences from that document into embeddings. Once you’ve done that, you essentially have a semantic representation of the document as a series of vectors. These vectors capture the meaning and context of the individual words or sentences.

Finding Similarities

Once you have embeddings for words or sentences, you can use them to find semantic similarities. A common approach to measuring the similarity between two embeddings is by calculating how close the vectors are to each other.

Calculating the distance between vectors is done by calculating the cosine similarity; if you’re really interested, you can read about that here. https://en.wikipedia.org/wiki/Cosine_similarity

Luckily the Python module that OpenAI ships has an implementation of this cosine_similarity, and you can simply use it like this:

import openai
from openai.embeddings_utils import get_embedding, cosine_similarity
openai.api_key = "<YOUR OPENAI API KEY HERE>"
embedding1 = get_embedding("the kids are in the house",engine='text-embedding-ada-002')
embedding2 = get_embedding("the children are home",engine="text-embedding-ada-002")
cosine_similarity(embedding1, embedding2)

Which prints: 0.9387390865828703, meaning they’re very close.

A real-life example

Below is a slightly longer example. It reads the document called words.csv, which looks like this:

text
"red"
"potatoes"
"soda"
"cheese"
"water"
"blue"
"crispy"
"hamburger"
"coffee"
"green"
"milk"
"la croix"
"yellow"
"chocolate"
"french fries"
"latte"
"cake"
"brown"
"cheeseburger"
"espresso"
"cheesecake"
"black"
"mocha"
"fizzy"
"carbon"
"banana"
"sunshine"
"orange carrot"
"sun"
"hay"
"cookies"
"fish"

The script below then calculates the embeddings for all these words and adds it all to a Panda data frame. Next, it will take a search term (hotdog) and calculates what words are closest to the word hotdog.

import openai
import pandas as pd
import numpy as np
from openai.embeddings_utils import get_embedding, cosine_similarity

openai.api_key = "<YOUR OPENAI API KEY HERE>"

# read the data
df = pd.read_csv('words.csv')

# Lamda to add embedding column
df['embedding'] = df['text'].apply(lambda x: get_embedding(x, engine='text-embedding-ada-002'))
# Safe it to a csv file, for caching later. So we dont need to call the API all the time
# You'd store this in a vector database
df.to_csv('word_embeddings.csv')
df = pd.read_csv('word_embeddings.csv')

# Convert the string representation of the embedding to a numpy array
# neeeded since we wrote it to a csv file
df['embedding'] = df['embedding'].apply(eval).apply(np.array)

# Hotdog is not in the CSV. Let calculate the embedding for it
search_term = "hotdog"
search_term_vector = get_embedding(search_term, engine="text-embedding-ada-002")

# now we can calculate the similarity between the search term and all the words in the CSV 
df["similarities"] = df['embedding'].apply(lambda x: cosine_similarity(x, search_term_vector))
# print the top 5 most similar words
print(df.sort_values("similarities", ascending=False).head(5))

The code above prints this:

Unnamed: 0          text                                          embedding  similarities
7            7     hamburger  [-0.01317964494228363, -0.001876765862107277, ...      0.913613
18          18  cheeseburger  [-0.01824556663632393, 0.00504859397187829, 0....      0.886365
14          14  french fries  [0.0014257068978622556, -0.016548126935958862,...      0.853839
3            3        cheese  [-0.0032112577464431524, -0.0088559715077281, ...      0.838909
13          13     chocolate  [0.0015315973432734609, -0.012976923026144505,...      0.830742

Pretty neat, right?! It calculated that a hotdog is most similar to a hamburger, cheeseburger, and fries!

Let’s do one more thing! In the example below, we add the embeddings for milk and coffee together, just like a simple math addition.

We then again calculate what this new embedding is most similar to (hint, what do you call a drink that adds coffee to milk?).

# Let's make a copy of the data frame we created earlier, so we can compare the embeddings of two words
food_df = df.copy()
milk_vector = food_df['embedding'][10]
coffee_vector = food_df['embedding'][8]

# lets add the two vectors together
milk_coffee_vector = milk_vector + coffee_vector

# now calculate the similarity between the combined vector and all the words in the CSV
food_df["similarities"] = food_df['embedding'].apply(lambda x: cosine_similarity(x, milk_coffee_vector))

print(food_df.sort_values("similarities", ascending=False).head(5))

The result is this

Unnamed: 0 text embedding similarities
8 8 coffee [-0.0007212135824374855, -0.01943901740014553,… 0.959562
10 10 milk [0.0009238893981091678, -0.019352708011865616,… 0.959562
15 15 latte [-0.015634406358003616, -0.003936054650694132,… 0.905960
19 19 espresso [-0.02250547707080841, -0.012807613238692284, … 0.898178
22 22 mocha [-0.012473775073885918, -0.026152553036808968,… 0.889710

Ha! Yes, it’s obviously similar to coffee and milk, as that’s what we started with, but next up, we see a latte! That’s pretty cool, right? Coffee + Milk = Late 😀

Vector database

Now that we’ve seen how embeddings work and how they can be used to find semantic similarities, let’s talk about vector databases. In our example, we saw that calculating the embeddings was done using an API call to the OpenAI API. This can be slow and will cost you credits. That’s why, in the example code, we saved the calculated embeddings to a CSV file for caching purposes.

While this approach works for small-scale experiments, it may not be practical for large amounts of data or production environments where performance and scalability are important. This is where vector databases come in.

There are a few popular ones; a well-known one is Pinecone, but even Postgres can be used as a vector database. These vector databases are specifically designed for storing, managing, and efficiently searching through large amounts of embeddings. They are optimized for high-dimensional vector data and can handle operations such as nearest neighbor search, which is crucial for finding the most similar items to a given query.

Wrap up

In this exploration of the technology behind LLMs and AI, I delved into some of the foundational building blocks that power these advanced systems; specifically, we looked at embeddings and vectors. My initial curiosity about vector databases and their potential applications for my own data led me to first understand the underlying principles and the importance of vectors. It’s pretty cool to see how easy it was to get going, thanks to the existing API’s and libraries.

Perhaps in another weekend adventure, I’ll look further into the next logical topic: vector databases. I’d also love to explore Langchain, a fascinating framework for developing applications powered by language models.

That’s it for now; thanks for reading!

Cheers
Andree

In one of my recent blog posts, we looked at the ~two billion dollars worth of IPv4 assets AWS has collected over the last few years. One of the lessons from that story is that the IPv4 market is still very much alive and that folks are still happy to sell IPv4 resources and cloud providers are willing to pay big money to grow their business.

It’s always interesting to trace where all the sold IPv4 networks came from and who owned them previously. I’ve seen all kinds of entities sell their IPv4 resources. Examples include local police departments, libraries, car manufactures, and even large telecom hardware vendors. So I’m not sure if there’s a real pattern, other than those folks that received a large allocation decades ago and no longer need it, so it’s better to sell it.

One of the recent large transfers that caught my attention was the sale of (the majority of) the 43.0.0.0/8 IPv4 space. Since it’s such a large range and the story behind it is a bit unique, let’s take a closer look.

The Asia Pacific Internet Development Trust (APIDT)

In March of last year (2020), APNIC and the Japanese WIDE project announced the establishment of the Asia Pacific Internet Development Trust (APIDT). The announcement on the APNIC blog reads:

APIDT was created in line with the wishes of the Founder of the WIDE Project, Professor Jun Murai of Keio University, following his decision on the future use of IPv4 address space in the 43/8 block, announced today.

The article then explains the purpose of the Trust.

The Trust will use the proceeds of the sale of the address space to create a fund to benefit Internet development in the Asia Pacific region. This will include funding technical skills development and capacity building, improvements to critical Internet infrastructure, supporting research and development, and improving the community’s capability to build an open, global, stable and secure Internet.

So APIDT, a joint initiative of the WIDE Project and APNIC, is a newly created Trust that will be funded by selling the unallocated (87.5%) portion of 43.0.0.0/8. The proceeds of the sale will be used in support of Internet development in the APNIC region.

Selling the 43/8 addresses space

APIDT has recently completed the sale of the IPv4 address holdings in three separate stages via a sale by tender process. The IPv4 blocks sold in each stage were as follows:

Stage 1 offered the 43.0.0.0/9 block, subdivided into eight /12 blocks.
Stage 2 offered the 43.128.0.0/10 block, subdivided into sixteen /14 blocks.
Stage 3 offered the 43.192.0.0/11 block, subdivided into thirty-two /16 blocks.

The whole process took a while, the results of stage 1 and stage 2 were publicly visible in the RIR database last year (2020). For whatever reason, stage 3 took quite a bit longer and is still not fully public.

So who bought all the IPv4 addresses?

The lucky winners and, likely, big spenders are perhaps not surprisingly some of the major cloud providers.

Stage 1 - 43.0.0.0/9 block, subdivided into eight /12 blocks

The big winner of the first stage of the tender was Alibaba. They now own all of the eight /12 blocks. These have now been allocated as two /10 to Alibaba, 43.0.0.0/10 and 43.64.0.0/10.

Stage 2 — 43.128.0.0/10 block, subdivided into sixteen /14 blocks

Again a winner takes all result here. The winner of stage two is the other major cloud player in the Asia Pacific region: Tencent Cloud Computing.

The sixteen /14 blocks are now allocated to Tencent as one large /10 allocation: 43.128.0.0/10

Stage 3 — 43.192.0.0/11 block, subdivided into thirty-two /16 blocks

This one took a while but as of June 21st, 2021 we can see that just over 50% of this block, 43.192.0.0/12,  has been allocated.

The first half, 43.192.0.0/12 goes to.. surprise surprise, AWS

The final two /16’s in this block are going to
43.222.0.0/16 NEC Corporation
43.223.0.0/16 LINE Corporation

Update: May 23, 2022

All of 43.192.0.0/11, with the exception of 43.222.0.0/16 (NEC Corporation) and 43.223.0.0/16 (LINE Corporation) has been allocated to AWS.

Interestingly there already was some movement around the first few /16 blocks. These used to be allocated to AWS. However, double-checking this today again, I noticed that some of these are now registered to companies that go by the name of "Sinnet Technology" and "Ningxia West Cloud Data". After some more digging, it turns out that these are the companies that operate AWS in China

From the AWS website: https://www.amazonaws.cn/en/about-aws/china/faqs/
Beijing Sinnet Technology Co., Ltd.("Sinnet") operates and provides services from the Amazon Web Services China (Beijing) Region, and Ningxia Western Cloud Data Technology Co., Ltd. (“NWCD”) operates and provides services from the Amazon Web Services China (Ningxia) Region, each in full compliance with Chinese regulations.

For a full list of what IPv4 block was sold to whom, also see this Google Sheet:
https://docs.google.com/spreadsheets/d/13Kzg8pIJKiIV33WsRwEjvVD9myhDiAFZvOlZ1eGyW4s/edit?usp=sharing

How much money was made?

That question is hard to answer as the dollar figures for the winning bids have not been made public (as far as I know). However, we can make an estimate based on the known market prices.

There’s limited public information on what price IPv4 addresses are sold for. Typically the per IP address depends on the size of the IP ranges that are being sold. One public benchmark is the transaction between AMPR.org and AWS in 2019. In that transaction, AWS paid $108 Million for 44.192.0.0/10. That’s $25.74 per IP address.

A more recent public source is the ipv4marketgroup website. According to them, the per IP price for IPv4 blocks of size of /17 or larger (/16, /15, etc) currently range between $38 to $40 USD per IP. The graph below shows how prices have steadily increased over the years.

APIDT sold off the equivalent of 14,680,064 IPv4 addresses. So if we estimate the price at say $38 per IP, the total value of the sold IP addresses would be around $558  Million. While on the higher end, at $40 per IP the value would be $587 Million.
At a more conservative price of $34 per IP, the combined value would still be $499 Million (USD).

Update: May 23, 2022

As of February 2022, APIDT has filed its first financial report to the Australian charities regulator, here (see page 20)

This annual financial report shows that the total amount raised with the sale of  the IP addresses was: US$440,716,493. This means that the per IP address price was almost exactly $30 (USD)

Wrapping up

The sale of the 43/8 block shows the IPv4 market is still alive and thriving. We observe the market continues to find new resources to sell. In the example of APIDT, we now know that the funds made available for the Trust as a result of the sale was 440 million (US) dollars. I’d say that’s a very healthy fund for the purpose of supporting Internet development in the Asia Pacific region.

This blog is also a call to action for all network engineers to clean up those lingering network loops as they aren’t just bad hygiene but a significant operational DDoS risk.

Network Loops

All network engineers are familiar with network loops. A network loop causes an individual packet to bounce around in a network while consuming valuable network resources (bandwidth and PPS). There are various reasons IP networks can have loops, typically caused by configuration mistakes. The only real “protection” against network loops is the Time To Live (TTL) check. However, the Time To Live check is less of protection against loops, and more protection against them looping forever.

The Time to Live (TTL) refers to the amount of time or “hops” that a packet is set to exist inside a network before being discarded by a router. The TTL is an 8-bit field in the IP header, and so it has a maximum value of 255. The typical TTL value on most operating systems is 64, which should work fine in most cases.

Most network engineers know loops are bad hygiene. I think it’s much less understood (or thought about) what the operational risk of a loop is. In my experience, most loops are for IP addresses that aren’t actually in use (otherwise, it would be an outage), so typically, solving this ends up on the backburner.

A Simple Loop example

Let’s look at a simple example. The diagram below shows two routers; imagine those being your core or edge routers in your datacenter.

For whatever reason, they both believe 192.0.2.0/24 is reachable via the other router. As a result, packets for that destination will bounce between the two routers. Depending on the TTL value, the bandwidth used for that one packet will be:

packet size in bytes x 8 x TTL.

Meaning for a 512 byte packet and a TTL of 60, the amount bandwidth used is 245Kbs. In other words, the 512 byte packet turned into 307,20 bytes (60x) before it was discarded. You could think of this number 60 as the amplification factor.

But.. but.. Network loops are rare

That depends on your definition of rare. The good folks at Qrator monitor for loops on the Internet. According to the Qrator measurements, there are roughly twenty million unique loops (measured as unique router pairs).

Combining loops and common Amplification attacks.

Now that we’ve covered the risk of loops and their potential > 100x (the TLL) amplification factor, this may remind you of the traditional DDoS attacks.

The record high DDOS attacks you read about in the news are now hitting hundreds of Gigabits per second and peaking into the Terabits. All these large volume metric attacks are mostly the same type of attack and are commonly known as amplification or reflection attacks.

They rely on an attacker sending a small packet with a spoofed source IP, which is then reflected and amplified to the attack target. The reflectors/amplifiers are typically some kind of open UDP service that takes a small request and yields a large answer. Typical examples are DNS, NTP, SSDP, and LDAP. The large attacks you read about in the news typically combine a few of these services.

By now, it may be clear what the danger of combining these two types of scenarios can be. Let’s look at an example. A typical DNS amplification query could be something like this, and RRSIG query for irs.gov

dig -t RRSIG irs.gov @8.8.8.8

This query is 64 bytes on the wire. The resulting answer is large and needs to be sent in two packets; the first packet is 1500 bytes, the second packet 944 bytes. So in total, we have an amplification factor of (1500+944)/64 = 38.

IP (tos 0x0, ttl 64, id 32810, offset 0, flags [none], 
proto UDP (17), length 64)
    192.168.0.30.57327 > 8.8.8.8.53: 42548+ [1au] RRSIG? irs.gov. (36)

IP (tos 0x0, ttl 123, id 15817, offset 0, flags [+], 
proto UDP (17), length 1500)
    8.8.8.8.53 > 192.168.0.30.57327: 42548 8/0/1 irs.gov. RRSIG, irs.gov. 
    RRSIG, irs.gov. RRSIG, irs.gov. RRSIG, irs.gov. RRSIG[|domain]
    
IP (tos 0x0, ttl 123, id 15817, offset 1448, flags [none], 
proto UDP (17), length 944)
    8.8.8.8 > 192.168.0.30: ip-proto-17

Note: There are many different types of amplification attacks. This is just a modest and straightforward DNS example. Also, note that common open reflectors, such as Public DNS resolvers typically have smart mechanisms to limit suspicious traffic to reduce the negative impact these services could have.

The tcpdump output above shows that when the answer arrives back from Google’s DNS service, the TTL value is 123; this is higher than most other public DNS resolvers (most appear to default to 64).

If we combine this attack with the ‘loop’ factor we looked at previously (determined by the TTL value), we have the total amplification factor.

Adding up the numbers

Ok, so let’s continue to work on the DNS amplification example. The amplification number of 38 and a TTL of 123, would result in a total amplification number of :

38 * (123 / 2) = 2,337

Note that I’m dividing the TTL number by two so that we get a per receive (RX) and transmit(tx) number.

For now, let’s use 2,337 as a reasonable total amplification number. What kind of traffic would an attacker need to generate 10G or 100G of traffic? One would need just about 5Mbs/s to saturate a 10g link and say ~50Mbs to saturate a 100Gbs link! These numbers are low enough to generate from a simple home connection. Now imagine what an attacker with bad intentions and access to a larger botnet could do…

Let’s double-check this with a Demo

To make sure this is indeed all possible and the math adds up, I decided to build a lab to reproduce the scenario we looked at above.

The lab contains four devices:

1. An attacker: initiating a DNS-based reflection attack. The (spoofed) source IP is set to 192.0.2.53
2. A DNS resolver: receiving the DNS queries (with a spoofed source IP) and replying with an answer that is 38 times larger than the original question. The IP TTL value in the DNS answer is 123.
3. A router pair (rtr1 — rtr2): Both routers have a route for 192.0.2.0/24 pointing to each other. As a result, the DNS answer with a destination IP of 192.0.2.53 will bounce back and forth between the two routers until the TTL expires.

In the screenshot above, we see the attacker on the top left sending queries at a rate of 5.9Mbs. On the bottom left, we see the DNS resolver receiving traffic at 5.9Mbs from the client and answering the queries at a rate of ~173mbs. The IP packets with the DNS responses have a TTL of 123.

We see the router pair on the right-hand side: rtr1 on the top right and rtr2 on the bottom right. As you can see, both devices are sending and receiving at 10Gb/s. So, in this case, we observe how the client (attacker) turned 6mb/s into 10Gb/s.

Wrapping up

In this blog, we looked at the danger of network loops. I hope it’s clear that loops aren’t just a hygiene or cosmetic issue but instead expose a significant vulnerability that should be cleaned up ASAP.

We saw that loops are by no means rare and that there are millions of router pairs with network loops. In fact, according to Qrator data, over 30% of all Autonomous Systems (ASns), including many of the big cloud providers, have networks with loops in them.

We observed that an attacker can easily saturate a 10G link with 85Mbs (at a TTL of 240) without any UDP amplification. Or if combined with a typical UDP amplification attack, 6Mbs of seed traffic will result in 10G on a looped path, or 60Mb/s could potentially fill up a 100Gbs path!

Not all loops are the same

Most loops happen between two adjacent routers; quite a few of those appear to occur between an ISP’s router and the customer router. I have also seen loops happen involving up to eight hops (routers) spanning various metro areas while looping between Europe and the US. These transatlantic loops are expensive and hard to scale up quickly. As a result, loops on links like these will have a more significant impact.

Call to Action

I hope this article convinced you to check your network for loops and make sure you won’t be affected by attacks like these. Consider signing up for the free Qrator service, and you’ll get alerted when new loops (or other issues) are detected in your network.

Last month we introduced our first zero trust features by introducing the concept of Identity Aware Sockets. It’s been great to see folks giving this a spin and start using it as a remote access alternative for the traditional VPN.

Most services out there today are HTTP based, typically served over HTTPS. However, there are a few other commonly used services that are not HTTP based and, as a result, up until today, didn’t benefit from our identity-aware sockets. In this article, we’ll introduce Zero trust support for non-HTTP based service, with the introduction Identity aware TCP sockets. Specifically, we’ll look at providing zero trust services for SSH as an example.

Determining the user’s identity, authentication, and authorization

Turning your mysocket services into an identity-aware socket is as simple as adding the — cloud_authentication flag to mysocketctl when creating the service. While doing so, you have the ability to add a list of email domains and/or a list of email addresses. Now each time a user tries to access your service, a browser will pop up asking the user to authenticate. Once authentication is finished, we know the user’s identity, and if that identity matches the list of authorized users, only then will the user be let through.

If you think about what is happening here, you’ll realize that what we have here is a per session, authenticating firewall. Only after the user is authenticated and authorized do we allow the network traffic through. Notice that this is much more advanced than your traditional firewall; now, every network flow has an identity. That’s powerful!

This flow of redirecting users to authenticate and then back to the service was do-able because it’s done in the browser and largely built of HTTP session management. Now we’d like to extend this with non-HTTP services, so we’ll need to find an alternative for the HTTP session part.
The solution for this comes with the help of Mutual TLS (MTLS). MTLS forces the client to authenticate itself when talking to the server. This is achieved by presenting a signed client certificate to the server.

Identity aware TCP sockets

With the introduction of identity-aware TCP sockets, the mysocket edge proxies act as an authenticating firewall. Since we are relying on client TLS certificates, all traffic is securely tunneled over a TLS connection.

As you can see in the flow below, there are a few actions to take before the user can get through. To make this a seamless experience for the users of your service, we’ve extended the mysocketctl command-line tool with the required functionality that kicks of the authentication flow. It starts the authentication process; after that, it requests a client certificate (your ticket in), and then it sets up the TLS tunnel for you. After that, users can send traffic over this authenticated and encrypted tunnel. In its simplest form, it will look something like this:

echo "hello" | mysocketctl client tls \ 
    --host muddy-pond-7106.edge.mysocket.io

In the example above, we’re sending the string hello, to the service served by muddy-pond-7106.edge.mysocket.io.

Before the string “hello” arrives at the service protected by mysocket, the mysocketctl client will take care of the authentication flow, requests the TLS client certificate, and sends whatever comes in over stdin to the mysocket edge services.

SSH zero trust

Now that we understand the high-level flow let’s look at a more practical example. In this example, we have a server for which I’d like to make the SSH service available to only a subset of users. The ssh is on a private network such as your corporate network, your home network, or even in a private VPC or just firewalled off from the Internet.

First, we’ll provision the service using mysocketctl on the server-side and set up the tunnel.

mysocketctl connect \  
	--name 'remote access for my ssh server' \ 
    --cloudauth \  
    --allowed_email_addresses'contractor@gmail.com,john@example.com' \ 
    --allowed_email_domains 'mycorp.com' \ 
    --port 22 --host localhost \ 
    --type tls

In this example, I'm creating a mysocket service of type TLS, and we enable cloud authentication. This will force the user to present a valid client TLS certificate. The certificate can only ever be handed out to users that authenticate with a mycorp.com email address or using the specific email addresses contractor@gmail.com or john@example.com.
The same command will also set up a secure tunnel to the closest mysocket tunnel servers and expose the ssh service running on port 22.

The result is that this SSH service is now accessible to allowed users only, as crimson-thunder-8434.edge.mysocket.io:38676
Only inbound traffic with a valid client TLS ticket will be let through. Valid TLS client certificates will only ever be issued to users with a mycorp.com domain or the two contractor email addresses we specified.

Setting up an SSH session

Ok, time to test this and connect to this ssh service. Remember that we need a valid TLS client certificate. These are issued only with a valid token, and the token is only handed out to authorized users. To make all of this easier, we’ve extended the mysocketctl tool to take care of this workflow. The example below shows how we use the ssh ProxyCommand to make that easier for us, like this.

ssh ubuntu@crimson-thunder-8434.edge.mysocket.io \ 
	-o 'ProxyCommand=mysocketctl client tls --host %h'

This will tell ssh to send all ssh traffic through this mysocketctl client command. This will start the authentication process, fetch the TLS client certificate for us, set up the TLS tunnel to the mysocket edge server, and transport the ssh traffic through this authenticated tunnel. The user can now log in to the ssh server using whatever method you’re used to.

With this, we’ve made our private ssh server accessible from the Internet, while the authenticating mysocket firewall is only allowing in session from client identities we approved beforehand. No VPN needed. Pretty cool, right?

Mysocket SSH Certificate authorities.

SSH is quite similar to TLS in terms of workflow. It too supports authenticating users using signed certificates.

So we decided to expand on this functionality. In addition to an API endpoint that is responsible for signing TLS certificates, we also created one for signing SSH keys.

If we build on the example above, the user can now, in addition to requesting a TLS client certificate, also request a signed SSH certificate. Our SSH certificate signing service will only sign the signing request if the user is authenticated and authorized, using the same logic as before.

Setting up the server

In order to use this, we’ll need to make a few minor changes to the SSH server. The configuration changes below are needed to enable authentication using CA keys.

echo "TrustedUserCAKeys /etc/ssh/ca.pub" >>/etc/ssh/sshd_config

echo "AuthorizedPrincipalsFile %h/.ssh/authorized_principals" >>/etc/ssh/sshd_config

echo "mysocket_ssh_signed" > /home/ubuntu/.ssh/authorized_principals

Finally, also make sure to get the Public key for the CA (mysocketctl socket show) and copy it into the ca.pub file (/etc/ssh/ca.pub)

Now the server is configured to work with and allow authentication based on signed SSH keys from the mysocket certificate authority. Note that all signed certificates will have two principles, the email address of the authenticated user, as well as ‘mysocket_ssh_signed’. In the example configuration above, we told the server to map users with the principle ‘mysocket_ssh_signed’ to the local user ubuntu.

Now we’re ready to connect, but instead of making the ssh command even longer, I’m going to add the following to my ssh config file ~/.ssh/config

Host *.edge.mysocket.io
    ProxyCommand bash -c 'mysocketctl client ssh-keysign --host %h; ssh -tt -o IdentitiesOnly=yes -i ~/.ssh/%h %r@%h.mysocket-dummy >&2 <&1'

Host *.mysocket-dummy
    ProxyCommand mysocketctl client tls --host %h

The above will make sure that for all ssh sessions to *.edge.mysocket.io we start the authentication flow, fetch a TLS client certificate, and set up the TLS tunnel. We’ll also submit an SSH key signing request, which will result in a short-lived signed SSH certificate that will be used for authenticating the SSH user.

Now the user can just SSH like this, and the whole workflow will kick-off.

ssh ubuntu@crimson-thunder-8434.edge.mysocket.io

For those interested, the ssh certificate will end up in your ~/.ssh/ directory and will look like this.

$ ssh-keygen -Lf ~/.ssh/nameless-thunder-8896.edge.mysocket.io-cert.pub
/Users/andreetoonk/.ssh/nameless-thunder-8896.edge.mysocket.io-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:0u6TICEhrISMCk7fbwBi629In9VWHaDG1IfnXoxjwlg
        Signing CA: ECDSA SHA256:MEdE6L0TUS0ZZPp1EAlI6RZGzO81A429lG7+gxWOonQ (using ecdsa-sha2-nistp256)
        Key ID: "atoonk@gmail.com"
        Serial: 5248869306421956178
        Valid: from 2021-02-13T12:15:20 to 2021-02-13T12:25:20
        Principals:
                atoonk@gmail.com
                mysocket_ssh_signed
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

With this, users can SSH to the same server as before, but the cool thing is that the server won’t need to know any traditional known credentials for its users. Things like passwords or a public key entry in the authorized_keys file belong to the past. Instead, with the help of mysocketctl, the user will present a short-lived signed ssh certificate, which the server will trust.

With this, we achieved true Single Sign-on (SSO) for your SSH servers. Since the certificates are short-lived, five minutes in the past(to allow for time drift) to five minutes in the future, we can be sure that for each log-in the authentication and authorization flow was successful.

Give it a try yourself using my SSH server

If you’re curious about what it looks like for the user and want to give it a try? I have a test VM running on 165.232.143.236, it has firewall rules to prevent SSH traffic from the Internet, but using mysocket, anyone with a gmail.com account can access it. I encourage you to give it a spin, it will take you less than a minute to set up, just copy-paste the one-time setup config.

Onetime setup

If you’re using a Mac laptop as your client, you’ll need the mysockectl tool which will request the short-lived certs and will setup up the TLS tunnel. To install the client just copy-paste the below (for Mac only, see download.edge.mysocket.io for other platforms).

curl -o mysocketctl https://download.edge.mysocket.io/darwin_amd64/mysocketctl 

chmod +x ./mysocketctl
sudo mv ./mysocketctl /usr/local/bin/

To make it easy to use we’ll add the following to our ssh client config file. This is a one-time setup and will make sure the ssh traffic to *.edge.mysocket.io is sent through the mysocketctl client tool.

cat <<EOF >> ~/.ssh/config
Host *.edge.mysocket.io
 ProxyCommand bash -c ‘mysocketctl client ssh-keysign --host %h; ssh -tt -o IdentitiesOnly=yes -i ~/.ssh/%h %r@%h.mysocket-dummy >&2 <&1’
Host *.mysocket-dummy
 ProxyCommand mysocketctl client tls --host %h
EOF

Now you should be able to ssh to my test server using

ssh testuser@frosty-feather-1130.edge.mysocket.io

When the browser pops up, make sure to use the “log in with Google” option, as this socket has been configured to only allow identities that have a Gmail.com email address.

Wrapping up

In this post, we showed how we continued to build on our previous work with “identity-aware sockets”. We Introduced support for identity-aware TCP sockets, by leveraging TLS tunnels and Mutual TLS for authentication.

I like to think of this as a cloud-delivered, authenticating firewall. With this, we can make your services available to the Internet on a very granular basis, and make sure that each flow has an identity attached to it. Ie, we know exactly, on a per TCP flow basis, what identity (user) is using this flow. That’s a really powerful feature when compared to a traditional firewall, where we had to allow SSH traffic from certain network ranges that were implicitly trusted. What we can now do with these identity-aware sockets is rewrite these firewall rules and replace the trusted IP ranges with trusted identities. This is incredibly powerful for those that need strict compliance, and need to answer things like, who (not an IP, but an identity) connected to what when.

We looked at how this can be used to provide zero trust remote access to your SSH servers. And how it can be further extended by using the new SSH key signing service.

That’s it for now, I hope you found this interesting and useful. As always, if you have any questions or feedback, feel free to reach out.

Hungry for more? check out all our demo’s on Youtube here

In this blog post, we’ll introduce an exciting new feature that, with the help of Mysocket, allows you to deploy your own Beyond Corp setup.

What is Zero Trust

The main concept behind Zero Trust is that users shouldn’t just be trusted because they are on your network. This implicit trust problem is something we typically see with, for example, corporate VPNs. With most corporate VPN’s once a user is authenticated, the user becomes part of the corporate network and, as a result, has access to many of the resources within the corporate infrastructure. In other words, once you’re on the VPN, you’re within the walls of the castle, you’re trusted, and you have lots of lateral access.

The world is changing, and the once traditional approach of trusting devices on your network are over. A few years ago, Google started the journey to their implementation of Zero trust called Beyond Corp. One of the core building blocks to get to a Zero Trust model is Identity aware application proxies. These proxies can provide strict access control on a per-application granularity, taking the users’ identity and contexts such as location and device status into account.

Identity aware proxies

As of today, the Mysocket proxies have support for OpenID Connect, and with that, your sockets are identity-aware. This means that mysocket users can now enable authentication for their services and provide authorization rules.

mysocketctl connect \
   --port 3000 \
   --name "My Identity aware socket" \
   --cloudauth \
   --allowed_email_domains "mycorp.com" \
   --allowed_email_addresses "contractor1@gmail.com,john@doe.com"

The example above shows how we can enable authentication by using the “cloudauth” CLI parameter. With cloudauth enabled, users will be asked to authenticate before access to your service is granted. Currently, we support authentication using Google, Facebook, Github, or locally created accounts.

The authentication flow uses OpenID connect to interact with the various Identity providers (IDP); thus, we can easily add more Identity providers in the future. We’re also looking at SAML as a second authentication flow. Please let us know if you have a need for more IDP’s or SAML, and we’ll work with you.

Authorizations rules

In addition to what the Identity Service Provider (IDP) provides, mysocket also provides two authorization rules. The — allowed_email_domain allows users to specify a comma-separated list of email domains. If your users are authenticating using their mycorp.com email address, then by adding this domain as an allowed_email_domain, will make sure only users with that domain have will be granted access.

Since multiple identities providers are supported, it’s easy to extend access to contractors or other 3rd party users. To provide access to external contractors that are not part of your mycorp.com domain, we can use the allowed_email_addresses parameter to add individual identities. This is great because now you can provide these contractors access without creating corporate accounts for them.

These are just two authorization rules; we’re planning to add more types of rules in the near future. Additional authorization rules that come to mind are, Geo-based rules (only allow access from certain countries or regions) or time of day type rules. If you require these types of rules or have suggestions for additional authorization rules, please let us know!

VPN replacement

One of the unique features of mysocket is that the origin server initiates the connection to the Mysocket edge. This means that the origin servers can be on a highly secure network that only allows outbound connections. Meaning the origins can be hosted behind strict firewall rules or even behind NAT, like for example, a Private AWS VPC. With this, your origin server remains private and hidden.

With the addition of authentication and authorization to Mysocket, we can now, on a very granular basis, provide access to your private services. Combining the secure outbound tunnel property and the identity-aware sockets, we can now look at this to provide an alternative to VPNs, while providing much more granular access to private or corporate resources.

Example use case

Imagine a scenario where you work with a contractor that needs access to one specific private application, say an internal wiki, ticket system, or the git server in your corporate network. With the traditional VPN setup, this means we’d need to provide the contractor with a VPN account. Typically this means the contractor is now part of the corporate network, has a corporate user account, and now has access to much more than just the application needed.

Instead, what we really want is to only provide access to the one application and be very granular in who has access. With the addition of identity-aware sockets, this is now possible.

Demo time!

Alright, let’s give this a spin, demo time! In this demo, we’re making a Grafana instance that’s on a private network and behind two layers of NAT available to our employees as well as a contractor.

We’ll start by setting up the socket and tunnel using the “mysocketctl connect” command.

This works great for demo’s; for more permanent setups, it’s recommended to use “mysocketctl socket create” and “mysocketctl tunnel create” so that you have a permanent DNS name for your service.

mysocketctl connect \
   --port 3000 \
   --name "My Identity aware socket" \
   --cloudauth \
   --allowed_email_domains "mycorp.com" \
   --allowed_email_addresses "andree@toonk.io,john@doe.com"

With this, we created a socket on the mysocket.io infrastructure, enabled authentication, and provided a list of authorization rules. The same command also created the secure tunnel to the closest mysocket.io tunnel server, and we’re forwarding port 3000 on localhost to the newly created socket.

Next, we launch a Grafana container. For fun, I’m passing in my AWS cloudwatch credentials, so I can create some dashboards for my AWS resources. I’ve configured grafana for proxy authentication. Meaning it will trust mysocket.io to do the authentication and authorization. Grafana will use the HTTP headers added by mysocket to determine the user information.

[auth.proxy]
enabled = true
header_name = X-Auth-Email
header_property = username
auto_sign_up = true
headers = Email:X-Auth-Email

The complete example grafana.ini config file I used can be found here. Now we’re ready to launch Grafana. I’m doing this from my laptop, using Docker.

docker run -i -v grafana.ini:/etc/grafana/grafana.ini \
 -e “GF_AWS_PROFILES=default” \
 -e “GF_AWS_default_ACCESS_KEY_ID=$ACCESS_KEY_ID” \
 -e “GF_AWS_default_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY” \
 -e “GF_AWS_default_REGION=us-east-1” \
 -p 3000:3000 grafana/grafana

Grafana is now listening on localhost port 3000. The mysocket connection we created earlier is relaying authenticated and authorized traffic to that local socket. With that, we should now be able to test and see if we have access to Grafana.

Wrapping up

In this article, we introduced identity aware sockets. We saw how Mysocket users can easily enable authentication for their HTTP(S) based sockets and how OpenID connect is used for the authentication flow to Google, Facebook, or Github (for now). We then looked at how authorization rules can be added by either matching the email domain or even a list of email addresses.

With this, it’s now easy to provide access to internal applications, from any device, any time anywhere, without the need for a VPN.

If you’re in the world of cloud infrastructure, then you’ve heard of Kubernetes. Some of you are experts already, while some of us are just learning or getting started. In this blog, we’ll introduce a mysocket controller for Kubernetes and demonstrate how easy it is to use mysocket.io as your cloud-delivered load balancer for your Kubernetes Services. If you’re a Kubernetes user already, then it should just take a minute to get this mysocket controller setup.

See this video for a demo of the Mysocket.io integration with Kubernetes

Pods, Deployments, and Services

Before we continue, let’s review some of the main Kubernetes building blocks we will be using to make this work.

A typical workload running in Kubernetes will look something like the diagram below. It typically starts with a deployment, in which you define how many replicas (pods) you’d like.

Since pods are ephemeral and can scale in and out as needed, the Pod IP addresses will be dynamic. As a result, communicating with the pods in your deployment from other Deployments would require constant service discovery, which may be challenging for certain apps. To solve this, Kubernetes has the concept of a Service. A service acts as a logical network abstraction for all the pods in a workload and is a way to expose an application running on a set of Pods as a network service.

In the diagram below the service is reachable via 10.245.253.152 on port 8000. The Service will make sure traffic is distributed over all the healthy endpoints for this service.

Taking your Service Global

Now that we know how a service makes your workload available within the cluster, it’s time to take your workload global! Kubernetes has a few ways to do this, typically using an ingress service. We’re going to use the architecture as outlined in the diagram below. We’ll use Mysocket.io to create a secure tunnel between our ‘myApp’ Service and the Mysocket cloud. From there on, it will be globally available via its anycasted edge nodes.

To make this work, we’ll deploy a controller pod that runs the mysocketd container. The Container Image can be found on Docker Hub, and the corresponding Dockerfile on our github repo here.

The mysocketd controller does two things:

1) it subscribes to the Kubernetes API and listens for events related to services. Specifically, it will watch for service events that have the annotation mysocket.io/enabled

If a service has the following annotation mysocket.io/enabled: “true” then the mysocketd app will start a thread for that service.

2) In the per service thread, using the Mysocket API, a new mysocket.io “Socket” object is created if needed. This will give the “myApp” service a public DNS name and public IP. Next, it will check with the mysocket.io API to see if a tunnel already exists; if not, then a new one is created.

Finally, the secure tunnel is established, and the “myApp” service is now globally available and benefits from the mysocket.io high-performance infrastructure.

Demo time. How to Deploy mysocketd to your cluster

The easiest way to get started is to download the mysocketd workload yaml file from the Mysocket Kubernetes controller repository  and update the following three secrets to the ones for your mysocket account (line 14,15,16).

email: <mysocket login email in base64> 
password: <mysocket password in base64> 
privatekey: <mysocket private ssh key in base64>

Then simply apply like this and you’re good to go!

kubectl apply -f mysocketd.yaml

Cool, now we have the controller running!

A simple demo work load

Next up, we’d like to make our myApp service available to the Internet by using the mysocket.io service. I’m going to use this deployment as our demo app. The deployment consists of three pods, with a little python web app, printing its hostname.

Next up, we’ll build a service (see definition here) that acts as an in-cluster load balancer for the demo workload, and we’ll request the Service to be enabled for Mysocket.io.

$ kubectl apply -f demo-deploy.yaml

$ kubectl apply -f demo-service.yaml

$ kubectl get all -n demo-app
NAME                            READY   STATUS    RESTARTS   AGE
pod/demo-app-6896cd4b88-5jzgn   1/1     Running   0          2m22s
pod/demo-app-6896cd4b88-78ngc   1/1     Running   0          2m22s
pod/demo-app-6896cd4b88-pzbc7   1/1     Running   0          2m22s
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/demo-service   ClusterIP   10.245.114.194   <none>        8000/TCP   64s
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-app   3/3     3            3           2m22s
NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-app-6896cd4b88   3         3         3       2m22s

The same can be done for existing services: to do so, just edit your existing Service definition and add the “mysocket,io/enabled” annotation, like this:

kind: Service
metadata:
  annotations:
    mysocket.io/enabled: "true"

After applying this change, the mysocketd controller will detect that the service has requested to be connected to the Mysocket global infrastructure. The controller will create the Socket and Tunnels as needed and start the tunnel.

A quick load balancing test shows that requests to our global mysocket endpoint are now being load-balanced over the three endpoints.

$ for i in {1..60}; do  curl -s \ 
  https://blue-snowflake-1578.edge.mysocket.io/ & ;done | sort | uniq -c | sort -n  
  
  19 Server: demo-app-6896cd4b88-pzbc7  
  20 Server: demo-app-6896cd4b88-5jzgn  
  21 Server: demo-app-6896cd4b88-78ngc  
  60

Just like that, in a few seconds, your workload is globally available. Pretty neat he!? Make sure to watch the video recording of this demo as well.

Wrapping up

In this article, we looked at how we can easily connect Kubernetes Services to Mysocket.io. We saw that it was easy to use. All that is needed is to: 1) start the Mysocketd controller, and: 2) add the mysocket annotation to your Services. By doing so, we are giving the Kubernetes service a publicly reachable IP address, a TLS cert, and it instantly benefits from the mysocket.io Anycast infrastructure. Pretty powerful and easy to use.

I should add that this Mysocket Controller for Kubernetes is for now just a starting point, ie. an MVP integration with Kubernetes. There are ways to improve this, both in terms of user-friendliness and high availability (i.e. more than one pod). The code for this Kubernetes integration is open source, and we’d be happy to accept improvements. Mostly it serves as an example of what’s possible and how we can continue to build on the Mysocket services.

Last but certainly not least, I’d like to thanks Brian for his code to improve the various mysocketctl client libraries. Also, a big thanks to Bas Toonk (yes, my brother) for his help with this MVP. Most of this was his work, thanks buddy!

Finally, I hope this sparked your imagination, and you’ll all give it a whirl, and let me know your experience.

Last week AWS had a major outage in its US-EAST1 region, lasting for most of the day, just before the big black Friday sales! Incidents like this are a great reminder of the importance of multi-region or even multi-cloud deployments for your services.

Depending on your “cloud maturity” and your products’ complexity, you may already be there or just getting started. Either way, in today’s blog, we will take a look at how we can use mysocket’s load balancing features to make deployments over multi-region easier.

A global load balancing service

In earlier blogs, we looked mostly at how the mysocket.io tunnel service can help securely connect your resources that may be behind NAT and firewalls to the Internet. In this article, we’ll look at Mysocket’s global load balancing feature.

Three types of load balancers

Mysocket today supports three different types of cloud-native load balancers.

1) Application load balancers, for your HTTP and HTTPS services.
2) Network load balancer, for your TCP services.
3) TLS Load balancer, for your TCP services, where we take care of the encryption.

When deploying Mysocket with your services, you now have a new front door. It just happens to be a front door that is anycasted, and as such has many doorbells around the globe made available to you. As a result, you’re always close to your users.

Demo: Creating a multi-region service in two minutes!

Alright, time to get building! In this demo, we’ll continue using the Gif service we built in our last blog. Just like your workloads, I think this is a “critical” service, so we need it to be deployed in multiple regions. The service will need a 100% uptime, which means that even if a region is down, the Gif service will need to be available to our users.

Infrastructure as code

In this example, we’re going to deploy the service on Digital ocean VM’s in two of its regions, New York and Toronto. We’re big fans of infrastructure as code and require our service to be deployed with a simple deploy script. So we’ll use Terraform to spin up the instances and use a cloud-init script to bootstrap the necessary work to install the required software, start the Gif service and connect it to the Mysocket global infrastructure.

Step 1: terraform

We’re using terraform to define what type VM’s we’d like, which regions we want them to be deployed in, and how many per region. For this demo we’re using just two VM’s in two regions. But I’m sure, by just looking at the terraform file, you can see how easy it is to deploy many VM’s per region by just changing the ‘count’ number. It’s also easy to add additional regions. So scaling our Gif service for the future will be easy ;)

Step 2: cloud-init

The VM’s we’re launching are just vanilla Ubuntu machines. So we’ll need to modify them slightly to get our software and our ‘secrets’ on the machine. To do that, we’ll use cloud-init.
Cloud-init allows us to define a set of tasks that need to be executed when the VM is created. In our case, we’re asking it to create a user ‘mysocket’, we’re adding the Giphy API key, and mysocket credentials as secrets to the mysocket home directory. Finally, we’re telling it to download and execute a bootstrap script.

Step 3: bootstrapping and starting our services.

Cloud-init’s last step was to start the bootstrap script that it download from here. Basically, all it does is install the two python packages (mysocketctl and giphy_client) and download and start two more programs. The first program is our Gif application, which will run on port 8000. The second program is a small python script that makes sure the VM registers and connects to the mysocket.io infrastructure.

The code to connect to mysocket is available on github. It’s pretty easy, and I’m sure it’s low friction to get started with, even if you have limited experience. Below are the most important parts:

#These variables were made available as secrets in cloud Init.
username = os.getenv("email")
password = os.getenv("password")
socket_id = os.getenv("socket_id")

#Login to mysocket API and get a token. We can contruct the Auth header using this token.
token = get_token(username, password)
authorization_header = {
   "x-access-token": token["token"],    
   "accept": "application/json",    
   "Content-Type": "application/json",}
   
# register the service by creating a new tunnel for this VM
tunnel = new_tunnel(authorization_header, socket_id)
# setup the tunnel to mysocket and ready to serve traffic!
ssh_tunnel(port, tunnel["local_port"], ssh_server, ssh_user)

Make sure you take a quick peak at the code, it’s pretty easy to get started with and a good “getting started” example for both terraform and cloud-init.

And with that, we now have four VM’s in two regions. All four VM’s registered with the Mysocket service and are now serving our ‘critical’ Gif app. You can see the demo service yourself here: https://fluffy-bunny-5697.edge.mysocket.io/

The example below shows how the mysocket load balancing service distributes traffic evenly over all four of the origin servers in New York and Toronto.

$ for i in {1..20}; do  curl -s \
 https://fluffy-bunny-5697.edge.mysocket.io/ | grep Server; 
done | sort | uniq -c

   5 Server: compute-000-nyc1
   5 Server: compute-000-tor1
   5 Server: compute-001-nyc1
   5 Server: compute-001-tor1

Horizontally scaling the service is easy; simply change the count number in the terraform file, or add / remove regions. Since the VM’s will call the mysocket api on boot, these new VM’s will automatically become part of the load balancing pool. Pretty neat and easy, right?!

Failover scenarios

So what happens when a server or region becomes unavailable? There are a few things that make this as painless as possible for users. First, the origin service can de-register itself; this will allow for the most graceful scenario.

For less graceful scenarios, mysocket load balancers will, after 60 seconds automatically detect that a tunnel has gone down and take the origin out of the load balancing pool. And even during this 60-second degradation before the tunnel is declared down, our load balancers will use a 10 second connect time out when connecting to origin service and automatically fail back to the remaining origins. So, all in all, failures should be hidden as much as possible from your users.

Wrapping up

In this blog, we looked at how Mysocket can be used as a global load balancer, for your multi-region load deployments. In our demo, we looked at two Digital ocean regions, but this could also be over multiple AWS regions, or even Multi-cloud, with one cluster in AWS, one in Digital Ocean, and throw in some Google cloud for good measure.

We saw how Mysocket provides users with a global anycasted ingress point and provides seamless load balancing for your services. Best of all, it only took us 90 seconds to get all of this going! I guess it’s fair to say that Mysocket makes going multi-region and even multi-cloud easier.

In my last blog post, I announced the mysocket.io service and demonstrated how to get started quickly. It’s been great to see people signing up and giving it a spin! Your feedback has been great, motivating, and has helped make the service better already.

Most users that gave mysocket a try used the mysocketctl connect (aka “quick connect”) feature. This is the easiest way to get started and instantly creates a global socket, great for quick testing. However, when you’re done and exit the program, the “connect” feature cleans up the socket. It’s easy to create new ones, but each time with a different name. Not surprisingly then, that a few users asked the question about sockets with static names.

Static names for your services

Most residential ISPs give their customers a dynamic IP address (one that changes from time to time) through DHCP. If you have a service you’d like to make available but don’t have a static IP, then this post is for you!

Perhaps you have a server at home, either a fancy setup, rack-mounted and UPS for emergency power, or perhaps just a modest Raspberry Pi. Either way, you’d like to make this available from the Internet, so you can access it when you’re not at your home network, or perhaps you are home, but your nice cooperate VPN client blocks access to your local network. Or maybe you’d like to make it available to your friend.

That’s great, but you have two challenges:

1) the server sits behind NAT, so you can’t simply connect to the server from the Internet.

2) your ISP gives you dynamic IPs, so every few days/weeks, the IP changes, so it’s hard to keep track of the IP. Ideally, you could use a static name, that no matter your dynamic IP, remains stable.

So what you’d need is a static DNS name! Good news, that’s the default on the Socket primitive.

Mysocket primitives

Let’s take a more in-depth look at the two main primitives that make up the mysocket service: Sockets and Tunnels

A Global Socket

The first primitive we need is a socket object. This object is the public endpoint for your service and can be created like this:

mysocketctl socket create \
	--type http \
	--name "my service at home"

This, among other things, returns a static DNS name that is yours to use. Think of it as your global public endpoint for your load balancer. In this case, mysocket runs the load balancer, and it’s made highly available through our anycast setup.

In the example above, we created an HTTP/HTTPS socket. Other options are TCP and TLS sockets. In those cases, the API will return not only a static DNS name for your service but also your own dedicated static TCP port number. A typical use case for a TCP socket is making your ssh service available.

Tunnels

The second primitive is a tunnel. A tunnel object represents your origin service and the secure connection between the origin and the mysocket global infrastructure. Let’s look at the example below.

mysocketctl tunnel create \
	--socket_id 1fab407c-a49d-4c5e-8287-8b138b7549c0

When creating the tunnel object, simply pass along the socket_id from the socket you’d like to be connected to.

Putting it together

In summary, we can create a globally available service by first creating a socket of type HTTP/HTTPS, TCP or TLS. This returns a static name for your service and optionally a unique port number. After that, we create a tunnel that links the origin to the socket.

Now that we have created a Socket and Tunnel, it’s time to connect to it, spin up the dataplane, and expose the local service port. We do that using the following command:

mysocketctl tunnel connect \
	--port 8000 \
	--socket_id 1fab407c-a49d-4c5e-8287-8b138b7549c0 \
	--tunnel_id 3f46a01f-ef5b-4b0c-a1ce-9a294be2be03

In the example above, we only create one tunnel for the socket, but nothing stops you from creating multiple tunnels (origins) per socket. In that case, mysocket will load balance over all available tunnels.

Demo time:
exposing a local Gif service to the Internet

Alright, time to look at a simple demo and get our hands dirty. For this demo, I developed a small proof of concept python web service that shows a random Gif from Giphy each time a visitor loads the webpage.

I’m running this service locally on a VM on my laptop. The goal is to make this service publicly available, and overcoming the two levels of NAT and my ISP that hands out dynamic IP addresses. At the end of this demo, I’m able to share a static DNS name with my users, and you’ll be able to try it!

Watch the video below for a live demo.

First, let’s download the demo python code and start the Gif web service

wget https://gist.githubusercontent.com/atoonk/0bfc784feb66ffc03541462fbc945df7/raw/812d839f60ea7bfd98a530a3fc549137abe0b329/gif_service.py

#make sure to update the API key in gif_service.py

python3 ./gif_service.py

Alright, now we have this service running, but it’s only reachable from my local network. Next up, we create a socket and a tunnel.

mysocketctl socket create \
	--name "my local http Gif service" \ 
	--type http

Ok, that returns a socket_id and our static DNS name! In my case, the DNS name is: wandering-shape-7752.edge.mysocket.io

Next up, we’ll use the socket_id to create the tunnel object:

mysocketctl tunnel create \
	--socket_id 1fab407c-a49d-4c5e-8287-8b138b7549c0

Cool, now all we need to do is start the tunnel connection.

mysocketctl tunnel connect \
	--port 8000 \
	--socket_id 1fab407c-a49d-4c5e-8287-8b138b7549c0 \
	--tunnel_id 3f46a01f-ef5b-4b0c-a1ce-9a294be2be03

This will securely connect the local gif web service listening on port 8000, to the mysocket infrastructure and make it available as https://wandering-shape-7752.edge.mysocket.io

Wrapping up

In this blog post, we looked at the two main mysocket primitives, Sockets, and Tunnels. We saw how users can create a Socket object and get a static DNS name and possibly even a dedicated TCP port. In the demo, we used the tunnel connect feature to make port 8000 on my local VM available to the Internet. With that, we made the Internet a little bit better by adding yet another Gif service, that just happens to run on a VM hosted on my laptop.