I love tinkering with new tech, and since it was a cold and snowy weekend here in Vancouver, I figured I’d explore Edge Containers from Stackpath. Earlier this week I read Stackpath’s blog in which they describe their new service: Introducing containers and virtual machines at the edge
Update June 21, 2020: a follow-up article is available here: Building a global anycast service in under a minute
The service offering resonated strongly with me, mostly the ability to deploy your containers or VM’s to many of their locations in one operation, whether that’s an API call or click in the portal. That combined with the fact that Stackpath called out BGP Anycast support in the blog post (for those who know me, I’m a BGP geek), convinced me to give this a spin.
Brand new Feature
So instead of building a snowman, I created an account with stackpath.com. I soon found out their new “Workload” feature is still gated, but within an hour or so Stackpath unlocked this feature for me and it showed up in the Stackpath portal.
Update Feb 12 2019: Stackpath advised me that the “Workload” feature is no longer gated and available for all users now.
Creating workloads
I’ll briefly describe how I created the workload, but also make sure to check out their documentation here.
I gave my workload a name “stackpath-anycast-fun” and used the following docker image “atoonk/stackpath-fun:latest”, more about that later. Next up I exposed port 80 and defined the compute spec for each container. Also, note that I selected the “Add Anycast IP Address.”
Next up I defined what locations I want to deploy the workload to. This is grouped by continent, in my case I used Europe and North America. In each continent, I chose two PoPs: Amsterdam, London, New York, and Los Angeles. So with that my deployment should be live within four datacenters globally, great for my Anycast testing! Finally, I made sure to bring up two instances in each datacenter just so that I can test the load balancing or ECMP routing within a datacenter.
Pricing
At the end, it nicely showed me the pricing. Since I have a particular interest in Anycast, I looked specifically at pricing for that, which at 14 cents per hour comes to $100 per month. All in all for the month for these eight instances & Anycast the price comes to about $627 per month.
Creating a Docker image
Earlier on I defined the Docker image that I wanted to have deployed. Stackpath will pull these images from Dockerhub, in my case I defined atoonk/stackpath-fun:latest. So what does this container do? Basically, it’s a container with Apache + PHP, and a little script that prints the hostname variable using gethostname();. This will allow me to, later on, test what instance I’m hitting by requesting a webpage.
Start the workload
Now that we have a Docker image on Dockerhub and the workload is created it’s time to test it. After you create the workload in the Stackpath portal, it takes about 2 minutes or so for all instances to come online. Eventually, it will look like this.
Cool, all my container instances went from Starting to Running. In addition to an external IP for each container, I also have an Anycast IP: 185.85.196.10
Time to test Anycast
I can hit all the containers via their external IP address using my browser. Next up checked to see if the Anycast address worked, and yes it did! The example below shows I’m hitting one of the LAX instances, which is as expected since that’s the closest location from my home in Vancouver.
I refreshed my browser a few times and indeed saw it flip between the two Los Angeles instances I created: stackpath-anycast-fun-na1-lax-0 and stackpath-anycast-fun-na1-lax-1 (note the last digit).
Global testing
Next up it’s time to test from a few more locations to make sure the global Anycast setup works as expected. To test this, I did a curl from some shell servers in Amsterdam, London, New York and Seattle like this:
time for i in `seq 1 15`; do curl -s 185.85.196.10/server.php | grep stackpath-anycast-fun; done
The screenshot above shows the full high availability and load balancing story in one picture.
My Amsterdam server is hitting the Amsterdam containers and being load balanced between the containers there. Seattle is routed to Los Angeles as expected, and similarly, the New York client is being served by the two containers in New York (JFK).
The one that stands out though is my London shell server. That client is following the Anycast route to New York. That’s odd, especially since I have a cluster deployed in both Amsterdam and London. Also note the last line, showing that to complete this request it took over 2secs, while the client in Amsterdam finished its requests in 200ms.
The London routing mystery
I did a bit of poking to understand why the London server is going to New York, instead of staying local in London. By looking at the routing table, I see the following in London:
> 185.85.196.0/24
*[BGP/170] 3d 13:19:37, MED 38020, localpref 100
AS path: 3356 209 33438 I, validation-state: unverified
[BGP/170] 23:46:15, MED 2011, localpref 100
AS path: 2914 1299 33438 I, validation-state: unverified
[BGP/170] 2w0d 09:36:24, MED 30, localpref 100
AS path: 3257 1299 33438 I, validation-state: unverifiedThe above indicates this router learns the path to 185.85.196.10 via three ISPs: Level3 (AS3356), NTT (AS2914) and GTT (AS3257). The router selected the path via Level3. Level3 then uses the path via Centurylink (AS209) to send it to Stackpath (AS33438).
This shows us that Stackpath (AS33438), is buying connectivity from Centurylink in the USA. Since Level3 prefers the US path via Centurylink all Level3 customers in Europe are routed to the US, New York in this case. Just announcing Anycast routes out of the US and not scoping them (make sure the announcements stay in the US) will inevitably cause the Anycast routing challenges we see here.
Update Feb 12 2019: The Anycast routing issue has now been resolved by the Stackpath network team. Routing from my test client in London is now correctly routed to the container cluster in London.
Summary
In summary, it was pretty easy to get going with Stackpath’s container service. It’s cool to be able to deploy your workloads around the world with just a single click. The icing on the cake for me is the global load balancing with Anycast feature.
Anycast used to be a feature used by mostly DNS operators and CDN folks. However over the last two years or so we’ve seen infrastructure providers such as Packet and AWS making this available to their customers directly. With these features, folks can now really deploy their high available services globally without necessarily needing to rely on CDN’s or any DNS tricks (or better, build your own DNS tricks now).
I think the pricing is a bit steep, but I’m guessing since it’s a new feature with possibly limited supply, this is one way to get to market while ramping up their capacity. The other thing to keep in mind is that folks like Packet and Stackpath are focusing on the Edge. They are deployed in some of the most expensive and most popular datacenters around the world. This will give its customers great low latency connectivity, but space and power are limited in these facilities, possibly pushing up the pricing.
Going forward I’d love to see some more features around, monitoring, metrics and logging for my containers.
Realizing this is early on, my first experience with Stackpath was a joy, it was easy to use, great support (thanks Ben and Sean) and I look forward to what’s next.
