What this change encompasses
As of 09:25 PST on Monday, November 14th, 2016, we added the following public IPs to our container-based build infrastructure:
The complete list of our public IPs is documented in our docs.
Why we made this change
We know that many of you rely on knowing our public IPs to allow traffic from us to sensitive resources and we understand that making changes like adding new IPs will cause some of your builds to fail.
Given that, we would not normally make a change like this without prior notification and so we wanted to provide some background on why we chose to make this type of change yesterday, without any prior warning.
Our container-based builds run on compute resources in EC2 and we use Auto-Scaling Groups to scale up and down the number of hosts, based on our build throughput. We run these builds in AWS’s
us-east-1 region, which is composed of multiple Availability Zones. In keeping with AWS’s best practices, we use NAT instances that are dedicated to each AZ, which helps us distribute the traffic load. We’d been using two of the AZs in us-east-1 prior to yesterday. At the end of last week we began seeing launch failures in both AZs due to unavailability of the instance type we use (c3.8xlarge). Unfortunately, this launch failures started happening more frequently on Monday morning, enough that we began seeing a large (and growing) backlog for our container-based builds. At that point, we made the hard decision to expand our usage into additional AZs in
us-east-1. This decision required that we add additional NAT hosts with new public IPs. We quickly implemented this expansion and were able to start bringing more hosts online and quickly catchup on the backlog we’d been accruing. As soon as we knew the new IPs, we added them to our docs and began communicating them via our status page, Twitter, etc.
We are sorry for any disruptions in your builds that this change caused and appreciate your understanding why this was the best option for us to take at the time.
As always, if you have any questions, please feel free to email firstname.lastname@example.org