Throughout 2017 and the start of this year, demand for both open and closed source macOS builds has continued to increase. As many of you have noticed, our infrastructure simply hasn’t been able to keep up. While we’ve been adding hardware and improving performance throughout this time, it’s clear that we haven’t been able to provide the service we truly want to. Many of you, especially those using our open source infrastructure, have suffered long delays before their macOS builds have run.

Why does macOS suffer so severely from this, but Linux does not?

The number of builds we need to run at any one time can vary greatly over the course of a day, week, or month. With our Linux-based infrastructure, scaling up and down in response to changing demand is relatively easy. With our macOS infrastructure, this hasn’t been true: we can’t easily add and remove servers at the press of a button, and this has made it difficult for us to respond in good time to variations in demand.

Our biggest bottleneck at this time is hardware. We have hit scaling limits on our Storage Area Network (SAN), thus adding more servers has severely diminishing returns. Significant hardware changes are needed to see much improvement at this point.

What are we doing to fix it?

We’re doing just that! Over the last few weeks we have been preparing to move our macOS infrastructure to a new datacenter (with a new SAN), and in the last few days we’ve started sending customer builds to the new hardware. Performance is looking good, and we’re confident that you will see significant improvements in build wait times and build times on macOS.

At our new datacenter, we have a brand-new dedicated SAN along with other new hardware. This helps alleviate the most severe of our bottlenecks immediately. In addition, we are exploring a number of options that would allow us to be more responsive to changing load on this infrastructure, and will be continuing to work on improving all things macOS throughout 2018.

Timelines and ETAs

We’ve already started rolling out our migration in segments and as of today, 35% of all our macOS builds are running in the new datacenter. We are already seeing huge improvements and we hope to continue the roll out over the next weeks. As it stands now, the plan is to have the all-new infrastructure up and running by March 1st.