Travis Response to Meltdown Attack

| Travis Team | news

A few days ago, security researchers published details of a group of vulnerabilities caused by modern CPU architecture called “Meltdown” and “Spectre.” Both vulnerabilities potentially allow software to read parts of system memory it should not be permitted to read.

Almost all modern computers are affected by these vulnerabilities. Of the two, Meltdown is much easier to exploit, but patches are available to prevent exploitation. Spectre is much harder to patch, but luckily also seems to be much harder to exploit. We will apply remediation for Spectre as it is made available.

This blog post explains the steps we have taken and are taking to ensure Travis CI’s Hosted infrastructure is not vulnerable to the Meltdown attack, and includes details of steps we recommend our customers using on-premises Travis CI Enterprise take.

Container-based builds temporarily rerouted

Update: As of roughly 1200 UTC on Jan 9, our container-based infrastructure is fully patched against Meltdown. sudo: false builds are once again being routed to their usual destination. If you updated IP safelists these changes can now be reverted.

We have two Linux-based infrastructures: one for sudo: required jobs which run on full Linux virtual machines, and one for our container-based sudo: false jobs. Our initial focus is on patching the sudo: false infrastructure, as in this environment multiple users’ jobs run on the same Linux host, which could potentially make a successful Meltdown exploitation more damaging.

By the time you read this post, we will be temporarily routing all sudo: false jobs to the infrastructure we usually use only for sudo: required jobs. This minimizes the risk that a successful Meltdown exploitation would leak users’ secrets, and it allows us to roll out patches to the sudo: false infrastructure more quickly.

We will update this blog post when patching of the sudo: false infrastructure is complete. Once this is done, we expect to route jobs back to their usual destinations.

While this temporary rerouting is in place, you may encounter slower build boot times. You may also need to update any IP safelists to include IP ranges for “sudo-enabled Linux”.

Cloud Providers

Furthermore, we use three different infrastructure providers for builds at Travis CI: GCE, AWS, and MacStadium (OSX). These providers are also directly working on preventing exploitation of these vulnerabilities.

You can find their public statements of disclosure and remediation items below:

These cloud providers have (mostly) finished rolling out fixes for Meltdown, and are now asking customers to patch guest OSes and hypervisors, which is what we are now doing. Details follow:

Build Images, Travis Internal Images, and macOS Hypervisors

Linux Build images

We offer Linux build images based on three different OS versions: Ubuntu Precise, Trusty, and Xenial(in beta). As mentioned above, they come in two different image formats:

Docker (sudo: false) - We will be updating our docker build images, which is why we are temporarily rerouting these jobs as we mentioned above. GCE (sudo: required) - We are updating the kernel in the Google Compute base images these build VMs boot from. In the meantime these build VMs are isolated from one another.

macOS Build images

Our macOS base images will be updated this week. All the base images (and the build VMs that are booted from these images) are isolated from one another.

Travis Internal images

Our internal infrastructure - everything that helps Travis CI run that is not a user-facing build image - is a mix of several different images across our cloud providers including AMIs on EC2, Google Compute images on GCE, and Linux/BSD images on our MacStadium OSX infrastructure.

The kernel update for the worker docker image is in progress while we temporarily reroute builds. All other internal images will be updated and internal infrastructure will be rebuilt using these new images in the coming days.

macOS hypervisors (vCenter 6.5) and ESXI hosts

Rolling updates will be performed on both the vCenter hypervisor used in the MacStadium infrastructure and the macOS servers that host the compute infrastructure for macOS builds.

Enterprise

Machine instances running either Travis CI Enterprise platform or workers may also be vulnerable to these attacks. We recommend immediate maintenance to apply the available patches for Meltdown.

Worker Updates

Regardless of whether you are running Precise or Trusty (beta) build environments, we recommend updating your worker host machines. If you are running Ubuntu 14.04 as the underlying operating system for your worker hosts, per the standard system requirements, the updates can be applied as follows:

sudo add-apt-repository ppa:canonical-kernel-team/pti
sudo apt-get update
sudo apt dist-upgrade -y
sudo apt-get autoremove
sudo reboot

Platform Updates

The updates required to the Enterprise Platform instances are very similar to those required for the worker. If you are running Ubuntu 14.04 as the underlying operating system for your platform host, per the standard system requirements, the updates can be applied as follows:

  1. Log into your Admin Dashboard, your-domain.tld:8800 and shut down Travis CI Enterprise
  2. SSH into your Travis CI Enterprise platform instance
  3. Stop the Replicated services with sudo service replicated stop, sudo service replicated-operator stop, sudo service replicated-ui stop
  4. Apply the kernel updates, with the same process as above for the worker
  5. After the platform instance restarts, ssh back in
  6. Restart the Replicated services with sudo service replicated restart, sudo service replicated-operator restart, sudo service replicated-ui restart

If you have any questions about the maintenance required for your Enterprise installation, please email our Enterprise Team.

Of course, if you have any other questions or concerns, please feel free to contact our support team