A few days ago, security researchers published details of a group of vulnerabilities caused by modern CPU architecture called “Meltdown” and “Spectre.” Both vulnerabilities potentially allow software to read parts of system memory it should not be permitted to read.
Almost all modern computers are affected by these vulnerabilities. Of the two, Meltdown is much easier to exploit, but patches are available to prevent exploitation. Spectre is much harder to patch, but luckily also seems to be much harder to exploit. We will apply remediation for Spectre as it is made available.
This blog post explains the steps we have taken and are taking to ensure Travis CI’s Hosted infrastructure is not vulnerable to the Meltdown attack, and includes details of steps we recommend our customers using on-premises Travis CI Enterprise take.
Container-based builds temporarily rerouted
Update: As of roughly 1200 UTC on Jan 9, our container-based infrastructure is fully patched against Meltdown.
sudo: false builds are once again being routed to their usual destination. If you updated IP safelists these changes can now be reverted.
We have two Linux-based infrastructures: one for
sudo: required jobs which run on full Linux virtual machines, and one for our container-based
sudo: false jobs. Our initial focus is on patching the
sudo: false infrastructure, as in this environment multiple users’ jobs run on the same Linux host, which could potentially make a successful Meltdown exploitation more damaging.
By the time you read this post, we will be temporarily routing all
sudo: false jobs to the infrastructure we usually use only for
sudo: required jobs. This minimizes the risk that a successful Meltdown exploitation would leak users’ secrets, and it allows us to roll out patches to the
sudo: false infrastructure more quickly.
We will update this blog post when patching of the
sudo: false infrastructure is complete. Once this is done, we expect to route jobs back to their usual destinations.
While this temporary rerouting is in place, you may encounter slower build boot times. You may also need to update any IP safelists to include IP ranges for “sudo-enabled Linux”.
Furthermore, we use three different infrastructure providers for builds at Travis CI: GCE, AWS, and MacStadium (OSX). These providers are also directly working on preventing exploitation of these vulnerabilities.
You can find their public statements of disclosure and remediation items below:
MacStadium, used on our macOS infrastructure: Update on the Intel Security Issues
Heroku (used by the Travis Platform): Meltdown and Spectre Security Update
These cloud providers have (mostly) finished rolling out fixes for Meltdown, and are now asking customers to patch guest OSes and hypervisors, which is what we are now doing. Details follow:
Build Images, Travis Internal Images, and macOS Hypervisors
Linux Build images
We offer Linux build images based on three different OS versions: Ubuntu Precise, Trusty, and Xenial(in beta). As mentioned above, they come in two different image formats:
sudo: false) - We will be updating our docker build images, which is why we are temporarily rerouting these jobs as we mentioned above.
sudo: required) - We are updating the kernel in the Google Compute base images these build VMs boot from. In the meantime these build VMs are isolated from one another.
macOS Build images
Our macOS base images will be updated this week. All the base images (and the build VMs that are booted from these images) are isolated from one another.
Travis Internal images
Our internal infrastructure - everything that helps Travis CI run that is not a user-facing build image - is a mix of several different images across our cloud providers including AMIs on EC2, Google Compute images on GCE, and Linux/BSD images on our MacStadium OSX infrastructure.
The kernel update for the worker docker image is in progress while we temporarily reroute builds. All other internal images will be updated and internal infrastructure will be rebuilt using these new images in the coming days.
macOS hypervisors (vCenter 6.5) and ESXI hosts
Rolling updates will be performed on both the vCenter hypervisor used in the MacStadium infrastructure and the macOS servers that host the compute infrastructure for macOS builds.
Machine instances running either Travis CI Enterprise platform or workers may also be vulnerable to these attacks. We recommend immediate maintenance to apply the available patches for Meltdown.
Regardless of whether you are running Precise or Trusty (beta) build environments, we recommend updating your worker host machines. If you are running Ubuntu 14.04 as the underlying operating system for your worker hosts, per the standard system requirements, the updates can be applied as follows:
sudo add-apt-repository ppa:canonical-kernel-team/pti sudo apt-get update sudo apt dist-upgrade -y sudo apt-get autoremove sudo reboot
The updates required to the Enterprise Platform instances are very similar to those required for the worker. If you are running Ubuntu 14.04 as the underlying operating system for your platform host, per the standard system requirements, the updates can be applied as follows:
- Log into your Admin Dashboard,
your-domain.tld:8800and shut down Travis CI Enterprise
- SSH into your Travis CI Enterprise platform instance
- Stop the Replicated services with
sudo service replicated stop,
sudo service replicated-operator stop,
sudo service replicated-ui stop
- Apply the kernel updates, with the same process as above for the worker
- After the platform instance restarts, ssh back in
- Restart the Replicated services with
sudo service replicated restart,
sudo service replicated-operator restart,
sudo service replicated-ui restart
If you have any questions about the maintenance required for your Enterprise installation, please email our Enterprise Team.
Of course, if you have any other questions or concerns, please feel free to contact our support team