Faster Carthage bootstrap with cartfilediff

Traffic travelling quickly on a multi-lane road Photo credit: Unknown via Visualhunt / CC BY.

At YPlan, we use Carthage to manage our dependencies for all of our iOS projects and we think it's a great tool. However, one feature which is not supported is only building dependencies which have changed when bootstrapping in response to a new Cartfile.resolved.

For some time we were using an optimisation suggested on the Thoughtbot blog, but this has some shortcomings. The approach is to cache the Carthage directory alongside the Cartfile.resolved file. We use Travis CI’s built in cache, but you could use any caching mechanism. When a new build job executes the cached Cartfile.resolved is compared against the current Cartfile.resolved and if the files are different, a complete carthage bootstrap is performed.

We have a modest number of dependencies, but some of these contain a number of frameworks, and build time spent compiling dependencies was approaching one hour. This time is greatly reduced for subsequent builds where the Cartfile.resolved is less likely to have modifications, but such a lengthy build time is clearly undesirable.

CartfileDiff

Enter CartfileDiff. This tool uses the CarthageKit framework, provided by Carthage to compare the two Cartfile.resolved files and outputs a list of any dependencies which have changed version or were added. This list can then be passed to carthage bootstrap to build only the necessary dependencies, which is much faster.

Our CI setup was described in detail in my post earlier this year, Continuous Integration for iOS Projects, but the required steps to optimise your bootstrap are shown below.

We use this tool with the following two scripts on Travis:

script/bootstrap

#!/bin/bash

carthage bootstrap $@ --platform iphoneos --no-use-binaries || exit $?

This script bootstraps the project, and can be provided with an optional list of dependencies to build.

script/bootstrap-if-needed

#!/bin/sh

SCRIPT_DIR=$(dirname "$0")
BOOTSTRAP="$SCRIPT_DIR/bootstrap"
CACHED_CARTFILE="Carthage/Cartfile.resolved"

if [ -e "$CACHED_CARTFILE" ]; then
  OUTDATED_DEPENDENCIES=$(cartfilediff "$CACHED_CARTFILE" Cartfile.resolved)

  if [ ! -z "$OUTDATED_DEPENDENCIES" ]
  then
    echo "Bootstrapping outdated dependencies: $OUTDATED_DEPENDENCIES"
    "$BOOTSTRAP" "$OUTDATED_DEPENDENCIES"
  else
    echo "Cartfile.resolved matches cached, skipping bootstrap"
  fi
else
  echo "Cached Cartfile.resolved not found, bootstrapping all dependencies"
  "$BOOTSTRAP"
fi

cp Cartfile.resolved Carthage

This script compares the cached Cartfile with the latest one, and calls the bootstrap script with the relevant arguments.

Finally, the .travis.yml looks like this:

language: objective-c
osx_image: xcode7.3
cache:
  directories:
  - Carthage
before_install:
- curl -L -O https://github.com/Carthage/Carthage/releases/download/0.16.2/Carthage.pkg
- sudo installer -pkg Carthage.pkg -target /
- curl -L -O https://github.com/YPlan/CartfileDiff/releases/download/0.1/CartfileDiff.pkg
- sudo installer -pkg CartfileDiff.pkg -target /
install: true
script: script/cibuild

Our script/cibuild simply calls script/bootstrap-if-needed, then calls xcodebuild.

We install the cartfilediff tool from a GitHub Release.

As you can see, it's not much work to modify an existing CI setup to take advantage of CartfileDiff.

Performance notes

Carthage

As a general optimisation, make sure to use the --platform argument with carthage bootstrap to only build frameworks for required platforms, e.g.

carthage bootstrap --platform iOS

Travis CI

If you're using Travis for CI, it's important to understand how caching works. When a new branch is created, a new cache is created for the branch as a copy of the master branch’s cache. If you aren't using the master branch, master is outdated, or you have otherwise cleared the cache, your new branches will start with an empty cache, and will therefore take much longer to build.

Finally

In the long term, Carthage contributors have discussed bootstrap improvements which would be smarter about what to build. In the meantime, I hope you find this tool useful.

Further reading