build, code, linux

Mesos, Modules, and Travis-CI

When my manager asked me to badge the Docker Volume Driver Isolator Module for Mesos (DVDI) project with a Travis-CI build status I thought “Sure, no problem.”

Hoo boy, little did I know…

Build Dependencies

The DVDI project is a Mesos Module and as such Mesos must be present for the DVDI project to be built. Let’s just check the Travis-CI apt package whitelist for Mesos and…not there.

…Okay, so that’s not a big deal, right? With dpkg -x and the Mesos DEB file we can easily grab Mesos and use it in a Travis-CI build.

…Except that building a Mesos module also requires some of the same dependencies necessary to build Mesos, such as:

I thought surely someone has already taken care of this. Well, not that I could find. The big issue is that Travis-CI’s container-based build infrastructure’s apt whitelist is missing several of the packages for the above dependencies:

  • libsvn1-dev
  • libsqlite3-dev:3.8.0
  • libboost-dev:1.53.0
  • python-boto
  • libgoogle-glog-dev
  • protobuf:2.5.0
  • libprotobuf-dev:2.5.0

Now, the folks at Travis-CI do provide an apt-source whitelist for using PPAs to install more recent versions of packages. For instance, it’s possible to use this whitelisted PPA to get a more up-to-date version of Boost. However, it’s not really worth it as the remaining dependencies are still unavailable. If we have to account for five missing build dependencies we may as well include a sixth :)

.travis.yml

And so the next step is to use a PPA to account for a missing build dependency. :) Well, not really a dependency so much as a driver of the build — the GNU compiler toolchain. Here’s the section from the DVDI’s .travis.yml file that defines the apt sources and packages to use:

addons:
  apt:
    sources:
    - ubuntu-toolchain-r-test
    packages:
    - gdb
    - g++-4.8
    - libapr1-dev
    - python-protobuf
    - libaprutil1-dev
    - libcurl4-nss-dev
    - p7zip-full

Makefile

As for the rest of the dependencies, they’re all handled by the DVDI project’s Makefile.

Pre-Compiled Dependency Binaries

The most important piece of the Makefile is on line 142:

DEPS_DIR := /tmp/mesos-ubuntu-12.04-deps

The variable DEPS_DIR defines where all of the build dependencies, including Mesos, are located. This is in effect the prefix for the build dependency sources and installation directories. It’s important that this value is the same both when building with this Makefile locally and on Travis-CI because it’s otherwise not possible to use the dependencies’ pre-compiled binaries.

Because building Mesos and its dependencies can take quite some time, especially if building multiple versions of Mesos, I thought it would be nice to provide pre-compiled versions of the binaries discussed in this blog, and so I did :)

The builds below are for Ubuntu 12.04 and must be inflated to /tmp/mesos-ubuntu-12.04-deps or else errors will occur when execution or linking is attempted as all builds were configured with a prefix beginning with the aforementioned path.

The rest of the file is rather large, so I won’t review the entire thing, only the Subversion dependency, Mesos, and the DVD Isolator module sections.

Subversion

Mesos depends upon Subversion and requires the headers found in the libsvn1-dev package, but this package is missing from Travis-CI’s apt whitelist. Therefore it’s necessary to build Subversion from source to satisfy the Mesos dependency. The Makefile‘s Subversion section begins at line 148 with the variable definitions used for building Subversion.

Subversion Variables

########################################################################
##                            Subversion                              ##
########################################################################
SVN_VER := 1.9.2
SVN_SRC_TAR := subversion-$(SVN_VER).tar.bz2
SVN_SRC_URL := http://apache.mirrors.tds.net/subversion
SVN_OPT_7Z := subversion-$(SVN_VER).7z
SVN_SRC_DIR := $(DEPS_DIR)/subversion-$(SVN_VER)/src
SVN_OPT_DIR := $(DEPS_DIR)/subversion-$(SVN_VER)/opt
SVN_CONFIGURE := $(SVN_SRC_DIR)/configure
SVN_MAKEFILE := $(SVN_SRC_DIR)/Makefile
SVN_SRC_BIN := $(SVN_SRC_DIR)/subversion/svn/svn
SVN := $(SVN_OPT_DIR)/bin/svn
Variable Description
SVN_VER The version of Subversion to fetch/build
SVN_SRC_TAR The name of the Subversion source tarball
SVN_SRC_URL The URL prefix from which to download the Subversion source tarball
SVN_OPT_7Z The name of the 7zip file that contains the Subversion sources along with a complete build of said sources for Ubuntu 12.04 (the version of Linux used by the Travis-CI container build infrastructure)
SVN_SRC_DIR The absolute path to the Subversion sources
SVN_OPT_DIR The absolute path to the directory to which the Subversion build is installed
SVN_CONFIGURE The absolute path to the Subversion build’s configure file
SVN_MAKEFILE The absolute path to the Subversion build’s Makefile
SVN_SRC_BIN The absolute path to the Subversion build’s resulting svn binary
SVN The absolute path to the Subversion build’s installed svn binary

The SQLite Amalgamation

Subversion depends on SQLite 3.80, but Ubuntu 12.04 only includes packages for up to SQLite 3.78. Therefore it’s necessary to download the SQLite amalgamation when building Subversion. The following variables are used to do just that:

SQLITE_VER := 3071501
SQLITE_SRC_ZIP := sqlite-amalgamation-$(SQLITE_VER).zip
SQLITE_SRC_URL := http://www.sqlite.org
SQLITE_SRC_ZIP_DIR := $(SVN_SRC_DIR)/sqlite-amalgamation-$(SQLITE_VER)
SQLITE_C := $(SVN_SRC_DIR)/sqlite-amalgamation/sqlite3.c
Variable Description
SQLITE_VER The version of SQLite on which Subversion depends
SQLITE_SRC_ZIP The name of the SQLite source zip file
SQLITE_SRC_URL The URL prefix from which to download the SQLite source zip file
SQLITE_SRC_ZIP_DIR The absolute path to the directory to which the SQLite sources are decompressed
SQLITE_C The absolute path to the sqlite3.c source file

Fetching the Subversion Sources

The Makefile section below can be invoked with make svn-src and will fetch the Subversion sources if they do not already exist locally.

svn-src: $(SVN_CONFIGURE)
$(SVN_CONFIGURE):
	mkdir -p $(@D) && cd $(@D) && \
		curl -SLO $(SVN_SRC_URL)/$(SVN_SRC_TAR) && \
		$(TARJZ) $(SVN_SRC_TAR) && \
		rm -f $(SVN_SRC_TAR)

Fetching the SQLite Amalgamation Sources

The Makefile section below can be invoked with make svn-sqlite and will fetch the SQLite Amalgamation sources if they do not already exist locally.

svn-sqlite: $(SQLITE_C)
$(SQLITE_C): $(SVN_CONFIGURE)
	cd $(<D) && \
		curl -SLO $(SQLITE_SRC_URL)/$(SQLITE_SRC_ZIP) && \
		unzip $(SQLITE_SRC_ZIP) && \
		rm -fr $(SQLITE_SRC_ZIP) && \
		mv $(SQLITE_SRC_ZIP_DIR) $(@D) && \
		touch $(@D) && touch $@

Creating the Subversion Makefile

The Makefile section below can be invoked with make svn-configure and will execute the configure command to generate Subversion’s Makefile.

svn-configure: $(SVN_MAKEFILE)
$(SVN_MAKEFILE): $(SVN_CONFIGURE) $(SQLITE_C)
	cd $(@D) && $(@D)/configure --prefix=$(SVN_OPT_DIR)

Subversion’s MAKEFLAGS

The Makefile section below defines the variable SVN_MAKEFLAGS to give users the opportunity to provide MAKEFLAGS specific to the Subversion build. If the variable is undefined or an empty string it will inherit the value of MAKEFLAGS.

ifeq "$(origin SVN_MAKEFLAGS)" "undefined"
SVN_MAKEFLAGS := $(MAKEFLAGS)
endif
ifeq ($(strip $(SVN_MAKEFLAGS)),)
SVN_MAKEFLAGS := $(MAKEFLAGS)
endif

Building Subversion

The Makefile section below can be invoked with make svn-make and will execute the make command to build Subversion using the value of SVN_MAKEFLAGS as the MAKEFLAGS value for this recipe.

svn-make: $(SVN_SRC_BIN)
$(SVN_SRC_BIN): MAKEFLAGS=$(SVN_MAKEFLAGS)
$(SVN_SRC_BIN): $(SVN_MAKEFILE)
	cd $(<D) && $(MAKE)

Installing Subversion

The Makefile section below can be invoked with make svn and will execute the make install command to install Subversion.

Sometimes.

Looking closely we can see that the there is some logic behind how the build proceeds:

  • If the build is occurring on an Ubuntu 12.04 system such as Travis-CI and Subversion’s pre-compiled 7z archive is available locally then simply inflate that and we’re done.
  • If the build is occurring on an Ubuntu 12.04 system such as Travis-CI and Subversion’s pre-compiled 7z archive is not available locally but is available via the link provided above then download the archive, cache it locally, inflate it, and we’re done.
  • If the build is occurring on a system other than Ubuntu 12.04 or if the pre-compiled 7z archive is not available locally or online then fetch the Subversion sources and build and install those.
svn: $(SVN)
ifeq ($(call USE_U1204_CACHED_DEP,$(SVN_OPT_7Z)),true)
$(SVN):
	cd $(DEPS_DIR) && \
		7z x $(DEPS_7ZS_DIR)/$(SVN_OPT_7Z) > /dev/null
else
ifeq ($(call USE_U1204_DEP,$(SVN_OPT_7Z)),true)
$(SVN):
	mkdir -p $(DEPS_7ZS_DIR) && \
		cd $(DEPS_7ZS_DIR) && \
		curl -SLO $(DEPS_URL)/$(SVN_OPT_7Z) && \
		cd $(DEPS_DIR) && \
		7z x $(DEPS_7ZS_DIR)/$(SVN_OPT_7Z) > /dev/null
else
$(SVN): MAKEFLAGS=$(SVN_MAKEFLAGS)
$(SVN): $(SVN_MAKEFILE) $(SVN_SRC_BIN)
	cd $(<D) && $(MAKE) install
endif
endif

Other Dependencies

The other dependencies follow logic very similar to Subversion and therefore will not be discussed in this post. Here are links to jump directly to their sections in the Makefile:

Mesos

The Makefile not only provides a way to easily build Mesos and its dependencies, but the Makefile makes it spectacularly simply to build and therefore build against multiple versions of Mesos.

On the first line of the Makefile we find the following:

# MESOS_VERSIONS is a list of space, separated versions of mesos that 
# will be built.
MESOS_VERSIONS := 0.23.0 0.23.1 0.24.0 0.24.1 0.25.0

The variable MESOS_VERSIONS is used by make to dynamically create targets at runtime for each of the specified Mesos versions. Using the above value as an example, the Makefile can be used to build four versions of Mesos with the following commands:

  • make mesos-0.23.0
  • make mesos-0.23.1
  • make mesos-0.24.0
  • make mesos-0.24.1
  • make mesos-0.25.0

The command make mesos will build all versions of Mesos. Keep in mind that these commands take care of fetching the Mesos sources as well as first checking for pre-compiled packages for Ubuntu 12.04 using the logic outlined earlier.

The Mesos build workflow is very similar to Subversion’s from above except it’s wrapped inside of an evaluated definition:

########################################################################
##                                 Mesos                              ##
########################################################################
MESOS_DEPS := $(SVN) $(BOOST) $(BOTO) $(GLOG) $(PROTOBUF) $(PICOJSON)

ifeq "$(origin MESOS_MAKEFLAGS)" "undefined"
MESOS_MAKEFLAGS := $(MAKEFLAGS)
endif
ifeq ($(strip $(MESOS_MAKEFLAGS)),)
MESOS_MAKEFLAGS := $(MAKEFLAGS)
endif

define MESOS_BUILD_RULES
MESOS_VER_$1 := $1
MESOS_SRC_TAR_$1 := mesos-$$(MESOS_VER_$1).tar.gz
MESOS_SRC_URL_$1 := http://archive.apache.org/dist/mesos
MESOS_OPT_7Z_$1 := mesos-$$(MESOS_VER_$1).7z
MESOS_SRC_DIR_$1 := $$(DEPS_DIR)/mesos-$$(MESOS_VER_$1)/src
MESOS_OPT_DIR_$1 := $$(DEPS_DIR)/mesos-$$(MESOS_VER_$1)/opt
MESOS_BUILD_DIR_$1 := $$(MESOS_SRC_DIR_$1)/build
MESOS_CONFIGURE_$1 := $$(MESOS_SRC_DIR_$1)/configure
MESOS_MAKEFILE_$1 := $$(MESOS_BUILD_DIR_$1)/Makefile
MESOS_SRC_BIN_$1 := $$(MESOS_BUILD_DIR_$1)/src/mesos
MESOS_$1 := $$(MESOS_OPT_DIR_$1)/bin/mesos

mesos-$1-src: $$(MESOS_CONFIGURE_$1)
$$(MESOS_CONFIGURE_$1):
	mkdir -p $$(@D) && cd $$(@D) && \
		curl -SLO $$(MESOS_SRC_URL_$1)/$$(MESOS_VER_$1)/$$(MESOS_SRC_TAR_$1) && \
		$$(TARGZ) $$(MESOS_SRC_TAR_$1) && \
		rm -f $$(MESOS_SRC_TAR_$1) && \
		touch $$@

mesos-$1-configure: $$(MESOS_MAKEFILE_$1)
$$(MESOS_MAKEFILE_$1): CXXFLAGS=-I$$(GLOG_OPT_DIR) \
														-I$$(PICOJSON_OPT_DIR)/include \
														-I$$(BOOST_OPT_DIR) \
														-I$$(PBUF_OPT_DIR)/include
$$(MESOS_MAKEFILE_$1): PYTHONPATH=$$(BOTO_OPT_DIR)
$$(MESOS_MAKEFILE_$1): $$(MESOS_CONFIGURE_$1) $$(MESOS_DEPS)
	mkdir -p $$(@D) && \
		cd $$(@D) && \
		env CXXFLAGS="$$(CXXFLAGS)" \
				CPPFLAGS="$$(CXXFLAGS)" \
				PYTHONPATH="$$(PYTHONPATH):$$$$PYTHONPATH" \
			$$< \
			--prefix=$$(MESOS_OPT_DIR_$1) \
	 		--disable-java \
			--disable-optimize \
			--with-svn=$$(SVN_OPT_DIR) \
			--with-boost=$$(BOOST_OPT_DIR) \
			--with-protobuf=$$(PBUF_OPT_DIR) \
			--with-picojson=$$(PICOJSON_OPT_DIR) \
			--with-glog=$$(GLOG_OPT_DIR)

mesos-$1-make: $$(MESOS_SRC_BIN_$1)
$$(MESOS_SRC_BIN_$1): MAKEFLAGS=$$(MESOS_MAKEFLAGS)
$$(MESOS_SRC_BIN_$1): $$(MESOS_MAKEFILE_$1)
	cd $$(<D) && $$(MAKE)

mesos-$1: $$(MESOS_$1)
ifeq ($$(call USE_U1204_CACHED_DEP,$$(MESOS_OPT_7Z_$1)),true)
$$(MESOS_$1): $$(MESOS_DEPS)
	cd $$(DEPS_DIR) && \
		7z x $$(DEPS_7ZS_DIR)/$$(MESOS_OPT_7Z_$1) > /dev/null
else
ifeq ($$(call USE_U1204_DEP,$$(MESOS_OPT_7Z_$1)),true)
$$(MESOS_$1): $$(MESOS_DEPS)
	mkdir -p $$(DEPS_7ZS_DIR) && \
		cd $$(DEPS_7ZS_DIR) && \
		curl -SLO $$(DEPS_URL)/$$(MESOS_OPT_7Z_$1) && \
		cd $$(DEPS_DIR) && \
		7z x $$(DEPS_7ZS_DIR)/$$(MESOS_OPT_7Z_$1) > /dev/null
else
$$(MESOS_$1): MAKEFLAGS=$$(MESOS_MAKEFLAGS)
$$(MESOS_$1): $$(MESOS_SRC_BIN_$1)
	cd $$(<D) && $$(MAKE) install
endif
endif

mesos-$1-clean-src:
	rm -fr $$(MESOS_SRC_DIR_$1)

mesos-$1-clean-build:
	rm -fr $$(MESOS_BUILD_DIR_$1)

mesos-$1-clean-opt:
	rm -fr $$(MESOS_OPT_DIR_$1)

mesos-$1-clean: mesos-$1-clean-src mesos-$1-clean-opt
endef

$(foreach V,$(MESOS_VERSIONS),$(eval $(call MESOS_BUILD_RULES,$(V))))

mesos-src: $(addsuffix -src,$(addprefix mesos-,$(MESOS_VERSIONS)))
mesos-configure: $(addsuffix -configure,$(addprefix mesos-,$(MESOS_VERSIONS)))
mesos-make: $(addsuffix -make,$(addprefix mesos-,$(MESOS_VERSIONS)))
mesos-clean: $(addsuffix -clean,$(addprefix mesos-,$(MESOS_VERSIONS)))
mesos: $(MESOS)

The Isolator Module

Much like Mesos itself, the Makefile makes it quite easy to build multiple versions of the Isolator module against multiple versions of Mesos:

# ISO_VERSIONS is either equal to or a subset of the MESOS_VERSIONS
# list. The versions in this list are the versions of Mesos against 
# which to build the isolator module.
ISO_VERSIONS := 0.23.0 0.23.1

The variable ISO_VERSIONS is used by make to dynamically create targets at runtime for each of the specified versions. Using the above value as an example, the Makefile can be used to build two versions of the Isolator module against the specified Mesos versions:

  • make isolator-0.23.0
  • make isolator-0.23.1

The command make isolator will build all versions of the Isolator module.

The resulting artifacts for version 1.1.1-dev of the module will look similar to the following:

  • libmesos_dvdi_isolator-0.1.1-dev+7+mesos-0.23.0.so
  • libmesos_dvdi_isolator-0.1.1-dev+7+mesos-0.23.1.so

The version is semantic and is calculated using git describe with logic inside the Makefile. The current version can be emitted with make print-version:

[0]akutz@pax:mesos-module-dvdi$ make print-version
SemVer: 0.1.1-dev+7
Branch: master
Commit: 179b7a8bb9247209746b370f0839b6ef63fb6712
Formed: Tue, 13 Oct 2015 16:25:16 CDT

Travis-CI Dependency Caching

One last thing before I go. If you take a look at the DVDI project’s .travis.yml file you’ll notice the following section:

cache:
  directories:
    - /tmp/mesos-ubuntu-12.04-deps/.7zs

This instructs Travis to cache the contents of the directory /tmp/mesos-ubuntu-12.04-deps/.7zs. This directory is where the pre-compiled 7z archives are downloaded prior to being inflated. If this directory is cached then Travis will take care of pushing it to an S3 storage location and reconstituting prior to each build. I highly encourage everyone to cache this directory as it will alleviate pressure off of Bintray if you’re using the 7z archives I’m providing. This way the first time those files are fetched Travis will take over the caching and reassembling of their contents.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s