Caffe on Win64: the Appveyor setup

In an ideal world, this is how we would build caffe with Appveyor:

md build

cd build

cmake ..
cmake --build . --config Debug
cmake --build . --config Release

Upon running this we find that a few things need fixing.

CMP0054, 0046 policy warnings

This warning is caused by a newer version of cmake. And this is how we fix it (at the top of CMakeLists.txt)

if(POLICY CMP0054)
  cmake_policy(SET CMP0054 OLD)
endif()

Boost libraries

Appveyor includes the following:

 C:\Libraries\boost\stage 
= > sample lib: libboost_date_time-vc120-mt-1_56.lib

C:\Libraries\boost_1_58_0\lib64-msvc-12.0 (or 1_59)
 => sample lib boost_date_time-vc120-mt-1_58.lib or dll

I target the local build (boost/stage) with the following cache entries:

BOOST_ROOT=C:/Libraries/boost
Boost_USE_STATIC_LIBS=ON
Boost_Compiler=-vc120

USE_STATIC_LIBS reflects the 'lib' prefix; The compiler setting is needed because I do not target a specific compiler for building caffe, causing CMake to default to 140 (latest installed on Appveyor) whereas boost libs are built with 120.
  
Protobuf

We download and build from source using the following script:

git clone -q https://www.github.com/google/protobuf protobuf
cd protobuf/cmake
md build
cd build
cmake .. -DBUILD_TESTING=OFF
cmake --build . --config Debug
cmake --build . --config Release
cd ..\..\..

The following cache entries are used in the Caffe build:

PROTOBUF_LIBRARY="../protobuf/build-output/Release/libprotobuf.lib" PROTOBUF_INCLUDE_DIR="../protobuf/src" PROTOBUF_PROTOC_EXECUTABLE="../protobuf/cmake/build/Release/protoc.exe"

Alternative: using releases? (in progress)

Currently there are several release and pre-release versions of protobuf (here). The latest release (2.6.1) does not appear to include win64 binaries.

HDF5 (1.30pm)

An *.msi installer is available. We download and install using the following script:

#download HDF5
$file = join-path (pwd) \hdf5.zip
$url = "http://www.hdfgroup.org/ftp/HDF5/current/bin/windows/hdf5-1.8.15-patch1-win64-vs2013-shared.zip"
(new-object net.webclient).downloadFile($url,$file)
#unzip archive
Add-Type -AssemblyName System.IO.Compression.FileSystem
$out = Join-Path (pwd) \hdf5_unzipped
[System.IO.Compression.ZipFile]::ExtractToDirectory($file, $out)
#install HDF5
cd hdf5_unzipped/hdf5
Start-Process "HDF5-1.8.15-win64.msi" /qn -Wait
cd ../..

Configuration

The HDF group provides a little information. FindHDF5 will leverage an environment variable (not a cache entry) set as follow:

$env:HDF5_ROOT = "C:\Program Files\HDF_Group\HDF5\1.8.15"

LMDB, LevelDB, OpenCV and CUDA

Since we don't use any of the above, turn them off.

ATLAS

We use OpenBLAS. Appveyor  build works already have MinGW (OpenBLAS dependency) so we need not install it.

Download and install

#download OpenBLAS binaries
$file = join-path (pwd) \OpenBlas.zip
$url = "http://sourceforge.net/projects/openblas/files/v0.2.14/OpenBLAS-v0.2.14-Win64-int64.zip/download"
(new-object net.webclient).downloadFile($url,$file)
#unzip OpenBLAS binaries
Add-Type -AssemblyName System.IO.Compression.FileSystem
$out = Join-Path (pwd) \unzipped
[System.IO.Compression.ZipFile]::ExtractToDirectory($file, $out)

Cache entries

BLAS="open"
OpenBLAS_INCLUDE_DIR="../unzipped/OpenBLAS-v0.2.14-Win64-int64/include" 
OpenBLAS_LIB="../unzipped/OpenBLAS-v0.2.14-Win64-int64/lib/libopenblas.dll.a"

Python

If we don't care about Python the following cache entries are used to speed things up and doesn't require numpy:

BUILD_python=OFF
BUILD_python_layer=OFF 

"Cuda is disabled" warning

When CPU_ONLY is enabled, we should not received a warning or error. Issue a STATUS message instead.

GLog and Gflags

GFlags and GLog appear to automatically download and build as part of a caffe build. However, this causes errors.

With GFlags (v2.1.2) we see an incompatibility with recent versions of VS (fixed on master).

error C2084: function 'int snprintf(char *const ,const size_t,const char *const ,...)' already has a body [C:\projects\caffe-cx7ht\build\ext

ernal\gflags-prefix\src\gflags-build\gflags-static.vcxproj] [C:\projects\caffe-cx7ht\build\gflags.vcxproj]

C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1932): note: see previous definition of 'snprintf'

Looking into gflags.cmake we see that we are downloading the latest tagged version (2.1.2).
One way to solve the above error is to downgrade the generator (-G) to Visual Studio 12 2013.

Like GFlags, Glog (v0.3.4) downloads and builds automatically. Building on Windows,  error:

Performing configure step for 'glog'

CMake Error at C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog-stamp/glog-configure-Debug.cmake:16 (message):

Command failed: 77

'env' 'CFLAGS= /DWIN32 /D_WINDOWS /W3' 'CXXFLAGS= /DWIN32 /D_WINDOWS /W3 /GR /EHsc' 'C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog/configure' '--prefix=C:/projects/caffe-cx7ht/build/external/glog-install' '--en

able-shared=no' '--enable-static=yes' '--with-gflags=C:/projects/caffe-cx7ht/build/external/gflags-install/lib/..'

This with a redirection to C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog-stamp/glog-configure-*.log where we find that CMake... didn't find the C compiler?

Looking into this, we find that caffe includes two CMake files used to download and install GLog and GFlags. See glog.cmake.

The DIY route

In our current setup we build Glog ourselves.  Because we do not build GFlags, on which there is a dependency, this causes a linkage error:

LINK : fatal error LNK1181: cannot open input file 'C:\projects\caffe-cx7ht\gflags\build\lib\Release\gflags.lib'

Even so, we have satisfied enough dependencies that CMake agrees to attempt a caffe build.

Appveyor notes

After a build we can ask Appveyor to wait so we can use remote desktop to diagnose issues:

$blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

YML setup

Appveyor gives the option to drive build works either using a script or via a GUI. We prefer to save the YML. Powershell scripts are defined separately because embedding PS within YML is somewhat inconvenient.

Caching

Caching build results is used to help speeding up builds. Currently we cache Protobuf and GLog. See here.