Continuous integration with Unity

Tech

• Jenkins Plugin on Github

References:

• Via Jenkins, Continuous integration @ Coding With Style In Unity3D (June 2015)
• Via Travis, Automatically Build...  on Jonathan Porta's Blog (Apr 2015)
• Custom Continuous Integration @ MDD(Jan 2015)
• Via Team City, Setting up Continuous Integration @Random Bits (March 2012)
• Via Jenkins, Build Automation with Unity @ Grand Cru game (August 2012)
• Topic on UnityAnswers

A C# geometry library

Recently I started work on a C# library with a neat Unity 3D integration allowing to interactively build and visualise geometric primitives (the core library doesn't require Unity).

I've been prioritising features which are less commonly available, and will help developers create innovative content.

• Intersections
• Containment tests
• Mesh slicing
• Ngons
  I much prefer the classic wording 'Polygons' :)

Other stuff I'm interested in implementing (or migrating to this library) includes:

• Implicit surfaces (e.g. using the marching cubes), 
• Efficient voxel maps using BSPs 
• CSG

Since I started working on my first 3D tool some years ago (~1995!), geometry has often been a hindrance - often proving to be too much of a time-sink. I hope that sharing this work will benefit other developers.

You can find my draft library and demo project on Bitbucket.

Stop recycling game objects

Many consider object pools a staple of game development. Here's a few examples:

• A whole chapter in Game Programming Patterns.
• An article on Gamasutra
• A tutorial on the Unity Website.

Although Wikipedia does hint at potential pitfalls, there is a much greater problem than having (or forgetting) to reset pooled objects: how do you ensure that references to your objects are cleared before pooling them?

Your object may have registered to receive events.

Or it may be added to a list somewhere.

Or it may be assigned to a field in about a couple dozen places

...

Yes, there are possibly hundreds of objects in your runtime which may hold references to an object you'd like to recycle.

In languages like C++, it is your responsibility to clear these references. Not clearing references before destroying an object leads to wild pointers. Wild pointers lead to intermittent crashes and bugs.

In C#, Unwanted references to pooled objects fail subtly: your application won't crash but a (usually unknown) number of objects will continue to talk to your object without knowing that it's been recycled.

Pooling is an optimisation technique. If you have to use it, know that just resetting these objects is the easy part.

Visualising C++ #1 - A quick look at Visual Paradigm

I've been wanting to visualise one of the C++ projects I'm working on for a while. First I tried Visual Paradigm. 

Surprisingly, VP choked on irrelevant files ( git meta, makefiles, ... ). At the time of writing (Dec. 2015) their help-desk confirmed practical limitations in this area.

Anyways. I somehow got rid of the gits then processed caffe through the so-called 'instant reverse' feature. Selected all output and generated a diagram. Output didn't look great. Here is a second attempt displaying only the 'caffe' namespace:



Not excluding the possibility that this can be configured to produce decent output.

Water in Unity

While fantastic at a glance (well, almost...) water in Unity can be tricky to setup. Water and Pro Water aren't too difficult to figure. Water 4 may not always reward your patience.

Unity ships with 3 water assets

• Simple water does not provide reflection or reflection
• ProWater supports reflective and refractive water (foam can be added)
• Water 4 supports reflective and refractive water with foam and mesh animation.

Compared with Pro Water, Water 4 looks "later and better". 

Foam with Pro Water

• There's a modified script here. Adds an interesting foam effect; works somewhat differently from foam in Water 4 (tested).
• Modified water here (inc. underwater)
• Shoreline generator

More about Water 4

Where does the color come from?

Base color, mostly. However, if you zero Fresnel power, colour is mostly the reflection colour.

What controls the foam?

Tip: Replace the foam texture with something obvious to get a better idea of what's going on.

Under 'Auto-blend', edge controls blending of water with land. High values (1 or more) make foam more visible. To make the foam ring thinner, use high values (1 or more) of shore whereas foam will be very spread out if your values are something like 0.5 or less.

Bump tiling decides how big the foam texture is tiling. Looks like they are tiling two layers of foam too. 

Foam intensity can help to make foam more visible. Above 1, tends to whiten everything.

An annoyance is that 'foam' is also used everywhere on the water surface. feels like a kind of rippling reflection. Disabling Gerstner displace can help.

Caffe on Win64: the Appveyor setup

In an ideal world, this is how we would build caffe with Appveyor:

md build

cd build

cmake ..
cmake --build . --config Debug
cmake --build . --config Release

Upon running this we find that a few things need fixing.

CMP0054, 0046 policy warnings

This warning is caused by a newer version of cmake. And this is how we fix it (at the top of CMakeLists.txt)

if(POLICY CMP0054)
  cmake_policy(SET CMP0054 OLD)
endif()

Boost libraries

Appveyor includes the following:

 C:\Libraries\boost\stage 
= > sample lib: libboost_date_time-vc120-mt-1_56.lib

C:\Libraries\boost_1_58_0\lib64-msvc-12.0 (or 1_59)
 => sample lib boost_date_time-vc120-mt-1_58.lib or dll

I target the local build (boost/stage) with the following cache entries:

BOOST_ROOT=C:/Libraries/boost
Boost_USE_STATIC_LIBS=ON
Boost_Compiler=-vc120

USE_STATIC_LIBS reflects the 'lib' prefix; The compiler setting is needed because I do not target a specific compiler for building caffe, causing CMake to default to 140 (latest installed on Appveyor) whereas boost libs are built with 120.
  
Protobuf

We download and build from source using the following script:

git clone -q https://www.github.com/google/protobuf protobuf
cd protobuf/cmake
md build
cd build
cmake .. -DBUILD_TESTING=OFF
cmake --build . --config Debug
cmake --build . --config Release
cd ..\..\..

The following cache entries are used in the Caffe build:

PROTOBUF_LIBRARY="../protobuf/build-output/Release/libprotobuf.lib" PROTOBUF_INCLUDE_DIR="../protobuf/src" PROTOBUF_PROTOC_EXECUTABLE="../protobuf/cmake/build/Release/protoc.exe"

Alternative: using releases? (in progress)

Currently there are several release and pre-release versions of protobuf (here). The latest release (2.6.1) does not appear to include win64 binaries.

HDF5 (1.30pm)

An *.msi installer is available. We download and install using the following script:

#download HDF5
$file = join-path (pwd) \hdf5.zip
$url = "http://www.hdfgroup.org/ftp/HDF5/current/bin/windows/hdf5-1.8.15-patch1-win64-vs2013-shared.zip"
(new-object net.webclient).downloadFile($url,$file)
#unzip archive
Add-Type -AssemblyName System.IO.Compression.FileSystem
$out = Join-Path (pwd) \hdf5_unzipped
[System.IO.Compression.ZipFile]::ExtractToDirectory($file, $out)
#install HDF5
cd hdf5_unzipped/hdf5
Start-Process "HDF5-1.8.15-win64.msi" /qn -Wait
cd ../..

Configuration

The HDF group provides a little information. FindHDF5 will leverage an environment variable (not a cache entry) set as follow:

$env:HDF5_ROOT = "C:\Program Files\HDF_Group\HDF5\1.8.15"

LMDB, LevelDB, OpenCV and CUDA

Since we don't use any of the above, turn them off.

ATLAS

We use OpenBLAS. Appveyor  build works already have MinGW (OpenBLAS dependency) so we need not install it.

Download and install

#download OpenBLAS binaries
$file = join-path (pwd) \OpenBlas.zip
$url = "http://sourceforge.net/projects/openblas/files/v0.2.14/OpenBLAS-v0.2.14-Win64-int64.zip/download"
(new-object net.webclient).downloadFile($url,$file)
#unzip OpenBLAS binaries
Add-Type -AssemblyName System.IO.Compression.FileSystem
$out = Join-Path (pwd) \unzipped
[System.IO.Compression.ZipFile]::ExtractToDirectory($file, $out)

Cache entries

BLAS="open"
OpenBLAS_INCLUDE_DIR="../unzipped/OpenBLAS-v0.2.14-Win64-int64/include" 
OpenBLAS_LIB="../unzipped/OpenBLAS-v0.2.14-Win64-int64/lib/libopenblas.dll.a"

Python

If we don't care about Python the following cache entries are used to speed things up and doesn't require numpy:

BUILD_python=OFF
BUILD_python_layer=OFF 

"Cuda is disabled" warning

When CPU_ONLY is enabled, we should not received a warning or error. Issue a STATUS message instead.

GLog and Gflags

GFlags and GLog appear to automatically download and build as part of a caffe build. However, this causes errors.

With GFlags (v2.1.2) we see an incompatibility with recent versions of VS (fixed on master).

error C2084: function 'int snprintf(char *const ,const size_t,const char *const ,...)' already has a body [C:\projects\caffe-cx7ht\build\ext

ernal\gflags-prefix\src\gflags-build\gflags-static.vcxproj] [C:\projects\caffe-cx7ht\build\gflags.vcxproj]

C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1932): note: see previous definition of 'snprintf'

Looking into gflags.cmake we see that we are downloading the latest tagged version (2.1.2).
One way to solve the above error is to downgrade the generator (-G) to Visual Studio 12 2013.

Like GFlags, Glog (v0.3.4) downloads and builds automatically. Building on Windows,  error:

Performing configure step for 'glog'

CMake Error at C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog-stamp/glog-configure-Debug.cmake:16 (message):

Command failed: 77

'env' 'CFLAGS= /DWIN32 /D_WINDOWS /W3' 'CXXFLAGS= /DWIN32 /D_WINDOWS /W3 /GR /EHsc' 'C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog/configure' '--prefix=C:/projects/caffe-cx7ht/build/external/glog-install' '--en

able-shared=no' '--enable-static=yes' '--with-gflags=C:/projects/caffe-cx7ht/build/external/gflags-install/lib/..'

This with a redirection to C:/projects/caffe-cx7ht/build/external/glog-prefix/src/glog-stamp/glog-configure-*.log where we find that CMake... didn't find the C compiler?

Looking into this, we find that caffe includes two CMake files used to download and install GLog and GFlags. See glog.cmake.

The DIY route

In our current setup we build Glog ourselves.  Because we do not build GFlags, on which there is a dependency, this causes a linkage error:

LINK : fatal error LNK1181: cannot open input file 'C:\projects\caffe-cx7ht\gflags\build\lib\Release\gflags.lib'

Even so, we have satisfied enough dependencies that CMake agrees to attempt a caffe build.

Appveyor notes

After a build we can ask Appveyor to wait so we can use remote desktop to diagnose issues:

$blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

YML setup

Appveyor gives the option to drive build works either using a script or via a GUI. We prefer to save the YML. Powershell scripts are defined separately because embedding PS within YML is somewhat inconvenient.

Caching

Caching build results is used to help speeding up builds. Currently we cache Protobuf and GLog. See here.

FileZilla installs nagware on your system.

Don't use FileZilla.