Jonathan's Blog

Distributed Hyperparameter Optimisation with Keras/Theano/Tensorflow/Hyperas/Hyperopt

2018-08-29T00:00:00+09:30

Overview

In this post I will descibe how to use a combination of tools to speed up hyperparameter optimisation tasks. Instructions are provided here for Ubuntu, but could reasonably be applied to any *nix system. My motivation was the long queue wait times to use my university’s supercomputer resources, so I hijacked the school’s computer lab and ran a large number of experiments in parallel using the process described here.

It can also be ran on Sun Grid Engine (SGE) with extra wrapper scripts, although I found my desktop + SGE head node ran all the models before the SGE jobs could run.

What is Hyperparameter Optimisation?

Hyperparameter Optimisation (HO) is a method choosing the optimal paramters for a machine learning task. These parameters include things like:

Number of layers
Learning rate
Batch size
Layer types
Dropout
Optimiser algorithm (SGD, Adam, rmsprop etc.)

It may not be immediately obvious for any given problem what network configuration is best for a given task, so we can use hyperparameter optimisation decide for us by intelligently iterating through a search space of the parameters you want to optimise. Hyperopt uses Tree-Structure Parzen estimators and is quite good at rapidly deciding on an optimal set of parameters. It works by running and evaluating a model, returning a loss score and then running another model with slightly different parameters that aim to minimise the error score. The hard part for you, is devising a search space for your problem, which may be quite large. To save time, we can run these models simultaneously across any number of machines, and even have each machine run multiple models (provided it has enough cores).

Luckily there are python libraries that do all this hard work for us!

Requirements

You will need:

Python with the following packages installed
- theano, tensorflow or tensorflow-gpu
- hyperopt
- hyperas
- pymongo
- pssh
A bunch of machines with all of the above installed
A single Mongodb instance with a jobs database

I strongly recommend using pyenv to use an up to date version of python and to prevent our installed packages from conflicting with system ones. If you have access to a network drive available to all your machines, set $PYENV_ROOT to something they can all see (or at least a common path on all the machines). Install these using (swap tensorflow with whatever keras backend you want to use, one of: theano tensorflow-gpu tensorflow cntk):

export PYENV_ROOT="$HOME/.pyenv"
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
source ~/.bash_profile
sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev
env PYTHON_CONFIGURE_OPTS="--enable-shared" MAKEOPTS="-j 8" pyenv install 3.6.5
pyenv local 3.6.5
pip install tensorflow git+https://github.com/hyperopt/hyperopt git+https://github.com/maxpumperla/hyperas keras pssh matplotlib h5py pymongo

You will now have a self contained python install in $HOME/.pyenv/versions/3.6.5. Keep in mind that it needs to build a python install and so might take a while. You can copy the ~/.pyenv folder to any machine you want to run on. Just remember to copy your ~/.bash_profile (or equivalent) to each machine you want as a worker.

Code

We also need pick a task to optimise! Hyperas uses templates to generate the code that hyperopt can use, so you need to follow this template closely. Create a file called: optimise_task.py as described blow. We’ll find the optimal layer size and dropout parameters (from within the search space, see l1_size and l1_dropout variables) for a single layer dense network to solve the MNIST task. The documentation for the different parameter distributions is here: https://github.com/hyperopt/hyperopt/wiki/FMin#21-parameter-expressions

For our purposes:

quniform is a normal distribution of discrete values with a given interval and step. In ours, it will return floats in the range [12,256] with step=4
uniform is a normal distribution of continuous values

Many more distributions are available in hyperas.distributions: https://github.com/maxpumperla/hyperas/blob/master/hyperas/distributions.py

from hyperas import optim
from hyperas.distributions import quniform, uniform
from hyperopt import STATUS_OK, tpe, mongoexp
import keras
from keras.layers import Dense, Dropout
from keras.models import Sequential
from keras.optimizers import RMSprop
from keras.datasets import mnist
import tempfile
from datetime import datetime


def data():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(60000, 784)
    x_test = x_test.reshape(10000, 784)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    x_train /= 255
    x_test /= 255
    num_classes = 10
    y_train = keras.utils.to_categorical(y_train, num_classes)
    y_test = keras.utils.to_categorical(y_test, num_classes)
    return x_train, y_train, x_test, y_test


def create_model(x_train, y_train, x_test, y_test):
    """
    Create your model...
    """
    
    l1_size = {{quniform(12, 256, 4)}}
    l1_dropout = {{uniform(0.001, 0.7)}}
    
    params = {
        'l1_size': l1_size,
        'l1_dropout': l1_dropout
    }
    num_classes = 10
    model = Sequential()
    model.add(Dense(int(l1_size), activation='relu'))
    model.add(Dropout(l1_dropout))
    model.add(Dense(num_classes, activation='softmax'))
    model.compile(loss='categorical_crossentropy',
                  optimizer=RMSprop(),
                  metrics=['accuracy'])
    start = datetime.now()
    model.fit(x_train, y_train, batch_size=128, epochs=10, validation_data=(x_test, y_test))
    score, acc = model.evaluate(x_test, y_test, verbose=0)
    out = {
        'loss': -acc,
        'score': score,
        'status': STATUS_OK,
        'duration': (datetime.now() - start).total_seconds(),
        'ho_params': params,
        'model_config': model.get_config()
    }
    # optionally store a dump of your model here so you can get it from the database later
    temp_name = tempfile.gettempdir()+'/'+next(tempfile._get_candidate_names()) + '.h5'
    model.save(temp_name)
    with open(temp_name, 'rb') as infile:
        model_bytes = infile.read()
    out['model_serial'] = model_bytes
    return out


if __name__ == "__main__":
    trials = mongoexp.MongoTrials('mongo://username:pass@mongodb.host:27017/jobs/jobs', exp_key='mnist_test')
    best_run, best_model = optim.minimize(model=create_model,
                                          data=data,
                                          algo=tpe.suggest,
                                          max_evals=10,
                                          trials=trials,
                                          keep_temp=True)  # this last bit is important
    print("Best performing model chosen hyper-parameters:")
    print(best_run)

Note the name of the experiment key: mnist_test, this will be the key in the jobs collection of the jobs database in mongodb. After every model has completed, it will be stored in mongodb. It may be possible to store the weights in the output document (the output of model.get_weights(), but mongodb has a limit of 4MB per document. To get around this, GridFS is used to transprently store blobs in the database of the model itself.

I’ve also stored the duration in the result object as well, since you may find 2 models with very similar loss, but the one with slightly better loss may have significantly higher runtime.

Running

Running this has 2 parts:

Trials controller that decides on the parameters that each model run will use
Workers that actually run the individual models

Controller

Run this from a machine (it must be active while all the jobs are running):

python optimise_task.py

You should get an output file called temp_model.py (if you don’t, make sure you have updated to the latest hyperas code from github). Make sure this file is visible to your workers.

Worker

These are all your other machines (may also be on the controller machine). Make sure you have pyenv installed on them too, it can be easy to just zip up the .pyenv folder and copy it your home directory on your worker machines and unzip it. This way you don’t miss any dependencies.

Run the following:

mkdir hyperopt_job
touch hyperopt_job/job.sh
chmod +x hyperopt_job/job.sh

Copy the temp_model.py file into the hyperopt_job folder ~/hyperopt_job/job.sh with:

#!/bin/bash
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
export PYTHONPATH=~/hyperopt_job/
cd ~/hyperopt_job
pyenv local 3.6.5
hyperopt-mongo-worker --mongo="mongo://username:password@mongodb.host:27017/jobs" --exp-key=mnist_test

Now you can run ~/hyperopt_job/job.sh on your worker machines! Keep in mind that they need to be able to access the mongodb. The workers will continue running jobs until you have reached max_evals as defined at the end of optimise_task.py.

You can also have this script fetch a compressed version of your .pyenv folder from a URL if it doesn’t already exist, by prepending the following lines:

if [ ! -d "$HOME/.pyenv" ]; then
    wget https://url.to/mypenv.zip
    unzip mypyenv.zip
fi

If your home drive on the worker machines has limited space, consider unzipping to a tmp directory and setting the PYENV_ROOT environment variable accordingly.

The problem with this can be that you need to:

Login to the machine via ssh or regular login
Start screen
Start the script

We can do better by using pssh to do the above automatically for a list of given hosts.

pssh -h hosts.txt  bash -c "nohup  ~/hyperopt_job/job.sh &"

This will start a start job.sh on each host listed in hosts.txt. nohup will prevent the process from stopping when the ssh session disconnects.

Results

Once all your jobs are done, you can look through your results using a mongodb browser such as Robo3T. Here’s a small script to fetch the model with lowest loss score from the database and deserialize your model:

from pymongo import MongoClient, ASCENDING
from keras.models import load_model
import tempfile
c = MongoClient('mongodb://username:pass@mongodb.host:27017/jobs')
best_model = c['jobs']['jobs'].find_one({'exp_key': 'mnist_test', 'result.status': 'ok'}, sort=[('result.loss', ASCENDING)])
temp_name = tempfile.gettempdir()+'/'+next(tempfile._get_candidate_names()) + '.h5'
with open(temp_name, 'wb') as outfile:
    outfile.write(best_model['result']['model_serial'])
model = load_model(temp_name)
# do things with your model here
model.summary()

You can use the following to visualise the results with a subset of your search space (typical search spaces with have more than 2 parameters so it may not be obvious which one (or combination thereof) has the most significant impact on your model performance):

from collections import defaultdict
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from pymongo import MongoClient
import numpy as np

if __name__ == "__main__":
    # get the data
    jobs = MongoClient('mongodb://username:pass@mongodb.host:27017/jobs')['jobs']['jobs']
    cursor = jobs.find({'exp_key': 'mnist_test', 'result.status': 'ok'})
    results = defaultdict(lambda: defaultdict(list))
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    for i in ax.get_xticklabels():
        i.set_rotation(90)

    for row in cursor:
        cc = row['tid']
        results[cc]['x'].append(row['result']['loss'])
        results[cc]['y'].append(row['result']['ho_params']['l1_size'])
        results[cc]['z'].append(row['result']['ho_params']['l1_dropout'])

    colors = cm.rainbow(np.linspace(0, 1, len(results)))
    it = iter(colors)
    for k, v in results.items():
        ax.scatter(v['x'], v['y'], v['z'], label=k, color=next(it))
    ax.set_xlabel('RMSE')
    ax.set_ylabel('Layer Count')
    ax.set_zlabel('L1 Size')
    plt.title("Hyperparameter Optimisation Results")
    plt.show()

Here’s an output from one of my experiments, as you can see, the optimser quickly finds models that cluster around a minimum achievable error score given the search space:

Building Tensorflow with OpenCL support on Ubuntu 16.04

2018-04-04T00:00:00+09:30

Overview

In this post I will describe the steps required to get Tensorflow running with OpenCL on an AMD GPU.

Install AMDGPU-PRO Drivers

Install with (applying sudo credentials where necessary):

wget -referer=https://support.amd.com https://www2.ati.com/drivers/linux/ubuntu/amdgpu-pro-17.50-511655.tar.xz
tar -xvf amdgpu-pro-17.50-511655.tar.xz
cd amdgpu-pro-17.50-511655
./amdgpupro-install --opencl=legacy --headless
sudo reboot

Install a recent python

I use pyenv to manage different versions of python, setup your python as follows:

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

Add the following lines to your ~/.bashrc file:

export PATH="~/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Restart your terminal and you have pyenv installed. Now install a python (I’ll use the latest 3.6) and its system dependencies:

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev
env PYTHON_CONFIGURE_OPTS="--enable-shared" MAKEOPTS="-j 8" pyenv install 3.6.5
pyenv local 3.6.5
pip install wheel numpy

Get The ComputeCPP SYCL Implementation

Tensorflow can use the SYCL interface to seamlessly run device agnostic c++ code on an OpenCL enabled device. There is an open source template based library called triSYCL. It requires c++17 support and thus you probably need to build your own gcc from a recent release. I won’t cover it here but you can if you want to.

https://developer.codeplay.com/computecppce/latest/overview

tar -xvf ComputeCpp-CE-0.6.1-Ubuntu.16.04-64bit.tar.gz
sudo mv ComputeCpp-CE-0.6.1-Ubuntu-16.04-64bit /usr/local/computecpp

Get Tensorflow with experimental opencl support

This fork of tensorflow is maintained by someone from Codeplay, who make ComputeCPP. It’s highly experimental, expect it to change in the future. We use the dev/amd_gpu branch which is currently under active development:

git clone https://github.com/lukeiwanski/tensorflow
cd tensorflow
git fetch
git checkout dev/amd_gpu
pyenv local 3.6.5

In order to build tensorflow, you need to use Google’s bazel build system:

wget https://github.com/bazelbuild/bazel/releases/download/0.11.1/bazel_0.11.1-linux-x86_64.deb
sudo dpkg -i bazel_0.11.1-linux-x86_64.deb

Environment and Building

You can now configure and build tensorflow, from the tensorflow directory run:

./configure

I used the following options (press enter to get the defaults):

$ ./configure 
You have bazel 0.11.1 installed.
Please specify the location of python. [Default is /home/user/.pyenv/versions/3.6.5/bin/python]: 

Please input the desired Python library path to use.  Default is [/home/user/.pyenv/versions/3.6.5/lib/python3.6/site-packages]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: Y
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Apache Kafka Platform support? [y/N]: n
No Apache Kafka Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: n
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: n
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: n
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: y
OpenCL SYCL support will be enabled for TensorFlow.

Please specify which C++ compiler should be used as the host C++ compiler. [Default is /usr/bin/g++]: 


Please specify which C compiler should be used as the hostC compiler. [Default is /usr/bin/gcc]: 


Do you wish to build TensorFlow with ComputeCPP support? [Y/n]: Y
ComputeCPP support will be enabled for TensorFlow.

Please specify the location where ComputeCpp for SYCL 1.2 is installed. [Default is /usr/local/computecpp]: 

Do you wish to build TensorFlow with double types in SYCL support? [Y/n]: Y
double types in SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with half types in SYCL support? [y/N]: n
No half types in SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: n
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:

Now you can build the package, consider going to get lunch because this will take a while:

bazel build --config=opt --config=sycl //tensorflow/tools/pip_package:build_pip_package

Hopefully it built without errors, if it did, try to fix them. Now install the generated wheel file:

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tf_cpp_whl
pip install /tmp/tf_cpp_whl/tensorflow-1.6.0rc0-cp27-cp27mu-linux_x86_64.whl

Now you’ve got tensorflow installed, but it’s not over yet, you need everything setup nicely to point to the appropriate OpenCL ICD and use everything in your updated amdgpu-pro drivers. Consider putting these environment variables in your ~/.bashrc file:

export $OPENCL_VENDOR_PATH=/opt/amdgpu-pro/etc/OpenCL/vendors/
mkdir -p $OPENCL_VENDOR_PATH
echo "/opt/amdgpu-pro/lib/x86_64-linux-gnu/libamdocl64.so" > $OPENCL_VENDOR_PATH/amdocl64.icd
export LD_LIBRARY_PATH=/usr/local/computecpp/lib:/opt/amdgpu-pro/lib/x86_64-linux-gnu
export PATH=/usr/local/computecpp/bin:/opt/amdgpu-pro/bin:$PATH

Here’s some explanation:

OPENCL_VENDOR_PATH is where libOpenCL.so will look for .icd files which point to a specific library for a driver. In our case it’s the libamdocl64.so, each one will appear as a different vendor, so you could install pocl or oclgrind
LD_LIBRARY_PATH is where the system looks for shared object files
PATH is where the system looks like for executable files. The two listed above contain clinfo and computecpp_info

Run clinfo and you should get something like this describing your GPU(s):

Number of platforms:				 1
  Platform Profile:				 FULL_PROFILE
  Platform Version:				 OpenCL 2.1 AMD-APP (2527.3)
  Platform Name:				 AMD Accelerated Parallel Processing
  Platform Vendor:				 Advanced Micro Devices, Inc.
  Platform Extensions:				 cl_khr_icd cl_amd_event_callback cl_amd_offline_devices 


  Platform Name:				 AMD Accelerated Parallel Processing
Number of devices:				 1
  Device Type:					 CL_DEVICE_TYPE_GPU
  Vendor ID:					 1002h
  Board name:					 AMD Radeon HD 8500 Series
  Device Topology:				 PCI[ B#1, D#0, F#0 ]
  Max compute units:				 6
  Max work items dimensions:			 3
    Max work items[0]:				 1024
    Max work items[1]:				 1024
    Max work items[2]:				 1024
  Max work group size:				 256
  Preferred vector width char:			 4
  Preferred vector width short:			 2
  Preferred vector width int:			 1
  Preferred vector width long:			 1
  Preferred vector width float:			 1
  Preferred vector width double:		 1
  Native vector width char:			 4
  Native vector width short:			 2
  Native vector width int:			 1
  Native vector width long:			 1
  Native vector width float:			 1
  Native vector width double:			 1
  Max clock frequency:				 780Mhz
  Address bits:					 64
  Max memory allocation:			 330696294
  Image support:				 Yes
  Max number of images read arguments:		 128
  Max number of images write arguments:		 8
  Max image 2D width:				 16384
  Max image 2D height:				 16384
  Max image 3D width:				 2048
  Max image 3D height:				 2048
  Max image 3D depth:				 2048
  Max samplers within kernel:			 16
  Max size of kernel argument:			 1024
  Alignment (bits) of base address:		 2048
  Minimum alignment (bytes) for any datatype:	 128
  Single precision floating point capability
    Denorms:					 No
    Quiet NaNs:					 Yes
    Round to nearest even:			 Yes
    Round to zero:				 Yes
    Round to +ve and infinity:			 Yes
    IEEE754-2008 fused multiply-add:		 Yes
  Cache type:					 Read/Write
  Cache line size:				 64
  Cache size:					 16384
  Global memory size:				 505212928
  Constant buffer size:				 65536
  Max number of constant args:			 8
  Local memory type:				 Scratchpad
  Local memory size:				 32768
  Max pipe arguments:				 0
  Max pipe active reservations:			 0
  Max pipe packet size:				 0
  Max global variable size:			 0
  Max global variable preferred total size:	 0
  Max read/write image args:			 0
  Max on device events:				 0
  Queue on device max size:			 0
  Max on device queues:				 0
  Queue on device preferred size:		 0
  SVM capabilities:
    Coarse grain buffer:			 No
    Fine grain buffer:				 No
    Fine grain system:				 No
    Atomics:					 No
  Preferred platform atomic alignment:		 0
  Preferred global atomic alignment:		 0
  Preferred local atomic alignment:		 0
  Kernel Preferred work group size multiple:	 64
  Error correction support:			 0
  Unified memory for Host and Device:		 0
  Profiling timer resolution:			 1
  Device endianess:				 Little
  Available:					 Yes
  Compiler available:				 Yes
  Execution capabilities:
    Execute OpenCL kernels:			 Yes
    Execute native function:			 No
  Queue on Host properties:
    Out-of-Order:				 No
    Profiling :					 Yes
  Queue on Device properties:
    Out-of-Order:				 No
    Profiling :					 No
  Platform ID:					 0x7f9728fbf510
  Name:						 Oland
  Vendor:					 Advanced Micro Devices, Inc.
  Device OpenCL C version:			 OpenCL C 1.2 
  Driver version:				 2527.3
  Profile:					 FULL_PROFILE
  Version:					 OpenCL 1.2 AMD-APP (2527.3)
  Extensions:					 cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_gl_event 

Running computecpp_info should get you something like this:

********************************************************************************

ComputeCpp Info (CE 0.6.1)

********************************************************************************

Toolchain information:

GLIBC version: 2.23
GLIBCXX: 20160609
This version of libstdc++ is supported.

********************************************************************************


Device Info:

Discovered 1 devices matching:
  platform    : <any>
  device type : <any>

--------------------------------------------------------------------------------
Device 0:

  Device is supported                     : UNTESTED - Vendor not tested on this OS
  CL_DEVICE_NAME                          : Oland
  CL_DEVICE_VENDOR                        : Advanced Micro Devices, Inc.
  CL_DRIVER_VERSION                       : 2527.3
  CL_DEVICE_TYPE                          : CL_DEVICE_TYPE_GPU 

If you encounter problems when using any of these OpenCL devices, please consult
this website for known issues:
https://computecpp.codeplay.com/releases/v0.6.1/platform-support-notes

********************************************************************************

Verification

You can now quickly check that everything installed correctly (don’t run this from the tensorflow source directory, you’ll run into issues):

cd ~
pyenv local 3.6.5
python -c "from tensorflow.python.client import device_lib;device_lib.list_local_devices()"
2018-04-04 16:39:13.673362: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:70] Found following OpenCL devices:
2018-04-04 16:39:13.673394: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 0, type: GPU, name: Oland, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE

Now you should be able to run some code that uses the AMD GPU! Here’s a small example using keras (which you need to install):

pip install keras

Here’s a toy regression example (it learns the sin function) that you can use to compare the speed of CPU vs. GPU:

#!/usr/bin/python
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, Activation
import numpy as np


x = np.arange(10000)
y = np.sin(x)
for i in  ('/cpu:0', '/gpu:0'):
    with tf.device(i):
        model = Sequential([
            Dense(128, input_shape=(1,), activation='relu'),
            Dense(1),
        ])

        model.compile(optimizer='adam', loss='mse')
        model.fit(x,y, validation_split=0.25, epochs=10)

Running this I get the following output:

Using TensorFlow backend.
Train on 7500 samples, validate on 2500 samples
Epoch 1/10
2018-04-04 16:53:55.766232: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:70] Found following OpenCL devices:
2018-04-04 16:53:55.766265: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 0, type: GPU, name: Oland, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
7500/7500 [==============================] - 0s 53us/step - loss: 29006.8066 - val_loss: 0.5027
Epoch 2/10
7500/7500 [==============================] - 0s 25us/step - loss: 0.5071 - val_loss: 0.5019
Epoch 3/10
7500/7500 [==============================] - 0s 26us/step - loss: 0.5091 - val_loss: 0.5005
Epoch 4/10
7500/7500 [==============================] - 0s 25us/step - loss: 0.5105 - val_loss: 0.5007
Epoch 5/10
7500/7500 [==============================] - 0s 30us/step - loss: 0.5166 - val_loss: 0.5006
Epoch 6/10
7500/7500 [==============================] - 0s 27us/step - loss: 0.5141 - val_loss: 0.6930
Epoch 7/10
7500/7500 [==============================] - 0s 26us/step - loss: 0.5173 - val_loss: 0.5968
Epoch 8/10
7500/7500 [==============================] - 0s 27us/step - loss: 0.5162 - val_loss: 0.5382
Epoch 9/10
7500/7500 [==============================] - 0s 26us/step - loss: 0.5195 - val_loss: 0.5377
Epoch 10/10
7500/7500 [==============================] - 0s 25us/step - loss: 0.5160 - val_loss: 0.5020

Train on 7500 samples, validate on 2500 samples
Epoch 1/10
7500/7500 [==============================] - 92s 12ms/step - loss: nan - val_loss: nan
Epoch 2/10
7500/7500 [==============================] - 2s 265us/step - loss: nan - val_loss: nan
Epoch 3/10
7500/7500 [==============================] - 2s 262us/step - loss: nan - val_loss: nan
Epoch 4/10
7500/7500 [==============================] - 2s 268us/step - loss: nan - val_loss: nan
Epoch 5/10
7500/7500 [==============================] - 2s 262us/step - loss: nan - val_loss: nan
Epoch 6/10
7500/7500 [==============================] - 2s 268us/step - loss: nan - val_loss: nan
Epoch 7/10
7500/7500 [==============================] - 2s 268us/step - loss: nan - val_loss: nan
Epoch 8/10
7500/7500 [==============================] - 2s 259us/step - loss: nan - val_loss: nan
Epoch 9/10
7500/7500 [==============================] - 2s 258us/step - loss: nan - val_loss: nan
Epoch 10/10
7500/7500 [==============================] - 2s 264us/step - loss: nan - val_loss: nan

As you can see, in the GPU run, it takes a long time to run the first epoch of the GPU the model, and significantly longer to run each training step (also that the GPU learnt nothing). You can play with the code to see if you can get it to work but waiting ~90s for the model to start the first epoch is a pain for testing.

It might just be I have an old workstation card, or that there is a lot of overhead involved in using SYCL, I hope this helps someone else!

OpenCL External Tool for IntelliJ

2017-04-03T00:00:00+09:30

In this post I will describe how to setup an external tool in your IntelliJ IDE that will give you an easy way to compile your kernel source and provide hyperlinked error messages. The code for compile.py is here (make sure you do chmod +x compile.py):

#!/usr/bin/env python
from __future__ import print_function
import pyopencl as cl
import sys
import os
import re

if __name__ == "__main__":
    dev = cl.get_platforms()[0].get_devices()[0]
    ctx = cl.Context([dev])
    fname = sys.argv[1]
    with open(fname, 'r') as src_file:
        src = src_file.read()
        fl = fname.lower()
        if fl.endswith('.py'):
            # pick out the string from the file that looks like a cl kernel

            docstrings = list(re.finditer('"""', src))
            allstrings = [(src[i[0].regs[0][1] + 1:i[1].regs[0][0] - 1], i[0].regs[0][0]) for i in zip(docstrings[::2], docstrings[1::2])]
            strings, positions = zip(*[s for s in allstrings if "//CL-KERNEL" in s[0]])
            cl_src = "\n".join(strings)
        elif fl.endswith('.cl'):
            cl_src = src
        print("Compiling {} on {}".format(fname, dev))
        try:
            prog = cl.Program(ctx, cl_src).build(options=['-I', os.path.dirname(src_file.name)])
    
        except cl.RuntimeError as e:
            out = str(e)
            out = re.sub(r'"/tmp/(.*)\.cl"', src_file.name, out).replace('\n\n', '\n')
            line_offset = src[:positions[0]].count("\n") + 1
            out = re.sub(r', line (\d+):', lambda m:":"+str(int(m.group(1)) + line_offset), out)
            print(out)#, file=sys.stderr)
    exit(1) # highlighting only occurs on non-zero exit status

Save this script somewhere and make sure you have pyopencl and python installed. If you are using pyopencl, you will need to mark the triple quoted strings that contain the kernel source code with a //CL-KERNEL somewhere in the code. The code could easily be modified to extract the kernel source from other languages such as Java.

In your IntelliJ IDE, go File -> Settings -> Tools -> External Tools -> Click the create button and fill out the form like so:

Next, create a filter from the “Output Filters” button. Create a new filter that has the following regex

$FILE_PATH$:$LINE$

Now you can easily right click on a file, click External Tools -> Compile OpenCL

Linux linker error: _ZTVN10cxxabiv117class_type_infoE

2017-01-27T00:00:00+10:30

If you ever get an error that looks like _ZTVN10__cxxabiv117__class_type_infoE or similar, the fix I found that works is to add -lstdc++ to your linker invocation. In python this means you need to modify the setup.py invocation of Extension to have:

libraries=['stdc++']

I hope this helps someone else.

libfreenect Python Depth Image

2017-01-27T00:00:00+10:30

Viewing the Kinect Depth Reading Using Python

libfreenect is a useful tool for reading the output of a Kinect camera. It offers some sample programs that will display the output of the camera along with its depth readings. If you want to use the python bindings though, viewing the depth reading in the same way as the example program is not possible and requires extra coding. In this post I will supply code that does this in a performant manner. Follow the instructions in the repo to install the library.

Pure Python Implementation

This code is basically a python translation of the code here: https://github.com/OpenKinect/libfreenect/blob/master/examples/glview.c#L350-L402

The difference here is that we generate every depth value to a colour before we start. The code uses numpy and pygame:

import pygame
import numpy as np
import sys
from freenect import sync_get_depth as get_depth


def make_gamma():
    """
    Create a gamma table
    """
    num_pix = 2048 # there's 2048 different possible depth values
    npf = float(num_pix)
    _gamma = np.empty((num_pix, 3), dtype=np.uint16)
    for i in xrange(num_pix):
        v = i / npf
        v = pow(v, 3) * 6
        pval = int(v * 6 * 256)
        lb = pval & 0xff
        pval >>= 8
        if pval == 0:
            a = np.array([255, 255 - lb, 255 - lb], dtype=np.uint8)
        elif pval == 1:
            a = np.array([255, lb, 0], dtype=np.uint8)
        elif pval == 2:
            a = np.array([255 - lb, lb, 0], dtype=np.uint8)
        elif pval == 3:
            a = np.array([255 - lb, 255, 0], dtype=np.uint8)
        elif pval == 4:
            a = np.array([0, 255 - lb, 255], dtype=np.uint8)
        elif pval == 5:
            a = np.array([0, 0, 255 - lb], dtype=np.uint8)
        else:
            a = np.array([0, 0, 0], dtype=np.uint8)

        _gamma[i] = a
    return _gamma


gamma = make_gamma()


if __name__ == "__main__":
    fpsClock = pygame.time.Clock()
    FPS = 30 # kinect only outputs 30 fps
    disp_size = (640, 480)
    pygame.init()
    screen = pygame.display.set_mode(disp_size)
    font = pygame.font.Font('slkscr.ttf', 32) # provide your own font 
    while True:
        events = pygame.event.get()
        for e in events:
            if e.type == pygame.QUIT:
                sys.exit()
        fps_text = "FPS: {0:.2f}".format(fpsClock.get_fps())
        # draw the pixels

        depth = np.rot90(get_depth()[0]) # get the depth readinngs from the camera
        pixels = gamma[depth] # the colour pixels are the depth readings overlayed onto the gamma table
        temp_surface = pygame.Surface(disp_size)
        pygame.surfarray.blit_array(temp_surface, pixels)
        pygame.transform.scale(temp_surface, disp_size, screen)
        screen.blit(font.render(fps_text, 1, (255, 255, 255)), (30, 30))
        pygame.display.flip()
        fpsClock.tick(FPS)

Markov Chain Image Generation

2017-01-18T00:00:00+10:30

Overview

In this post I will describe a method of generating images using a Markov Chain built from a training image. We train a markov chain to store pixel colours as the node values and the count of neighbouring pixel colours becomes the connection weight to neighbour nodes. To generate an image, we randomly walk through the chain and paint a pixel in the output image. The result is images that have a similar colour pallette to the original, but none of the coherence. They still look nice though.

Description

At a high level, this algorithm works in the following manner:

Read in a training `image` into an array of pixels
Create a mapping `colourMap` from pixel colour to a pixel counter colour.
For each `pixel` in `image`:
  For each `neighbour` of `pixel`:
    `mapping[pixel.colour][neighbour.colour] += 1`
Create a blank `newImage`
Choose a random `startingPosition` within the bounds of `newImage`
Choose a random `startingState` from `mapping`
Create a `stack` with `[startingPosition]`
`image[startingPosition] = startingState`
while `stack` is not empty:
  `pixel = stack.pop()`
  For each `neighbour` of `pixel`:
    if `neighbour.isColoured`:
      continue
    `neighbour.isColoured = true`
    Push `neighbour` to `stack`
    `neighbour.colour = ` randomly chosen colour from `mapping[pixel.colour].keys()`. Probability distribution is `mapping[pixel.colour].counts() / sum(mapping[pixel.colour].counts())`
Display `newImage`

Outputs

Here’s some example outputs:

An image generated from my black and white portrait:

Code

To run this you’ll need python and the following packages installed:

pip install pillow numpy pyprind pygame

Feel free to mess around with bucket_size and four_neighbour. In my testing I found they didn’t give much difference in ouput.

The final code is here:

from PIL import Image
import numpy as np
import pyprind
import random
import os
import pygame
from collections import defaultdict, Counter


class MarkovChain(object):
    def __init__(self, bucket_size=10, four_neighbour=True):
        self.weights = defaultdict(Counter)
        self.bucket_size = bucket_size
        self.four_neighbour = four_neighbour

    def normalize(self, pixel):
        return pixel // self.bucket_size

    def denormalize(self, pixel):
        return pixel * self.bucket_size

    def get_neighbours(self, x, y):
        if self.four_neighbour:
            return [(x + 1, y), (x - 1, y), (x, y + 1), (x, y - 1)]
        else:
            return [(x + 1, y),
                    (x - 1, y),
                    (x, y + 1),
                    (x, y - 1),
                    (x + 1, y + 1),
                    (x - 1, y - 1),
                    (x - 1, y + 1),
                    (x + 1, y - 1)]

    def get_neighbours_dir(self, x, y):
        if self.four_neighbour:
            return {'r': (x + 1, y), 'l': (x - 1, y), 'b': (x, y + 1), 't': (x, y - 1)}
        else:
            return dict(zip(['r', 'l', 'b', 't', 'br', 'tl', 'tr', 'bl'],
                            [(x + 1, y),
                             (x - 1, y),
                             (x, y + 1),
                             (x, y - 1),
                             (x + 1, y + 1),
                             (x - 1, y - 1),
                             (x - 1, y + 1),
                             (x + 1, y - 1)]))

    def train(self, img):
        """
        Train on the input PIL image
        :param img:
        :return:
        """
        width, height = img.size
        img = np.array(img)[:, :, :3]
        prog = pyprind.ProgBar((width * height), width=64, stream=1)
        for x in range(height):
            for y in range(width):
                # get the left, right, top, bottom neighbour pixels
                pix = tuple(self.normalize(img[x, y]))
                prog.update()
                for neighbour in self.get_neighbours(x, y):
                    try:
                        self.weights[pix][tuple(self.normalize(img[neighbour]))] += 1
                    except IndexError:
                        continue
        self.directional = False

    def train_direction(self, img):
        self.weights = defaultdict(lambda: defaultdict(Counter))
        width, height = img.size
        img = np.array(img)[:, :, :3]
        prog = pyprind.ProgBar((width * height), width=64, stream=1)
        for x in range(height):
            for y in range(width):
                pix = tuple(self.normalize(img[x, y]))
                prog.update()
                for dir, neighbour in self.get_neighbours_dir(x, y).items():
                    try:
                        self.weights[pix][dir][tuple(self.normalize(img[neighbour]))] += 1
                    except IndexError:
                        continue
        self.directional = True

    def generate(self, initial_state=None, width=512, height=512):
        import cv2
        fourcc = cv2.VideoWriter_fourcc(*'MP4v')
        writer = cv2.VideoWriter('markov_img.mp4', fourcc, 24, (width, height))
        os.environ['SDL_VIDEO_WINDOW_POS'] = "%d,%d" % (0, 0)
        pygame.init()

        screen = pygame.display.set_mode((width, height))
        pygame.display.set_caption('Markov Image')
        screen.fill((0, 0, 0))

        if initial_state is None:
            initial_state = random.choice(list(self.weights.keys()))
        if type(initial_state) is not tuple and len(initial_state) != 3:
            raise ValueError("Initial State must be a 3-tuple")
        img = Image.new('RGB', (width, height), 'white')
        img = np.array(img)
        img_out = np.array(img.copy())

        # start filling out the image
        # start at a random point on the image, set the neighbours and then move into a random, unchecked neighbour,
        # only filling in unmarked pixels
        initial_position = (np.random.randint(0, width), np.random.randint(0, height))
        img[initial_position] = initial_state
        stack = [initial_position]
        coloured = set()
        i = 0
        prog = pyprind.ProgBar((width * height), width=64, stream=1)
        # input()
        while stack:
            x, y = stack.pop()
            if (x, y) in coloured:
                continue
            else:
                coloured.add((x, y))
            try:
                cpixel = img[x, y]
                node = self.weights[tuple(cpixel)]  # a counter of neighbours
                img_out[x, y] = self.denormalize(cpixel)
                prog.update()
                i += 1
                screen.set_at((x, y), img_out[x, y])
                if i % 128 == 0:
                    pygame.display.flip()
                    # writer.write(cv2.cvtColor(img_out, cv2.COLOR_RGB2BGR))
                    pass
            except IndexError:
                continue

            for e in pygame.event.get():
                if e.type == pygame.QUIT:
                    sys.exit()


            if self.directional:
                keys = {dir: list(node[dir].keys()) for dir in node}
                neighbours = self.get_neighbours_dir(x, y).items()
                counts = {dir: np.array(list(node[dir].values()), dtype=np.float32) for dir in keys}
                key_idxs = {dir: np.arange(len(node[dir])) for dir in keys}
                ps = {dir: counts[dir] / counts[dir].sum() for dir in keys}
            else:
                keys = list(node.keys())
                neighbours = self.get_neighbours(x, y)
                counts = np.array(list(node.values()), dtype=np.float32)
                key_idxs = np.arange(len(keys))
                ps = counts / counts.sum()
            np.random.shuffle(neighbours)
            for neighbour in neighbours:
                try:
                    if self.directional:
                        direction = neighbour[0]
                        neighbour = neighbour[1]
                        if neighbour not in coloured:
                            col_idx = np.random.choice(key_idxs[direction], p=ps[direction])
                            img[neighbour] = keys[direction][col_idx]
                    else:
                        col_idx = np.random.choice(key_idxs, p=ps)
                        if neighbour not in coloured:
                            img[neighbour] = keys[col_idx]
                except IndexError:
                    pass
                except ValueError:
                    continue
                if 0 <= neighbour[0] < width and 0 <= neighbour[1] < height:
                    stack.append(neighbour)
        writer.release()
        return Image.fromarray(img_out)


if __name__ == "__main__":
    import sys

    chain = MarkovChain(bucket_size=16, four_neighbour=True)
    try:
        fnames = sys.argv[1:]
    except IndexError:
        fnames = ['me.jpg']
    for fname in fnames:
        im = Image.open(fname)
        # im.show()
        print("Training " + fname)
        chain.train(im)
    # print chain.weights
    print("\nGenerating")
    chain.generate().show()

Jonathan's Blog

Distributed Hyperparameter Optimisation with Keras/Theano/Tensorflow/Hyperas/Hyperopt

Overview

What is Hyperparameter Optimisation?

Requirements

Code

Running

Controller

Worker

Results

Building Tensorflow with OpenCL support on Ubuntu 16.04

Overview

Install AMDGPU-PRO Drivers

Install a recent python

Get The ComputeCPP SYCL Implementation

Get Tensorflow with experimental opencl support

Environment and Building

Verification

OpenCL External Tool for IntelliJ

Linux linker error: _ZTVN10__cxxabiv117__class_type_infoE

libfreenect Python Depth Image

Viewing the Kinect Depth Reading Using Python

Pure Python Implementation

Markov Chain Image Generation

Overview

Description

Outputs

Code

Linux linker error: _ZTVN10cxxabiv117class_type_infoE