Public EC2 AMI with Torch and Caffe deep learning toolkits
tl;dr I built an EC2 AMI on AWS including my favorite deep learning libraries Torch7 and Caffe, and all necessary tools and libraries to run them on a GPU. It is publicly available as ami-ffba7b94 (launch on AWS). When launching the instance, pick the type g2.2xlarge or g2.8xlarge which feature Nvidia GPUs.
I finally decided to take a closer look at deep learning toolkits Torch and Caffe.
Torch is a general purpose scientific framework developed at NYU and used among others by machine learning teams at Facebook, Google/DeepMind, Nvidia, AMD, Intel.
Caffe is a deep learning tool focused on convolutional networks and image recognition that provides pre-trained models and a simple file format to describe deep learning networks.
Amazon offers EC2 instances with Nvidia GPUs which are suitable to train deep learning models. Starting from the Linux Ubuntu AMI ami-6238470a (launch on AWS) mentioned at the Torch7 cheatsheet, I added:
fbcunn, a Torch package developed by Facebook and recently open-sourced
dp, a deep learning package for Torch
loadcaffe, a package to load Caffe networks and models on Torch
See below the commands issued to install the above packages and their dependencies:
luarocks install dp # update cunn and cutorch. This is necessary to avoid a cmake problem in both fbcunn and loadcaffe luarocks install cunn luarocks install cutorch # Install Folly, fbthrift, thpp and fblualib curl -sk https://raw.githubusercontent.com/soumith/fblualib/master/install_all.sh | bash # Install fbcunn git clone https://github.com/facebook/fbcunn.git cd fbcunn && luarocks make rocks/fbcunn-scm-1.rockspec # Install loadcaffe sudo apt-get install protobuf-compiler # protobuf-dev is not sufficient because protoc command is needed by loadcaffe luarocks install loadcaffe
The AMI is publicly available with the identifier ami-90206ef8 (launch on AWS, instances with Nvidia GPUs are called g2.2xlarge).
Update: I installed the Caffe package, compiled to take advantage of Nvidia's cudnn library, in addition to the above.
# Install caffe dependencies as described at http://caffe.berkeleyvision.org/installation.html sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev # Install a BLAS library (ATLAS) sudo apt-get install libatlas-base-dev # Clone caffe from github git clone https://github.com/BVLC/caffe.git # Install python dependencies for Caffe cd caffe cd python for req in $(cat requirements.txt); do sudo pip install $req; done # Install cuDNN library v1 from developer.nvidia.com (needs registration) # Build succeds but tests fail with v2 as of February 15th, 2015 # Edited caffe/Makefile.config to include Anaconda python directories and turn on cuDNN library from Nvidia # Build caffe # crazy trick to pass make all and avoid a conflict # documented at https://github.com/BVLC/caffe/issues/757 mv $HOME/anaconda/lib/libm.so $HOME/anaconda/lib/libm.so.tmp make all make test # crazy trick to have make runtest work # documented at https://github.com/BVLC/caffe/issues/1463 sudo ln /usr/lib/x86_64-linux-gnu/libhdf5.so.7 /usr/lib/x86_64-linux-gnu/libhdf5.so.8 sudo ln /usr/lib/x86_64-linux-gnu/libhdf5_hl.so.7 /usr/lib/x86_64-linux-gnu/libhdf5_hl.so.8 make runtest make distribute
The AMI including both Torch7 and Caffe is publicly available with the identifier ami-a0e4b1c8 (launch on AWS, instances with Nvidia GPUs are called g2.2xlarge).
Update 2: following the discussion below, a few more fixes with regards to importing caffe modules into python scripts.
# similar as above to address anaconda distribution bug # documented at https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/-DLG2ZdTkw0 mv $HOME/anaconda/lib/libm.so.6 $HOME/anaconda/lib/libm.so.6.tmp # Add protobuf with pip install to ensure import caffe works in python pip install protobuf # add to .bashrc # export PYTHONPATH=/home/ubuntu/caffe/python:$PYTHONPATH
Update 3: Updated and re-installed fbcunn and its dependencies (folly, fbthrift, thpp and fblualib) for multi-GPU parallel training and testing.
Update 4: Added Torch demos & tutorials, LSTM, Neural Turing Machines and Andrew Karpathy's char-rnn to the torch-gits folder.
The AMI including both Torch7 and Caffe, including fixes to Python module imports, is publicly available with the identifier ami-ffba7b94 (launch on AWS, instances with Nvidia GPUs are called g2.2xlarge and g2.8xlarge).
stormborns
yousharknotpass
gummygunk
titsay
dailybridgerton
origamiaround
pixelskylines
vaporwave