George Markham

MSc Advanced Computing Science Student. Software Engineer, Computer Scientist

[gmarkham-blog1]

In admiration of scripts

One of the things I always tell people when I talk about programming is it’s like you’re magic. I think I heard or read this somewhere when I started writing code as a teenager but it’s stuck with me ever since. Most code I write, as soon as I run it, makes me think about how magical it is to make a computer do something amazing. My undergraduate dissertation project  really hammered home this notion of magic. I was taking a single image of a face and figuring out where they were looking and it was working!

More recently I've been reminded of the magic of programming. I am completing an audio processing/machine learning project that requires the labelling of a reasonable amount of data. Due to the nature of how the data was collected this process was made even more difficult, the solution: scripts. More precisely python scripts (aside: python is fast becoming one of my favourite languages for testing ideas and algorithms quickly). I ended up writing a number of scripts to format my data and label files, reducing the risk of human error and speeding up the process so I could focus more on things like different filterbank implementations.

Back to the original point. The fact that, as a programmer, you have the ability to take a problem and find a solution no matter how small or large, that is absolutely magical. If that solution can then help other people then that's even better.

Extra stuff: I'm considering writing a post sometime soon about processing speech using the MFCC pipeline and also considering using Tensorflow to build some kind of simple speech recogniser (of course using python, at least for now).

[gmarkham-blog1]

WWDC, Apple and the Deprecation of OpenCL

So WWDC just happened and it brought some pretty cool changes; Mojave, UIKit for MacOS and finally a dark mode for Xcode. I’d, however, like to focus on a not so cool change, namely the depreciation of the OpenGL and OpenCL frameworks. OpenCL was originally created by Apple in collaboration with companies like AMD, IBM, Nvidia and Intel, the idea being that you could run parallel software on the different platforms offered by the different companies using a single code-base. Kind of like how you can run Cordova PhoneGap on everything. This meant that no matter whether you were writing high performance software for MacOS or Windows you didn’t need to worry so much about the hardware available it would be able to run basically everywhere. No more though. This deprecation comes as Apple is pushing it’s Metal framework so it’s really no surprise that they want devs to focus on Metal rather than OpenCL. With Metal Apple is trying to frame itself and it’s devices as a platform for high-performance computing. Just look at the iMac Pro, it’s an insanely powerful machine in the (modern) classic iMac design that’s capable of being used for computer vision and potentially even some small ML tasks. And it’s not just hardware that proves Apple’s push for high-performance computing, their Core ML framework got an update aswell. Core ML is now faster and allows developers to create their own models rather than just converting previously created models to be used with the framework. So what’s the issue right? Apple’s adopting and developing some pretty awesome frameworks so what’s the issue? The issue comes with the Apple-ness of it. Ever tried to take an iPhone apart? Probably not, it’s difficult and Apple really doesn’t want you doing it. It’s like that with the software too, they built it to be used a certain way and they don’t want you straying from that path. Take this excerpt from the Metal docs found when you look at the MPSKernel documentation:

You should not use the MPSKernel class directly. Instead, a number of subclasses are available that define specific high-performance data-parallel operations.

They then give an example of a sobel operator. This is all great but what happens when you need to write a custom kernel. They actively discourage you doing so and I’d bet that means they’re not going to make it easy for you. Not like OpenCL. The other issue is OpenCL was cross-platform. I could run the same kernels on PC and Mac with no issues. But Metal isn’t built for PC or Linux so if you want write a kernel once to run everywhere you can’t, you have to write once for Metal (which you can’t do) and then once for OpenCL. This isn’t all doom and gloom though. Let’s face it no one writes high-performance code for MacOS, if you’re writing it it’s probably going to run on Linux. The code you are going to write for MacOS will benefit greatly from the care Apple has put into Metal and the fact that it will run better on Mac hardware than comparable code would run on comparable windows hardware (that’s Apple’s thing after all). Finally you don’t have to ever write your own kernels really just use boost (or something else) so it’s not really an issue that Apple discourages this. All in all I’m disappointed with the decision but not surprised. I’m also optimistic, Apple is doing everything it can to help devs write faster, more efficient and more intelligent software and that means better experiences for everyone. And let’s face it a deprecation isn’t an execution as much as a death sentence.

[gmarkham-blog1]

Analysing a person's gaze

Working out where a person is looking is an incredibly useful feature for robotics. It enables a system to more adequately understand a human being's intentions and meaning during and preceding interactions. Think, for a second, about how much you use cues given by a person's gaze when interacting with another person. They may be looking straight at you indicating an interest in what you are saying, or they may be focused somewhere else and indicating that they are indeed not focused on the conversation. Even before interaction begins you interpret a person's desire and intent to interact with you based on their gaze, if they are holding mutual gaze with you it is a signal they may intend to stop and talk to you. All of these cues can be missed by robots that don't have an ability to determine gaze. The goal for my undergraduate dissertation was to develop an algorithm that allowed a system, using just a webcam, to find where a person's gaze lay. The reason I wanted to use just a webcam was to simplify the process and make it non-invasive. To accurately determine gaze it is often useful for a person to wear hardware that enables eye tracking but I aimed to prove that this was not necessary in order to enable a system to get cues given by a person's gaze. My algorithm allows a system that has already got facial landmarks, either through a REST API service such as the Microsoft Face API or through a local facial landmark detection tool such as dlib to determine the angle of gaze. If you can know the distance a person is from a target then it is trivial to determine the point in 3D space that a person is looking. To find out how I did it and to see the algorithm my dissertation is available at the link below. A Study of Non-Invasive Gaze Detection Methods

[gmarkham-blog1]

Parallel Reduction in OpenCL

Parallelisation is an important part of modern computing. It is important to be able to utilise all of the hardware at one's disposal in order to increase execution time of software. It is also quite useful to run calculations on the GPU as opposed to the CPU as the GPU is in most cases better suited to dealing with arithmetic than a CPU. As an example of how quick parallelisation can make a program I wrote code using C++ and OpenCL to read in 1.8 million data points from a text file, find the minimum, maximum and standard deviation and produce a frequency histogram. This program executes in around 2 seconds on the lab computer's GTX 1050. My code uses the fscanf c++ function to quickly read the text file containing the data into a vector. This vector is then passed to a series of functions to find the mean, minimum, maximum, standard deviation and histogram of the data. The mean, minimum, maximum and standard deviation algorithms are all based on a reduction outlined below using OpenCL

__kernel void find_sum(__global int* input, __global     int* output, __local int* local_cache){     int global_id = get_global_id(0);     int local_id = get_local_id(0);     int local_size = get_local_size(0);     //Copy values into local memory     local_cache[local_id] = input[id];     //Wait for all threads to catch up     barrier(CLK_LOCAL_MEM_FENCE);     for (int i = local_size/2; i > 0; i /= 2){         if(local_id < i){             local_cache[local_id] += local_cache[local_id + i];         }         //Wait for all threads to catch up         barrier(CLK_LOCAL_MEM_FENCE);     }     if (!local_id) {         //Can use atomic add as we're dealing with ints         atomic_add(&output[0], local_cache[local_id]);     } }

The idea of this kernel is that it uses strides to perform an associative operation such as addition on two values and propagate down so that the result is inserted into the first memory space of that particular local memory group. That might be a bit heavy if you've not done any parallel computing before so let's look a couple of diagrams: In the example above input is an array of 40 integers from 0 to 39 and output has been filled with 0's. Notice how each work group or thread contains 10 values from the input array. This can be defined as whatever you want (as long as your hardware can handle it) in your host code but 10 seems to work well enough for me. The reason local memory is used is because it's much quicker for a thread to access local memory than to repeatedly access global memory. This optimisation as with any optimisation is essential to speeding up your program and utilising the available hardware. The reduction is actually run on local memory, after which the values in work item 0 (the first local memory item) are, in this case, summed with the value in the first memory location of the output array. This means you can get the sum of all inputs by operating on different parts of the input array in parallel (hence the term parallel computing). The second diagram crudely shows how the reduction would operate on the local memory of thread 0. Notice how each curve connecting two numbers represents a stride and how the input is reduced from 10 values down to 1 value (the sum of the 10). Also notice how the strides increase by a power of 2 each time and operate on the outputs of the last strides. Hopefully the kernel above makes this clearer. This is, in part, how I managed to achieve the 2 second execution time for my 1.8 million values most of which was taken up by reading in the data. The full code including the kernel and host code is on my GitHub. I'll try and write more about parallelisation in the future. Thanks for reading and happy coding!

[gmarkham-blog1]

Gaze Detection

I'm in my final year of my Computer Science undergraduate degree in Computer Science. This requires me to complete a dissertation project of my choosing. Due to my interests in Computer Vision and Robotics I decided to work on a gaze detection system to enable a robot to understand where a person is looking. The idea is that if a robot can understand where you are looking it can make certain decisions about whether you want to interact with it, what you might currently be interested in and a multitude of other things that humans can determine about each other but currently robots aren't able to. My system uses pupil coordinates to determine the angle between where they should be at rest (the centre of the eye) as opposed to where they currently are. I'll hopefully be posting my work here so check back in a few months to read the full paper.   All of my projects are available on my Github.

[gmarkham-blog1]

Passport.JS REST API Tip

A new project required me to build a REST API that can authenticate a user and pass them back a json web token (JWT) for a cross-platform app. I decided that, as I wanted the ability to expand the service to use authentication providers such as Google and Facebook that passport.js was a pretty good option. Having no experience with it I searched around but couldn't find how to use it without enabling sessions. Eventually I worked it out and thought I'd share the code snippet in case anyone else is struggling with this.

I'm using a local strategy that I created to insert a user into the database (there are loads of tutorials on strategies and stuff all over the place but this one at scotch.io is really nice and very comprehensive). Instead of redirecting the user to another view I simply carry on with the standard Express req, res flow, sign the user object to create a JWT and then passing that back to the client as shown in the image below.

This allows you to use the passport middleware but also have control over what you pass back to the user and how you do it.

Oh yeah and these are the packages you'll need to do this:

Image generated using https://carbon.now.sh

Thanks for reading!

[gmarkham-blog1]

High Sierra Root Hack

The other day I came across a tweet which I'm sure by now many of you have already seen or at least heard of, it was tweeted to Apple's Support account and detailed a way of gaining root access to a system through a flaw in a login box that gets displayed in System Preferences. Firstly here's a link to an article about the flaw and how to temporarily stop it before Apple releases their software update: https://www.macrumors.com/2017/11/28/macos-high-sierra-bug-admin-access/ Firstly this should not have been reported on Twitter at all. This is a serious security flaw in a widely used operating system and therefore should have been disclosed to Apple directly and privately, companies like Apple offer bug bounty programs for exactly these kinds of situations. The implications of this being released so publicly could be disastrous for those who don't regularly update their systems or who haven't heard about the news for some reason or another. Secondly how on earth could this have slipped through testing? I understand it's impossible to test every single use case but this is a basic boundary test "what happens if a user tries to log in to root without supplying any data". This is stuff that we get taught at university as undergrads, Apple can and needs to do better. In saying this I understand that it's entirely possible for plenty of bugs to slip through testing, as software engineers sometimes we assume things like the user will set up a root account with a root password (that's the fix to the bug by the way). It's easy to forget that end users are not all software engineers or people interested in tech, especially when we are, often, surrounded by or engage with people who are. This is as much a testing problem as it is indicative of a wider issue of UX design and making things intuitive and simple for all users. In any case I still love Apple's products and will continue to run MacOS as my main OS.

[gmarkham-blog1]

What I'm up to

As part of the final year in my undergraduate course I, like everyone else, have to complete an independent project. For my computer science course this has to be some scientific research with an artefact attached to it relating to the field of computer science. I decided to do my project in the fields of artificial intelligence, robotics and computer vision because robots are cool (the dream is to have a real life C3PO or R2D2) and machine learning is even cooler (in my opinion). So right now I'm starting research into the project, I've got a lot to learn and it's certainly going to be a challenging project but given the continuing increase of importance and interest in the aforementioned areas I hope it will help me learn something new and interesting. The other thing I'm interested in at the moment is Google's Pixel 2 portrait mode implementation (Google's blog post). It is a really interesting, practical use of neural networks and computer vision and the fact it appears to be working even better than other companie's dual lens approach is extremely promising, when I get some free time I think it'd be fun to try and hack together a similar implementation. I'll be keeping this blog up to date (hopefully) with some interesting things I uncover as I go through this project and my other modules. I might also write some posts about the other modules I'm currently engaged in studying.

[gmarkham-blog1]

Back again...

Right so I started this blog (I think i was 15 when I started this) back in 2013 and made a couple of posts. It's now 4 years later, I'm 20 and in my final year of my undergraduate degree in computer science and I'm reviving this blog to post things I find interesting. I'm leaving the first two posts up for posterity and so I can read back and see what's changed. As a bit of a starting point I'm going to comment on some differences from then to now:

Hobby to work - Back in 2013 I was writing code because it was fun, I went on to study Computing, Physics and Electronics at A-Levels and I'm now studying Computer Science at university. Whilst I still immensely enjoy writing code and building things it has transitioned into something that has become my main focus for my studies and my future.

My knowledge - This is obvious, everyone learns things and going through 4 years of education is obviously going to teach me things my tiny 15 year old brain could only have dreamed of. In saying that I've gained skills and interests I never thought I'd have or even knew existed back then, most notably I think functional programming (I did an assignment in Haskell earlier this year) and now Computer Vision/Machine Learning/Robotics.

An appreciation - Back then I found it fun, I can remember I quite liked writing JavaScript that spat out an infinite number of alert boxes and running it on my friends computers in school. That has now graduated (or nearly graduated) into an appreciation for the art and skill of Software Engineering and Computer Science as a whole, not only the subject but the incredibly talented people who have a hand in building the modern world.

Pythagoras - Bit of a strange header but over my last 4 years Pythagoras and his theorem has popped up constantly in everything I do, I found myself wanting to hop in a DeLorean to go back to my younger self and tell him to appreciate the beauty of mathematics and especially this insanely useful theorem.

So there you have it, I want to post on this blog regularly (probably not going to happen). Most of what I write will most likely be rambling pieces but maybe you'll find something interesting.

[gmarkham-blog1]

Getting MEAN

If you haven't heard there's this new thing in town known as a MEAN stack. It is my understanding that this stands for Mongo, Express, Angular, and Node. I've just started learning, individually, each of these technologies and I must say they are all very intuitive. I've been writing javascript (properly) for just over a year now and that's been long enough for me to get to grips with Javascript basics, Jquery and (just recently) AngularJS. Somewhere along my travels I encountered a server-side language known as Node and, as a javascript lover, I was intrigued. Here was a way of writing awesome back-end systems in a language I love. I eventually started learning it and found it very intuitive and clear. It is also extremely quick. I am still learning Node but I am already looking to expand into socket.io, express, and mongoDb and of course on the client side Angular. So that's been what I've been up to recently, oh yeah and a whole lot of Java programming on the side, which I also love.

[gmarkham-blog1]

Phonegap with Android Studio

*** DISCLAIMER: THIS IS DEFINITELY OUT OF DATE NOW ***   Alright, I havent got much time cos Eurovision's about to start but today I used Android studio as an editor for the phonegap API, which is cool. Android studio is the new IDE from Google that was released at this year's (2013's) I/O. Phonegap and Android studio seem to get along brilliantly, which is hardly surprising but hey, just thought you should know. I'll be posting more about this later but for now you can follow me on twitter: www.twitter.com/gmarkhamse