How Exif Data Can Affect Your Image Recognition Training
We apologize for the recent radio silence! We’ve been working on some amazing things for the past months and today we would like to share a small discovery we made.
When doing TensorFlow image recognition retraining there are a lot of things to take into consideration. The biggest one we found is image rotation and it might be affecting your training data negatively!
We were playing around with our TensorFlow training tool and wanted to create a soda can recognition model. We handed out different cans to everyone at the office and asked them to take pictures of them in different settings (ie: different backgrounds, angles, lighting conditions). For about 20 minutes we were all running around the office taking pictures of soda cans with our phones and in the end we sent them all to our training tool.
We began training the TensorFlow model using the transfer learning method and then loaded the trained model into our simple testing application. Eager to see the results the team gathered around and we began testing. In the past, we’ve trained models pooling images from the internet but this time we used the data on our phones. When taking pictures, some phones add exif data which is a set of information about the conditions in which the picture was taken. One of these pieces of information is the rotation of the phone when the picture was taken.
The way the phone takes the picture varies with each manufacturer or even OS version. In our case 5 out of the 6 phones would take the picture and add exif data noting that the picture had been taken with a 90º rotation since we were holding our phones in portrait mode.
The 6th phone was natively taking the pictures in portrait mode and did not add any rotation in the exif data. It was simply denoting the orientation as “Normal”.
This means that the Diet Coke can was actually being pictured in landscape mode and the rotation of 90º was being added by the phone after taking the picture where as the Coke Zero was being captured in portrait so there was no need to add the rotation flag in the exif data.
What does this mean for TensorFlow Image Recognition training?
When testing the trained model we were getting incorrect results because the Diet Coke was being tagged as Coke Zero with a very high accuracy. This came as a surprise because both cans are completely different color and were pictured in a wide range of backgrounds.
When looking into the issue we noticed that in our data set, the Diet Coke images were being shown sideways which came as a surprise because we took the images in portrait mode.
We proved our suspicions by pointing the testing app at the can in landscape mode which in fact showed a very accurate tag.
Digging into it we found out that TensorFlow actually ignores exif data included in the jpeg encoding when processing images.
We concluded that the exif data does play a big role when training Image Recognition models in TensorFlow.
What can you do about it?
There are a lot of ways of dealing with this. You could pre-process the images and apply the exif rotation to them before sending them to TensorFlow or you could also set the flags that add random rotation to the image dataset so that the neural network does not take rotation into consideration when training.
Expect more frequent updates from the BlueChasm team. We are very excited to talk about our new ventures with artificial intelligence. If you have any questions, the best way to contact us is at [email protected] or via twitter @bluechasmco