Auto Chat Sticker: Foreground Extraction using the Dual Lens HTC One M8
HTC recently released a new version of the HTC One, called the M8, with dual lenses on the back that allows lots of interesting uses of the produced depth data - for example quick and easy foreground extraction.
You can start with the DualLensDemo included in their public API: https://www.htcdev.com/devcenter/opensense-sdk/htc-dual-lens-api
This examples produces a depth map where the color changes based on the distance. Here is a screenshot of what it looks like:
Here is the code to draw the depth visualization:
DualLens.Holder<byte[]> buf = mDualLens.new Holder<byte[]>(); DualLens.DataInfo datainfo = mDualLens.getStrengthMap(buf); int [] depthData = new int[datainfo.width * datainfo.height]; int leftByte; for(int i = 0; i < datainfo.width * datainfo.height; i++) { leftByte = buf.value[i] & 0x000000ff; depthData[i] = mColorBar[leftByte*500]; } Bitmap bmp = Bitmap.createBitmap( depthData, datainfo.width, datainfo.height, Config.ARGB_8888); image.setImageBitmap(bmp); image.setBackgroundColor(Color.WHITE);
You can keep a reference to the original image bitmap and then either pull colors from it or white out pixels based on the depth like this:
DualLens.Holder<byte[]> buf = mDualLens.new Holder<byte[]>(); DualLens.DataInfo datainfo = mDualLens.getStrengthMap(buf); int [] outputImage = new int[datainfo.width * datainfo.height]; int pixelDepth; for(int i = 0; i < datainfo.width * datainfo.height; i++) { pixelDepth = buf.value[i] & 0x000000ff; int depthX = i % datainfo.width; int depthY = i / datainfo.width; int originalX = originalImageBitmap.getWidth() * depthX / datainfo.width; int originalY = originalImageBitmap.getHeight() * depthY / datainfo.height; //White out background, pick original color from foreground. outputImage[i] = pixelDepth > 64 ? Color.WHITE : originalImageBitmap.getPixel(originalX, originalY); } Bitmap bmp = Bitmap.createBitmap( outputImage, datainfo.width, datainfo.height, Config.ARGB_8888); image.setImageBitmap(bmp); image.setBackgroundColor(Color.WHITE);
Source code on GitHub. Here is a screenshot with the modifications:
Boom! Instant chat stickers just like are zooming all over the place in hit communications apps like Line and Facebook Messager. Foreground extraction is also very useful for making product images for user marketplace apps.
Traditionally it has been very labor intensive and in real graphics and retail industries interns and other employees just starting out get tasked with having to carefully edit photos or use more complex imaging setups than we have on consumer phones.
So using the M8 Dual Lens API to make the results better and with no effort is really exciting! They also offer other ways to get data, like OpenGL geometries.