#pose detection

[prostheticknowledge]

The Alternative Late Show with Stephen Colbert

Short video demo by Cristóbal Valenzuela combines DensePose with Pix2PixHD image translation frameworks presents a form of visual puppetry based on pose detection. Of course, this isn’t perfect, but the results are interesting, and it looks like there will be an interactive demo sometime in the future.

You can view the video here

[prostheticknowledge]

MonoPerfCap

Graphics research from The Max Planck Institut can use single camera footage to reconstruct a 3D model of a person and replicate their movement from the same footage:

We present the first marker-less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium-scale non-rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non-rigid deformations, even from multi-view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch-based pose estimation strategy. Joint recovery of per-batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium-scale non-rigid alignment. We demonstrate state-of-the-art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.

More Here

[prostheticknowledge]

Real-time Human Pose Estimation in the Browser with TensorFlow.js

Machine Learning web demo from Google Creative Lab and Dan Oved can detect human poses via a webcam with web code:

In collaboration with Google Creative Lab, I’m excited to announce the release of a TensorFlow.js version of PoseNet¹,² a machine learning model which allows for real-time human pose estimation in the browser.

So what is pose estimation anyway? Pose estimation refers to computer vision techniques that detect human figures in images and video, so that one could determine, for example, where someone’s elbow shows up in an image. To be clear, this technology is not recognizing who is in an image — there is no personal identifiable information associated to pose detection. The algorithm is simply estimating where key body joints are.

Ok, and why is this exciting to begin with? Pose estimation has many uses, from interactive installations that react to the body to augmented reality, animation, fitness uses, and more. We hope the accessibility of this model inspires more developers and makers to experiment and apply pose detection to their own unique projects. While many alternate pose detection systems have been open-sourced, all require specialized hardware and/or cameras, as well as quite a bit of system setup. With PoseNet running on TensorFlow.js anyone with a decent webcam-equipped desktop or phone can experience this technology right from within a web browser. And since we’ve open sourced the model, Javascript developers can tinker and use this technology with just a few lines of code. What’s more, this can actually help preserve user privacy. Since PoseNet on TensorFlow.js runs in the browser, no pose data ever leaves a user’s computer.

You can find more background to the project here

You can try out the web demo yourself here

[prostheticknowledge]

DensePose

Graphics research from INRIA, CentraleSupélec and Facebook AI proposes method of calculating 3D model of people from 2D video using neural networks in realtime, and can even apply texture mapping to its subjects:

In this work, we establish dense correspondences  between an RGB image and a surface-based representation of the human body, a task we refer to as dense humanpose estimation. We first gather dense correspondences for 50K persons appearing in the COCO dataset by introducing an efficient annotation  pipeline. We then use our dataset to train CNN-based systems that deliver dense correspondence ‘in the wild’, namely in the presence of back-ground, occlusions and scale variations. We improve ourtraining set’s effectiveness by training an ‘inpainting’ net-work that can fill in missing ground truth values, and report clear improvements with respect to the best results that would be achievable in the past. We experiment with fully-convolutional networks and region-based models and observe a superiority of the latter; we further improve accuracy through cascading, obtaining a system that delivers highly-accurate results in real time.

More Here

[prostheticknowledge]

Human Pose Detection - Mining Body Language from Videos

Machine Learning investigation from samim examines body language in video using recently released open-source code library OpenPose:

From Gene Kelly’s Step-Dance to Bruce Lee’s Kung-Fu — iconic movement has made history. Communicating through Body Language is an ancient art form, currently evolving in fascinating ways: Computationally detecting human body language is becoming effective and accessible. This experiment explores enabling technologies, applications & implications.

For over 20 years, Motion Capture has enabled us to record actions of humans and then use that information to animate a digital character or analyse poses. While movie makers and game developers embraced such technologies — it until recently required expensive equipment which captured only few aspects of the overall performance.

Today, a new generation of machine learning based systems is making it possible to detect human body language directly from images. A growing number of research papers and open-source libraries addresses key aspects: Body, Hand, Face, Gaze Tracking. Identity, Gender, Age, Emotion and Muscle strain Detection. Action Classification & Prediction. We now can...

More Here

[prostheticknowledge]

Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

Computer Science research from Carnegie Mellon University Robotics Institute can detect the pose positions of multiple people without the need of additional hardware:

We present a realtime approach for multi-person 2D pose estimation that predicts vector fields, which we refer to as Part Affinity Fields (PAFs), that directly expose the association between anatomical parts in an image. The architecture is designed to jointly learn part locations and their association, via two branches of the same sequential prediction process. The sequential prediction enables the part confidence maps and the association fields to encode global context, while allowing an efficient bottom-up parsing step that maintains tractable runtime complexity. Our method has set the state-of-the-art performance on the inaugural MSCOCO 2016 keypoints challenge, and significantly exceeds the previous state-of-the-art result on the MPII Multi-Person benchmark, both in performance and efficiency. 

More Here and Github code here