Can one infer the hand segmentation map of a person from images captured by wearable cameras in the virtual reality setup? If yes, can we predict more accurate segmentation maps with the help of keyboard responses (i.e., we know which letter the person is typing, can we make better predictions)?
Hand segmentation from a single image is a very difficult problem since an image captures only the 2D projection of a 3D hand. Without additional knowledge (e.g., depth information), the hand shape cannot be inferred easily. However, what if we know which button of the keyboard is pressed? There are dependencies between hand poses and keyboard. The button response could give us a clue. Can an algorithm learn and exploit these to infer the hand segmentation map? To go one step further, can we consider this chanllenging task reversely. Given the hand segmentation map and keyboard, can we know which button is pressed?
We are positive that this task is possible, but by which certainty? Will the algorithm take advantage of the button response?
To find answers to these questions, we will apply data analysis and machine learning. Building on our experties [1,2] we will use deep learning techniques such as CNNs. The project is structured into three tasks: First, it requires building a suitable evaluation dataset (Recording Keyboard typing videos with depth information, labelling hands in videos). Second, the core algorithm that exploits the button responses and image cues needs to be developed. Third, the estimation performance (e.g., mean-IOU, accuracy, f1 score) is to be validated.
 Urooj et al. "Analysis of Hand Segmentation in the Wild." CVPR. 2018.
 Wei Wang et al. "Beyond One Glance: Gated Recurrent Architecture for Hand Segmentation." arXiv 2018.
Back to the project list.
The candidate should have programming experience, ideally in Python. Previous experience with machine learning, computer vision, deep learning platforms (e.g., pytorch) is a plus.
20% Theory, 30% Implementation, 50% Research and Experiments