A Practical System for 3 D Hand Pose Tracking Using EMG WearablesWith Applications to Prosthetics an

Ubiquitous finger motion tracking enables a number of exciting applications in augmented reality, sports analytics, rehabilitation-healthcare, haptics, etc. This article presents NeuroPose, a system that shows the feasibility of 3-D finger motion tracking using a platform of wearable electromyography (EMG) sensors. EMG sensors can sense electrical potential from muscles due to finger activation, thus offering rich information for fine-grained finger motion sensing. However, converting the sensor information to 3-D finger poses is non trivial since signals from multiple fingers superimpose at the sensor in complex patterns. Toward solving this problem, NeuroPose fuses information from anatomical constraints of finger motion with machine learning architectures on recurrent neural networks (RNNs), encoder–decoder networks, and ResNets to extract 3-D finger motion from noisy EMG data. The generated motion pattern is temporally smooth as well as anatomically consistent. Furthermore, a transfer learning algorithm is leveraged to adapt a pretrained model on one user to a new user with minimal training overhead. A systematic study with 12 users demonstrates a median error of 6.24° and a 90%-ile error of 18.33° in tracking 3-D finger joint angles. The accuracy is robust to natural variation in sensor mounting positions as well as changes in wrist positions of the user. In addition, this article validates the feasibility of mirrored bilateral training approach with applications in prosthetic devices. Finally, NeuroPose is comprehensively evaluated on both low-end and recent smartphones with a processing latency of 0.019 s and low energy overhead.