Digitizing a user's hands for use in interactive systems has been a long standing research area, with seminal systems such as the Sayre Glove and DataGlove presented more than half a century ago. Since then, advanced in sensors and machine learning have allowed systems to be miniaturized and become less invasive to the user, with the ultimate aim to not encumber the hands at all. Uses of hand pose tracking are numerous, including domains as diverse as virtual and augmented reality, spatial user interfaces, sign language recognition, and context awareness.

While hand pose sensing via external cameras and other remote sensors is possible, in this work we focus on worn systems that provide pervasive input capabilities. Today, the most capable worn hand pose systems in the literature use optical methods (e.g., RGB cameras, thermal cameras, range finders). While successful, they are sensitive to occlusion from clothing and the user's hand itself in certain poses. Secondarily, wrist-worn camera-based methods innately have privacy implications that can deter consumers. For this reason, researchers continue to explore new methods that can either stand alone or, in the future, contribute to multimodal sensing approaches.

To this literature, we contribute a new system called EtherPose (a homage to Etherphone, the original name of Leon Theremin's hand sensing musical instrument that broadly utilizes the same phenomena). Instead of measuring proximity via capacitive coupling with an external antenna (as Theremin did), we use a small worn antenna emitting a swept-frequency RF signal and measure the reflected signal's magnitude and phase shift (i.e., S11 parameter) with a compact, battery-powered vector network analyzer (VNA). As the user's hand changes geometry (i.e., to form different poses), the expanded antenna ground plane formed by the user tissue changes, therefore changing the antenna self-resonance and thus the impedance characteristic of the antenna observed at a predetermined frequency.

To inform the design of our final prototype, we conducted a series of simulation and empirical studies, which we detail in subsequent sections. The diameter of the wrist allowed us to include an additional, second antenna, which helps to capture other hand geometry changes. This process culminated in a proof-of-concept device, coupled with a machine learning backend, on which we ran user studies. Briefly here, for continuous hand pose tracking, we found a mean euclidean joint error of 11.6 mm across our nine participants. Inspired by recent work, we also investigated wrist rotation estimation, finding a mean angular error of 5.87 degrees.

The contributions of this paper are multifold. Foremost, EtherPose is the first demonstration of continuous hand pose tracking using antenna impedance characteristic. This signal is robust to varying clothing and lighting conditions, and is more privacy-preserving than comparable camera-based methods. Our iterative development approach is also uncommon, relying on tandem real-world experiments and computer simulations. These results informed the design of our untethered, battery-powered, and real-time EtherPose prototype. We then use this setup to evaluate three input modalities, whereas most prior systems explore a single modality.