Appropriating the human body as an interactive surface is attractive for many reasons. Foremost, skin provides a natural and immediate surface for dynamic, digital projection. Although it introduces some color and physical distortion, the resolution, framerate, and overall quality can be high. More importantly, it offers considerable surface area for interactive tasks –many times that of e.g., a smartwatch display. With today’s smartwatches containing multi-core, multi-gigahertz CPUs, one could argue their small touchscreens are the chief bottleneck to unlocking richer and more useful applications. Indeed, several widely publicized, conceptual on-skin projection watches have been proposed, most notably Cicret and Ritot.

A second benefit is that our bodies are always with us, and are often immediately available. This stands in contrast to conventional mobile devices, which typically reside in pockets or bags, and must be retrieved to access even basic functionality. This generally demands a high level of attention, both cognitively and visually, and can be socially disruptive. Further, physically retrieving a device incurs a non-trivial time cost, and can constitute a significant fraction of a simple operation’s total time.

Lastly, as the colloquialism “like the back of your hand” suggests, we are intimately familiar with our own bodies. Through proprioception, we can easily navigate a finger to our palm, even with our eyes closed. We have finely tuned muscle memory and hand-eye coordination, providing a high level of input performance, for both absolute touch location and relative gesturing –powerful interaction modalities that worn systems can leverage.

However, despite these significant benefits, building practical, worn projection systems has remained elusive. In order to achieve sufficient projection brightness, sensing robustness, and compute power, past on-body systems have employed full-sized components (e.g., USB depth cameras, portable projectors). The resulting size of these systems mean that they must be worn on the upper arm or shoulder, and most often tethered for compute and power.

We present LumiWatch, a custom, tightly integrated, and fully self-contained on-skin projection wristwatch. It incorporates a 15-lumen scanned-laser projector, a ten-element time-of-flight depth-sensing array, quad-core CPU running Android 5.1, and a battery good for one hour of continuous projector operation(or one day of occasional use). Our prototype measures 50×41×17mm, nominally larger than the production 42mm Apple Watch Series 3 (43×36×11mm). With our watch, we demonstrate continuous 2D finger touch tracking on the skin with coordinated interactive graphics. Owing to the shallow angle of projection, our graphics pipeline must rectify interfaces to the complex, non-planar geometry of the arm.

Our hardware and software, taken together, transform the arm into a coarse touchscreen, offering roughly 40 cm2 of interactive surface area, more than five times that of a typical smartwatch. This interactive area supports common touchscreen operations, such as tapping and swiping, allowing it to offer similar interactions to that of a (single-touch) smartphone. Although obstacles remain for practical adoption, we believe our work demonstrates the first functional projection smartwatch system and constitutes a significant advance in the state of the art.

Research Team: Robert Xiao, Teng Cao, Ning Guo, Jun Zhuo, Yang Zhang, Chris Harrison

This research was done in collaboration with ASU Tech.

Additional media can be found on Robert Xiao's site.