Touchscreen technologies have made tremendous progress over the past half-century. Metrics such as spatial accuracy, maximum screen size, scratch resistance, water rejection, optical clarity, and number of touch points tracked have steadily improved, while costs have continued to fall. For this reason, touchscreens can be found in domains as diverse as kitchen appliances to airplane cockpits, and of course the ubiquitous smartphone. However, there is one metric that has been notoriously stubborn to improve: touch latency. Specifically, the motion-to-photon delay between what a finger does and how fast coordinated graphics are drawn. Most touchscreen systems have improved latency below the 100ms threshold for keyboards and mice that has been the industry standard for decades, yet even modern generation smartphones have dragging latencies on the order of 80ms.

This delay causes a rubber-banding effect that impacts precision in tasks such as drawing, and, in general, breaks the realism and responsiveness of direct manipulation interfaces. Thus, techniques to reduce touchscreen latency are an active area of research. Many hardware and software approaches have been proposed with varying degrees of success. Software extrapolation is inherently limited by old and ambiguous positional data, and closing the gap to zero perceptible latency with acceptable jitter may be impossible. Hardware approaches appear more promising, but it is not trivial to simply increase system scan rate due to factors such as sensor noise, timing constraints, data overhead/computation, and power consumption.

In this paper, we present a new hardware+software approach to reducing touchscreen latency. Specifically, we apply a film embossed with a 2D pattern of small bumps with a pitch of 5 microns. When a finger, stylus, or tangible is translated across this patterned surface, interaction with the micro-features induces an acoustic vibration with a fundamental frequency that directly encodes x and y velocities. This vibration is low amplitude and, at many velocities, ultrasonic, meaning it is imperceptible to the user. We sample this signal using a conventional audio pipeline at 192kHz with a mean latency of 28ms – less than half the time of most touch pipelines. We fuse this high-bandwidth, low-latency, 1D vibroacoustic signal with low-framerate but high-spatial-accuracy 2D touch data reported by a conventional projected capacitance touchscreen (latency of 69ms). We note for the reader that our technique could also work on resistive, self capacitive, optical, acoustic, and a plethora of other touchscreen technologies, but we focus on projected capacitance because of its dominance in the touchscreen market. By fusing this multimodal data, our machine learning model can make a more accurate prediction of touch location than using touch alone, reducing latency from 96ms to 16ms with mean distance error of 5.13mm.

Research Team: Craig Shultz, Daewha Kim, Karan Ahuja and Chris Harrison

Awards: Best Paper Honorable Mention