Touch input is the dominant mode of interaction on mobile computers and is increasingly common in laptops, self-service kiosks, domestic appliances, and cars. The most prevalent touchscreen technology used today is projective-capacitive, which uses a row-column arrangement of sensing electrodes (typically made from transparent conductive materials, such as ITO, sandwiched between cover glass and a display). A specialized sensing IC measures the coupling capacitance not only at a single point, but at each row-column intersection, building a 2D signal often called a "capacitive image" in the literature. Capacitive objects (e.g., fingers and metal items) touching the screen appear as "blobs" in the image, which can be tracked over time by standard computer vision algorithms, enabling inputs such as taps and swipes.

As touchscreens are primarily designed to capture finger input, the pitch of the capacitive matrix is generally sized such that fingertips overlap at least two pixels horizontally and vertically, as this permits a fairly accurate sub-capacitive-pixel interpolation of the true touch centroid. For this reason, capacitive touchscreen sensor resolution has not changed much over time or device category: e.g., LG 'G' smartwatch (3.5mm pitch), Nexus 5 smartphone (4.1mm pitch), Samsung S4 smartphone (3.9mm pitch), Samsung Galaxy Tab S2 tablet (4.0mm pitch), and Microsoft 55" Surface touchscreen (5.9mm pitch). This coarse resolution immediately precludes many interesting applications. Moreover, increasing capacitive touchscreen sensor resolution is not trivial, as adding rows and columns incurs a quadratic cost in terms of the number of intersections to sense, increasing latency when the current trend is towards more responsive touchscreens. Thus, if we wish to have higher-resolution capacitive image data, we must turn to other approaches.

In this paper, we show how super-resolution techniques – long used in fields such as biology and astronomy – can be applied to capacitive images. The fundamental operation begins by capturing a series of images at slightly different perspectives or offsets. In the case of imaging a celestial body, this might be different perspectives as the Earth orbits the Sun. In our case, it is translations of an object on a touchscreen’s surface that quantizes the object along many different capacitive pixel boundaries. Although single frames inherently contain no details smaller than a pixel, sub-pixel details can now be resolved when fused together through super-resolution. Fortunately, there is often sufficient sub-pixel movement created when a user naturally places an object down onto a touch surface. However, this does come at the cost of latency, as more than one frame is needed. For this reason, we envision a single-frame touch-tracking pipeline and multi-frame super-resolution pipeline running in parallel, maintaining existing touch input responsiveness while also opening new interactive opportunities.

Research Team: Sven Mayer, Xiangyu Xu, and Chris Harrison