Recent developments in touch surface technologies have seen a proliferation of proposals for new interaction techniques. These include multi-touch actions for the direct manipulation of interface elements, as well as gestures for command invocation and multi-element selection. Indeed, multi-touch interaction affords many new and interesting avenues for expressive input and collaborative interaction. However, little attention has been paid to how mice could operate and supplement interaction in touch-driven systems. This is unfortunate given the well-established benefits of the mouse and its familiarity among users, with potentially thousands of hours of training.

We developed SurfaceMouse, a virtual mouse implementation for multi-touch surfaces. A key design objective was to leverage as much pre-existing knowledge (and potentially muscle memory) regarding mice as possible, making interactions immediately familiar. To invoke SurfaceMouse, a user simply places their hand on an interactive surface as if there was a mouse present (see images on right). The system recognizes this characteristic gesture and renders a virtual mouse under the hand, which can be used like a real mouse. In addition to two-dimensional movement (X and Y axes), our proof-of-concept implementation supports left and right clicking, as well as up/down scrolling. Importantly, SurfaceMouse does not preempt any existing touch interaction or multi-touch gestures. It simply acts as an additional gesture.