Creating interfaces in the world, where and when we need them, has been a persistent goal of research areas such as ubiquitous computing, augmented reality, and mobile computing. In this work, we describe the WorldKit system, which makes use of a paired depth camera and projector to make ordinary surfaces instantly interactive. Using this system, touch-based interactivity can, without prior calibration, be placed on nearly any unmodified surface literally with a wave of the hand, as can other new forms of sensed interaction. From a user perspective, such interfaces are easy enough to instantiate that they could, if desired, be recreated or modified “each time we sat down” by “painting” them next to us. From the programmer’s perspective, our system encapsulates these capabilities in a simple set of abstractions that make the creation of interfaces quick and easy. Further, it is extensible to new, custom interactors in a way that closely mimics conventional 2D graphical user interfaces, hiding much of the complexity of working in this new domain.

This work was generously supported by a Qualcomm Innovation Fellowship.

Research Team: Robert Xiao, Chris Harrison, and Scott E. Hudson