Robust activity detection is the foundation for context-aware computing, where systems can not only take commands from users, but also proactively adapt or respond to users’ tasks. Today, the closest we have come to achieving this vision commercially is with “smart” appliances (e.g., refrigerators, coffee machines). Another approach that has seen some commercial success are aftermarket “sensor tags” that can be affixed to existing objects to enable some level of smartness (e.g., Notion, Motion Cookies). Though direct physical contact yields high signal fidelity, it also limits sensor placement (possibly forcing the use of batteries) and typically means that many sensors must be deployed to monitor an entire room. These sensors can be costly per unit (often tens of dollars), potentially aesthetically obtrusive, need to be water and impact resistant, and require wireless communication infrastructure. We are also beginning to see camera-driven products with activity recognition capabilities (e.g., Matrix Sensor, Lighthouse), which have the significant benefit of being able to monitor a wide area without direct instrumentation, though simultaneously suffer from privacy implications, especially in the home.

In this work, we leverage vibrations, which have been shown to be a rich signal source for detecting a wide array of events. Indeed, almost all physical activities generate vibration as a byproduct, whether it be chopping vegetables, writing on a whiteboard, typing on a laptop, running on a treadmill, or even sitting and reading a book. Likewise, many devices and appliances produce characteristic vibrations (e.g., faucets, kitchen appliances, HVAC, power tools, and even electronics if they contain fans). In response, there has been significant prior work leveraging vibrations for activity detection, including sensors coupled to floors and walls, as well as plumbing, gas, and HVAC infrastructure.

In this paper, we describe our work on Vibrosight – a low-cost, vibration sensing approach that works at long distances, affording flexibility in placement. Users affix small, inexpensive, passive stickers to objects they wish to reveal to our system – surfaces and objects without tags are invisible to Vibrosight. By using a steerable mirror, we can direct our sensing to any point with line of sight. We use this ability to intensively scan a scene for tags, and then once found, rapidly cycle between tags to sense the vibrations of their host surfaces. We use this data to produce vibrational spectrograms for each object, which we feed to a machine learning pipeline for recognition. In our evaluation, we investigated sensing accuracy across 24 objects in four locations. Our system can detect activation at 98.4% accuracy, with a false positive rate of 0.7%. To underscore Vibrosight’s robustness to interference, most of our study data were collected with multiple active appliances. Overall, we believe this work illuminates a new sensing approach with unique strengths.

Research Team: Yang Zhang, Gierad Laput, and Chris Harrison

Additional media can be found on Yang Zhang's site.