Handheld controllers, offering several buttons and six degree-of-freedom tracking, are the most common input approach seen in today’s augmented and virtual reality (AR/VR) systems (e.g., HTC Vive, Oculus Rift). Of course, there are many other facets that could be valuable to digitize, including user body pose, facial expression, skin tone and apparel. Unfortunately, very few AR/VR systems capture these dimensions, and when they do, it is most often via special worn sensors (e.g., instrumented gloves, additional cameras mounted on the headset). Alternatively, external infrastructure can be deployed (e.g., multiple room-mounted cameras) that capture body pose without having to instrument the user.

In this work, we take advantage of an emerging use case: co-located, multi-user AR/VR experiences. In such contexts, participants are often able to see each other’s bodies, hands, mouths, apparel, and other visual facets, even though they generally do not see themselves. Using the existing outwards-facing cameras on smartphones and AR/VR headsets (e.g., Microsoft Hololens, Google Cardboard), these visual dimensions can be opportunistically captured and digitized, and then relayed back to their respective users in real time. This is the key insight that motivated our work on BodySLAM.

Our system name was inspired by SLAM (simultaneous localization and mapping) approaches to mapping unknown environments. In these systems, many viewpoints are used to reconstruct the geometry of the environment and objects. In a similar vein, BodySLAM uses disparate camera views from many participants to reconstruct the geometric arrangement of bodies in the environment, as well digitize the individual bodies themselves (bodies, hands, mouths, skin and apparel). When a person is seen by two or more user, we can estimate 3D data.

We evaluated our system in a multi-part user study, incorporating two tasks and two group sizes. To explore how BodySLAM might scale to larger numbers of people, we ran software simulations in virtual rooms. Although we did not build any demo applications, we note that digitization of bodies has been well motivated in prior work, including uses in social VR and telepresence, entertainment and gaming, and 3D manipulation.

Where a person is looking is an important social cue in human-human interaction, allowing someone to address a particular person in conversation or denote an area of interest. For several decades, human-computer interaction researchers have looked at using gaze data to ease and enhance interactions with computing systems, ranging from social robots to smart environments. However, to capture gaze direction, special sensors must either be worn on the head (unlikely for consumer adoption) or external cameras are used (which can be privacy invasive).

In this research, we explored the use of speech as a directional communication modality. In addition to receiving and processing spoken content, we propose that devices also infer the Direction of Voice (DoV). Note this is different from Direction of Arrival (DoA) algorithms, which calculate from where a voice originated. In contrast, DoV calculates the direction along which a voice was projected.

Such DoV estimation innately enables voice commands with addressability, in a similar way to gaze, but without the need for cameras. This allows users to easily and naturally interact with diverse ecosystems of voice-enabled devices, whereas today’s voice interactions suffer from multi-device confusion. With DoV estimation providing a disambiguation mechanism, a user can speak to a particular device and have it respond; e.g., a user could ask their smartphone for the time, laptop to play music, smartspeaker for the weather, and TV to play a show. Another benefit of DoV estimation is the potential to dispense with wakewords (e.g., “Hey Siri”, “OK Google”) if devices are confident that they are the intended target for a command. This would also enable general commands – e.g., “up” – to be innately device-context specific (e.g., window blinds, thermostat, television).

Our approach relies on fundamental acoustic properties of both human speech and multipath effects in human environments. Our machine learning model leverages features derived from these phenomena to predict both angular direction of voice, and more coarsely, if a user is facing or not facing a device. Our software is lightweight, able to run on a wide variety of consumer devices without having to send audio to the cloud for processing, helping to preserve privacy.

Inertial Measurement Units (IMUs) with gyroscopic sensors are standard in today’s mobile devices. We show that these sensors can be co-opted for vibroacoustic data reception. Our approach, called VibroComm, requires direct physical contact to a transmitting (i.e., vibrating) surface. This makes interactions targeted and explicit in nature, making it well suited for contexts with many targets or requiring and intent. It also offers an orthogonal dimension of physical security to wireless technologies like Bluetooth and NFC. We achieve a transfer rate over 2000 bits/sec with less than 5% packet loss – an order of magnitude faster than prior IMU-based approaches at a quarter of the loss rate. Published at MobileHCI 2020.

Acoustic activity recognition has emerged as a foundational element for imbuing devices with context-driven capabilities, enabling richer, more assistive, and more accommodating computational experiences. Traditional approaches rely either on custom models trained in situ, or general models pre-trained on preexisting data, with each approach having accuracy and user burden implications. We present Listen Learner, a technique for activity recognition that gradually learns events specific to a deployed environment while minimizing user burden. More specifically, we built an end-to-end system for self-supervised learning of events labelled through one-shot voice interactions. Published at CHI 2020.

Wireality is a worn VR haptic system that allows for individual joints on the hands to be accurately arrested in 3D space through the use of retractable wires that can be locked. This allows for convincing tangible interactions with large and complex geometries, such as walls, furniture and railings. Our approach is lightweight (11g worn on the hands), low-cost (~$35) and low-power (0.024mWh per actuation).

Download Paper

Contemporary voice assistants require that objects of interest be specified in spoken commands. Of course, users are often looking directly at the object or place of interest – fine-grained, contextual information that is currently unused. We present WorldGaze, a software-only method for smartphones that provides the real-world gaze location of a user that voice agents can utilize for rapid, natural, and precise interactions. We achieve this by simultaneously opening the front and rear cameras of a smartphone. The front-facing camera is used to track the head in 3D, including estimating its direction vector. As the geometry of the front and back cameras are fixed and known, we can raycast the head vector into the 3D world scene as captured by the rear-facing camera. This allows the user to intuitively define an object or region of interest using their head gaze. We started our investigations with a qualitative exploration of competing methods, before developing a functional, real-time implementation. We conclude with an evaluation that shows WorldGaze can be quick and accurate, opening new multimodal gaze+voice interactions for mobile voice agents.

Download Paper

Augmented Reality allows for the overlay of digital information and interactive content onto scenes and objects. In order to provide tight registration of data onto objects in a scene, it is most common for markers to be employed, such as QR Codes and ArUco markers. LightAnchors enables spatially-anchored data in augmented reality applications without special hardware. Unlike most prior tracking methods, which instrument objects with markers, LightAnchors takes advantage of point lights already found in many objects and environments. For example, most electrical appliances now feature small (LED) status lights, and light bulbs are common in indoor and outdoor settings. In addition to leveraging point lights for in-view anchoring, we also co-opt these lights for data transmission, blinking them rapidly to encode binary data. This allows “dumb” devices to become smarter through AR with minimal extra cost. For example, a glue gun can transmit its live temperature and a security camera can broadcast its privacy policy.

Download Paper

Robust, wide-area sensing of human environments has been a long-standing research goal. In this project, we developed Sozu, a low-cost sensing system that can detect a wide range of events wirelessly, through walls and without line of sight, at whole-building scale. Instead of using batteries, Sozu tags convert energy from activities that they sense into RF broadcasts, acting like miniature self-powered radio stations. We describe the results from a series of iterative studies, culminating in a deployment study with 30 instrumented objects. Results show that Sozu is very accurate, with true positive event detection exceeding 99%, with almost no false positives. Beyond event detection, we show that Sozu can be extended to detect richer signals, such as the state, intensity, count, and rate of events. Source code of this project can be found here: https://github.com/FIGLAB/Sozu

Download Paper

Contemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. This overlooks the skin as a convenient surface for tactile, touch-driven interactions, which are generally more accurate and comfortable than free space interactions. In response, we developed ActiTouch, a wearable system that allows users to use their hands and arms as readily available touch input surfaces for AR and VR, opening a new interaction opportunity beyond conventional controllers and in-air gestures. We invented a powerful sensor fusion method which combines an electrical method with computer vision. This enables precise on-skin touch segmentations, which uniquely enables many fine-grained touch interactions such as scrolling and swiping.

Download Paper

Low-cost, smartphone-powered VR/AR headsets are becoming more popular. These basic devices – little more than plastic or cardboard shells – lack advanced features, such as controllers for the hands, limiting their interactive capability. Moreover, even high-end consumer headsets lack the ability to track the body and face. For this reason, interactive experiences like social VR are underdeveloped. We introduce MeCap, which enables commodity VR headsets to be augmented with powerful motion capture (“MoCap”) and user-sensing capabilities at very low cost (under $5). Using only a pair of hemi-spherical mirrors and the existing rear-facing camera of a smartphone, MeCap provides real-time estimates of a wearer’s 3D body pose, hand pose, facial expression, physical appearance and surrounding environment – capabilities which are either absent in contemporary VR/AR systems or which require specialized hardware and controllers. We evaluate the accuracy of each of our tracking features, the results of which show imminent feasibility.

Download Paper

IoT appliances are gaining consumer traction, from smart thermostats to smart speakers. These devices generally have limited user interfaces, most often small butons and touchscreens, or rely on voice control. Further, these devices know little about their surroundings – unaware of objects, people and activities happening around them. Consequently, interactions with these “smart” devices can be cumbersome and limited. We describe SurfaceSight, an approach that enriches IoT experiences with rich touch and object sensing, offering a complementary input channel and increased contextual awareness. For sensing, we incorporate LIDAR into the base of IoT devices, providing an expansive, ad hoc plane of sensing just above the surface on which devices rest. We can recognize and track a wide array of objects, including finger input and hand gestures. We can also track people and estimate which way they are facing. We evaluate the accuracy of these new capabilities and illustrate how they can be used to power novel and contextually-aware interactive experiences.

Download paper.

Providing university teachers with high-quality opportunities for professional development cannot happen without data about the classroom environment. Currently, the most effective mechanism is for an expert to observe one or more lectures and provide personalized formative feedback to the instructor. Of course, this is expensive and unscalable, and perhaps most critically, precludes a continuous learning feedback loop for the instructor. In this paper, we present the culmination of two years of research and development on EduSense, a comprehensive sensing system that produces a plethora of theoretically-motivated visual and audio features correlated with effective instruction, which could feed professional development tools in much the same way as a Fitbit sensor reports step count to an end user app. Although previous systems have demonstrated some of our features in isolation, EduSense is the first to unify them into a cohesive, real-time, in-the-wild evaluated, and practically-deployable system. Our two studies quantify where contemporary machine learning techniques are robust, and where they fall short, illuminating where future work remains to bring the vision of automated classroom analytics to reality.

Download Paper

Capturing fine-grained hand activity could make computational experiences more powerful and contextually aware. Indeed, philosopher Immanuel Kant argued, "the hand is the visible part of the brain." However, most prior work has focused on detecting whole-body activities, such as walking, running and bicycling. In this work, we explore the feasibility of sensing hand activities from commodity smart- watches, which are the most practical vehicle for achieving this vision. Our investigations started with a 50 participant, in-the-wild study, which captured hand activity labels over nearly 1000 worn hours. We then studied this data to scope our research goals and inform our technical approach. We conclude with a second, in-lab study that evaluates our classification stack, demonstrating 95.2% accuracy across 25 hand activities. Our work highlights an underutilized, yet highly complementary contextual channel that could un- lock a wide range of promising applications.

Download paper.

BeamBand is a wrist-worn system that uses ultrasonic beamforming for hand gesture sensing. Using an array of small transducers, arranged on the wrist, we can ensem-ble acoustic wavefronts to project acoustic energy at spec-ified angles and focal lengths. This allows us to interro-gate the surface geometry of the hand with inaudible sound in a raster-scan-like manner, from multiple view-points. We use the resulting, characteristic reflections to recognize hand pose at 8 FPS. In our user study, we found that BeamBand supports a six-class hand gesture set at 94.6% accuracy. Even across sessions, when the sensor is removed and reworn later, accuracy remains high: 89.4%. We describe our software and hardware, and future ave-nues for integration into devices such as smartwatches and VR controllers.

Download Paper.

Interferi is an on-body gesture sensing technique using acoustic interferometry. We use ultrasonic transducers resting on the skin to create acoustic interference patterns inside the wearer’s body, which interact with anatomical features in complex, yet characteristic ways. We focus on two areas of the body with great expressive power: the hands and face. For each, we built and tested a series of worn sensor configurations, which we used to identify useful transducer arrangements and machine learning fea-tures. We created final prototypes for the hand and face, which our study results show can support eleven- and nine-class gestures sets at 93.4% and 89.0% accuracy, re-spectively. We also evaluated our system in four continu-ous tracking tasks, including smile intensity and weight estimation, which never exceed 9.5% error. We believe these results show great promise and illuminate an inter-esting sensing technique for HCI applications.

Download Paper.

Low cost virtual reality headsets powered by smartphones are becoming ubiquitous. Their unique position on the user's face opens interesting opportunities for interactive sensing. We describe EyeSpyVR, a software-only eye sensing approach for smartphone-based VR, which uses a phone's front facing camera as a sensor and its display as a passive illuminator. Our proof-of-concept system, using a commodity smartphone, enables four sensing modalities: detecting when the VR headset is worn, detecting blinks, recognizing the wearer's identity, and coarse gaze tracking - features typically found in high-end or specialty VR headsets. We describe our implementation and results from a 70 participant user study.

Download Paper

Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. For example, a smart speaker sitting on a kitchen countertop cannot figure out if it is in a kitchen, let alone know what a user is doing in a kitchen—a missed opportunity. In this work, we describe a novel, real-time, sound-based activity recognition system. We start by taking an existing, state-of-the-art sound labeling model, which we then tune to classes of interest by drawing data from professional sound effect libraries traditionally used in the entertainment industry. These well-labeled and high-quality sounds are the perfect atomic unit for data augmentation, including amplitude, reverb, and mixing, allowing us to exponentially grow our tuning data in realistic ways. We quantify the performance of our approach across a range of environments and device categories and show that microphone-equipped computing devices already have the requisite capability to unlock real-time activity recognition comparable to human accuracy.

Download Paper

Smart and responsive environments rely on the ability to detect physical events, such as appliance use and human activities. Currently, to sense these types of events, one must either upgrade to "smart" appliances, or attach aftermarket sensors to existing objects. These approaches can be expensive, intrusive and inflexible. In this work, we present Vibrosight, a new approach to sense activities across entire rooms using long-range laser vibrometry. Unlike a microphone, our approach can sense physical vibrations at one specific point, making it robust to interference from other activities and noisy environments. This property enables detection of simultaneous activities, which has proven challenging in prior work. Through a series of evaluations, we show that Vibrosight can offer high accuracies at long range, allowing our sensor to be placed in an inconspicuous location. We also explore a range of additional uses, including data transmission, sensing user input and modes of appliance operation, and detecting human movement and activities on work surfaces. Source code of this project can be found here: https://github.com/FIGLAB/vibrosight

Download Paper

Human environments are typified by walls -- homes, offices, schools, museums, hospitals and pretty much every indoor context one can imagine has walls. In many cases, they make up a majority of readily accessible indoor surface area, and yet they are static -- their primary function is to be a wall, separating spaces and hiding infrastructure. We present Wall++, a low-cost sensing approach that allows walls to become a smart infrastructure. Instead of merely separating spaces, walls can now enhance rooms with sensing and interactivity. Our wall treatment and sensing hardware can track users' touch and gestures, as well as estimate body pose if they are close. By capturing airborne electromagnetic noise, we can also detect what appliances are active and where they are located. Through a series of evaluations, we demonstrate Wall++ can enable robust room-scale interactive and context-aware applications. Published at CHI 2018.

Download paper

Paper continues to be a versatile and indispensable material in the 21st century. Of course, paper is a passive medium with no inherent interactivity, precluding us from computationally-enhancing a wide variety of paper-based activities. In this work, we present a new technical approach for bringing the digital and paper worlds closer together, by enabling paper to track finger input and also drawn input with writing implements. Importantly, for paper to still be considered paper, our method had to be very low cost. This necessitated research into materials, fabrication methods and sensing techniques. We describe the outcome of our investigations and show that our method can be sufficiently low-cost and accurate to enable new interactive opportunities with this pervasive and venerable material. Published at CHI 2018.

Download paper

Compact, worn computers with projected, on-skin touch interfaces have been a long-standing yet elusive goal, largely written off as science fiction. Such devices offer the potential to mitigate the significant human input/output bottleneck inherent in worn devices with small screens. In this work, we present the first, fully-functional and self-contained projection smartwatch implementation, containing the requisite compute, power, projection and touch-sensing capabilities. Our watch offers roughly 40 cm² of interactive surface area – more than five times that of a typical smartwatch display. We demonstrate continuous 2D finger tracking with interactive, rectified graphics, transforming the arm into a touchscreen. We discuss our hardware and software implementation, as well as evaluation results regarding touch accuracy and projection visibility. Published at CHI 2018.

Download paper

Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this project, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.

Download Paper

The promise of smart environments and the Internet of Things (IoT) relies on robust sensing of diverse environmental facets. In this work, we explore the notion of general-purpose sensing, wherein a single, highly capable sensor can indirectly monitor a large context, without direct instrumentation of objects. Further, through what we call Synthetic Sensors, we can virtualize raw sensor data into actionable feeds, whilst simultaneously mitigating immediate privacy issues. We deployed our system across many months and environments, the results of which show the versatility, accuracy and potential of this approach.

Download Paper

Electrick is a low-cost and versatile sensing technique that enables touch input on a wide variety of objects and surfaces, whether small or large, flat or irregular. This is achieved by using electric field tomography in concert with an electrically conductive material, which can be easily and cheaply added to objects and surfaces. We show that our technique is compatible with commonplace manufacturing methods, such as spray/brush coating, vacuum forming, and casting/molding – enabling a wide range of possible uses and outputs. Published at CHI 2017.

Download paper

Homes, offices and many other environments will be increasingly saturated with connected, computational appliances, forming the “Internet of Things” (IoT). At present, most of these devices rely on mechanical inputs, webpages, or smartphone apps for control. We propose an approach where users simply tap a smartphone to an appliance to discover and rapidly utilize contextual functionality. To achieve this, our prototype smartphone recognizes physical contact with uninstrumented appliances, and summons appliance-specific interfaces and contextually relevant functionality. Published at CHI 2017.

Download paper

Many research systems have demonstrated that depth cameras, combined with projectors for output, can turn nearly any reasonably flat surface into an ad hoc, touch-sensitive display. However, even with the latest generation of depth cameras, it has been difficult to obtain sufficient sensing fidelity across a table-sized surface to get much beyond a proof-of-concept demonstration. In this research, we present DIRECT, a novel touch-tracking algorithm that merges depth and infrared imagery captured by a commodity sensor. Our results show that our technique boosts touch detection accuracy by 15% and reduces positional error by 55% compared to the next best-performing technique in the literature.

Download paper

We present CapCam, a novel technique that enables smartphones (and similar devices) to establish quick, ad-hoc connections with a host touchscreen device, simply by pressing a device to the screen’s surface. Pairing data, used to bootstrap a conventional wireless connection, is transmitted optically to the phone’s rear camera. This approach utilizes the near-ubiquitous rear camera on smart devices, making it applicable to a wide range of devices, both new and old. CapCam also tracks phones’ physical positions on the host capacitive touchscreen without any instrumentation, enabling a wide range of targeted and spatial interactions. 

Download paper

Smartwatches and wearables are unique in that they reside on the body, presenting great potential for always-available input and interaction. Additionally, their position on the wrist makes them ideal for capturing bio-acoustic signals. We developed a custom smartwatch kernel that boosts the sampling rate of a smartwatch’s existing accelerometer, enabling many new applications. For example, we can use bio-acoustic data to classify hand gestures such as flicks, claps, scratches, and taps. Bio-acoustic sensing can also detect the vibrations of grasped mechanical or motor-powered objects, enabling object recognition. Finally, we can generate structured vibrations using a transducer, and show that data can be transmitted through the human body.

Download paper.

AuraSense enables rich, around-device, smartwatch interactions using electric field sensing. To explore how this sensing approach could enhance smartwatch interactions, we considered different antenna configurations and how they could enable useful interaction modalities. We identified four configurations that can support six well-known modalities of particular interest and utility, including gestures above the watchface and touchscreen-like finger tracking on the skin. We quantify the feasibility of these input modalities in a series of user studies, which suggest that AuraSense can be low latency and robust across both users and environments. 

Download paper.

Electrical Impedance Tomography (EIT) can be used to detect hand gestures using an instrumented smartwatch (see Tomo), demonstrating great promise for non-invasive, high accuracy recognition of gestures for interactive control. In Tomo 2, we introduce a new system that offers improved sampling speed and resolution. This, in turn, enables superior interior reconstruction and gesture recognition. More importantly, we use our new system as a vehicle for experimentation – we compare two EIT sensing methods and three different electrode resolutions. Results from in-depth empirical evaluations and a user study shed light on the future feasibility of EIT for sensing human input.

Download paper.

SkinTrack enables continuous touch tracking on the skin. It consists of a ring, which emits a continuous high frequency AC signal, and a sensing wristband with multiple electrodes. SkinTrack measures phase differences to compute a 2D finger touch coordinate. Our approach is compact, non-invasive, low-cost and low-powered. We envision the technology being integrated into future smartwatches, supporting rich touch interactions beyond the confines of the small touchscreen.

Download paper.

We use speakers and microphones already present in a wide variety of devices to open new sensing opportunities. Our technique sweeps through a range of inaudible frequencies and measures the intensity of reflected sound to deduce information about the immediate environment, chiefly the materials and geometry of proximate surfaces. We offer several example uses, two of which we implemented as self-contained demos.

Download Paper

A new method that estimates a finger’s angle relative to the screen.  Our approach works in tandem with conventional multitouch finger tracking, offering two additional analog degrees of freedom for a single touch point. We prototyped our solution on two platforms— a smartphone and smartwatch—each fully self-contained and operating in real-time. 

Download Paper
More Info

Research conducted at:
Qeexo

A technique that uses existing, low-level touchscreen data, combined with machine learning classifiers, to provide real-time authentication and even identification of users. Our user study demonstrates twenty-participant authentication accuracies of 99.6%. For twenty-user identification, our software achieved 94.0% accuracy and 98.2% on groups of four, simulating family use.

Download Paper
More Info

By fusing gaze and gesture into a unified and fluid interaction modality, we can enable rapid, precise and expressive free-space interactions that mirror natural use. Although both approaches are independently poor for pointing tasks, combining them can achieve pointing performance superior to either method alone. This opens new interaction opportunities for gaze and gesture systems alike.

Download Paper
More Info

A sensing technology that allows a smartwatch to know what object the user is touching.  When the user operates an electrical or electro-mechanical object, the electro-magnetic signals (EM) propagate through the user.  These characteristic signals flow through the user and detected by the watch, which can be used for on-touch object detection.

Download Paper
More Info on Gierad's Webpage

In collaboration with:
Disney Research

Tomo is a wearable, low-cost system using Electrical Impedance Tomography (EIT) to recover the interior impedance geometry of a user’s arm.  We ultimately envision this technique being integrated into future smartwatches, allowing hand gestures and direct touch manipulation to work synergistically to support interactive tasks on small screens. 

Download Paper
More Info

A technique for 3D printing hair, fibers and bristles, by exploiting the stringing phenomena inherent in 3D printers using fused deposition modeling. This technique extends the capabilities of 3D printing in a new and interesting way, without requiring any new hardware

Download Paper
More Info

Zensors is a new sensing approach that fuses real-time human intelligence from online crowd workers with automatic approaches to provide robust, adaptive, and readily deployable intelligent sensors. With Zensors, users can go from question to live sensor feed in less than 60 seconds. Through our API, Zensors can enable a variety of rich end-user applications and moves us closer to the vision of responsive, intelligent environments. 

Download Paper
More Info at Gierad's Webpage

Sponsor:
Yahoo! InMind

Acoustruments are low-cost, passive, and powerless mechanisms, made from plastic, that can bring tangible functionality to handheld devices. The operational principles were inspired by wind instruments, which produce expressive musical output despite being simple in physical design. Through a structured exploration, we built an expansive vocabulary of design primitives, providing building blocks for the construction of tangible interfaces utilizing smartphonesʼ existing audio functionality (the speaker and microphone). 

Download Paper
More Info at Gierad's Webpage

In collaboration with:
Disney Research

Tiny projectors integrated into the smartwatch to render icons on the user’s skin. These icons can be made touch sensitive, significantly expanding the interactive region without increasing device size. Through a series of experiments, we show that these “skin buttons” can have high touch accuracy and recognizability, while being low cost and power-efficient.

Download Paper
More Info

Air+Touch is a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events. 

Download Paper
More Info

Toffee is a sensing approach that extends touch interaction beyond the small confines of a mobile device and onto ad hoc adjacent surfaces, most notably tabletops. This is achieved using a novel application of acoustic time differences of arrival (TDOA) correlation. This enables radial interactions in an area many times larger than a mobile device.

Download Paper
More Info

We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction. With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps. 

Download Paper
More Info

Commercialized by:
Qeexo

We propose using the face of a smartwatch as a multi-degree-of-freedom mechanical interface. This enables rich interaction without occluding the screen with fingers, and can operate in concert with touch interaction and physical buttons. We developed a series of example applications, many of which are cumbersome – or even impossible – on today’s smartwatch devices.

Download Paper
More Info

TapSense enables touchscreens to know how users are touching the screen - with finger tips, knuckles and nails, or even a passive stylus. TapSense moves beyond just counting the number of fingers on the screen, revolutionizing the way we interact with touch-enabled devices. By distinguishing between different parts of the hand, TapSense takes the pain out of performing these actions, making mobile devices faster and easier to use than ever.

Download Paper
More Info

Commercialized by:
Qeexo

TeslaTouch brings rich, dynamic physical feedback to otherwise flat, featureless touchscreens. The technology is based on the electrovibration principle, which can programmatically vary the electrostatic friction between fingers and a touch panel. When combined with an interactive graphical display, this approach enables touch experiences with rich textures and physical affordances.

Download Paper
More Info at Disney Research

In collaboration with:
Disney Research