2017 (to be posted)
2016 (to be posted)
2015
We describe a new method that estimates a finger’s angle relative to the screen. The angular vector is described using two angles – altitude and azimuth – more colloquially referred to as pitch and yaw. Our approach works in tandem with conventional multitouch finger tracking, offering two additional analog degrees of freedom for a single touch point. Uniquely, our approach only needs data provided by commodity touchscreen devices, requiring no additional hardware or sensors. We prototyped our solution on two platforms – a smartphone and smartwatch – each fully self-contained and operating in real-time. We quantified the accuracy of our technique through a user study, and explored the feasibility of our approach through example applications and interactions. Published at ITS 2015.
User identification and differentiation have implications in many application domains, including security, personalization, and co-located multiuser systems. In response, dozens of approaches have been developed, from fingerprint and retinal scans, to hand gestures and RFID tags. We propose CapAuth, a technique that uses existing, low-level touchscreen data, combined with machine learning classifiers, to provide real-time authentication and even identification of users. As a proof-of-concept, we ran our software on an off-the-shelf Nexus 5 smartphone. Our user study demonstrates twenty-participant authentication accuracies of 99.6%. For twenty-user identification, our software achieved 94.0% accuracy and 98.2% on groups of four, simulating family use. Published at ITS 2015.
Gaze interaction is particularly well suited to rapid, coarse, absolute pointing, but lacks natural and expressive mechanisms to support modal actions. Conversely, free space hand gesturing is slow and imprecise for pointing, but has unparalleled strength in gesturing, which can be used to trigger a wide variety of interactive functions. Thus, these two modalities are highly complementary. By fusing gaze and gesture into a unified and fluid interaction modality, we can enable rapid, precise and expressive free-space interactions that mirror natural use. Moreover, although both approaches are independently poor for pointing tasks, combining them can achieve pointing performance superior to either method alone. This opens new interaction opportunities for gaze and gesture systems alike. Published at ICMI 2015.
Most everyday electrical and electromechanical objects emit small amounts of electromagnetic (EM) noise during regular operation. When a user makes physical contact with such an object, this EM signal propagates through the user, owing to the conductivity of the human body. By modifying a small, low-cost, software-defined radio, we can detect and classify these signals in real-time, enabling robust on-touch object detection. Unlike prior work, our approach requires no instrumentation of objects or the environment; our sensor is self-contained and can be worn unobtrusively on the body. We call our technique EM-Sense and built a proof-of-concept smartwatch implementation. Our studies show that discrimination between dozens of objects is feasible, independent of wearer, time and local environment. Published at UIST 2015.
Tomo is a wearable, low-cost system using Electrical Impedance Tomography (EIT) to recover the interior impedance geometry of a user’s arm. This is achieved by measuring the cross-sectional impedances between all pairs of eight electrodes resting on a user’s skin. Our approach is sufficiently compact and low-powered that we integrated the technology into a prototype wrist- and armband, which can monitor and classify hand gestures in real-time. We ultimately envision this technique being integrated into future smartwatches, allowing hand gestures and direct touch manipulation to work synergistically to support interactive tasks on small screens. Published at UIST 2015.
We introduce a technique for 3D printed hair, fibers and bristles, by exploiting the stringing phenomena inherent in 3D printers using fused deposition modeling. Our approach offers a range of design parameters for controlling the properties of single strands and also of hair bundles. We further detail a list of post-processing techniques for refining the behavior and appearance of printed strands. We provide several examples of output, demonstrating the immediate feasibility of our approach using a low cost, commodity printer. Overall, this technique extends the capabilities of 3D printing in a new and interesting way, without requiring any new hardware. Published at UIST 2015.
The promise of “smart” homes, workplaces, schools, and other environments has long been championed. Unattractive, however, has been the cost to run wires and install sensors. More critically, raw sensor data tends not to align with the types of questions humans wish to ask, e.g., do I need to restock my pantry? In response, we built Zensors, a new sensing approach that fuses real-time human intelligence from online crowd workers with automatic approaches to provide robust, adaptive, and readily deployable intelligent sensors. With Zensors, users can go from question to live sensor feed in less than 60 seconds. Through our API, Zensors can enable a variety of rich end-user applications and moves us closer to the vision of responsive, intelligent environments. Published at CHI 2015.
Acoustruments are low-cost, passive, and powerless mechanisms, made from plastic, that can bring rich, tangible functionality to handheld devices. Through a structured exploration, we identified an expansive vocabulary of design primitives, providing building blocks for the construction of tangible interfaces utilizing smartphones’ existing audio functionality. By combining design primitives, familiar physical mechanisms can all be constructed. On top of these, we can create end-user applications with rich, tangible interactive functionalities. Acoustruments adds a new method to the toolbox HCI practitioners and researchers can draw upon, while introducing a cheap and passive method for adding interactive controls to consumer products. Published at CHI 2015.
2014
Smartwatches are a promising new interactive platform, but their small size makes even basic actions cumbersome. Hence, there is a great need for approaches that expand the interactive envelope around smartwatches, allowing human input to escape the small physical confines of the device. We propose using tiny projectors integrated into the smart- watch to render icons on the user’s skin. These icons can be made touch sensitive, significantly expanding the interactive region without increasing device size. Through a series of experiments, we show that these “skin buttons” can have high touch accuracy and recognizability, while being low cost and power-efficient. Published at UIST 2014.
Air+Touch is a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events. For example, a user can draw a circle in the air and tap to trigger a context menu, do a finger 'high jump' between two touches to select a region of text, or drag and in-air ‘pigtail’ to copy text to the clipboard. Published at UIST 2014.
Toffee is a sensing approach that extends touch interaction beyond the small confines of a mobile device and onto ad hoc adjacent surfaces, most notably tabletops. This is achieved using a novel application of acoustic time differences of arrival (TDOA) correlation. Our approach requires only a hard tabletop and gravity – the latter acoustically couples mobile devices to surfaces. We conducted an evaluation, which shows that Toffee can accurately resolve the bearings of touch events (mean error of 4.3° with a laptop prototype). This enables radial interactions in an area many times larger than a mobile device; for example, virtual buttons that lie above, below and to the left and right. Published at MobileHCI 2014.
The space around the body provides a large interaction volume that can allow for 'big' interactions on 'small' mobile devices. Prior work has primarily focused on distributing information in the space around a user's body. We extend this by demonstrating three new types of around-body interaction: canvas, modal and context-aware. We also present a sensing solution that uses standard smartphone hardware: a phone's front camera, accelerometer and inertial measurement units. Published at MobileHCI 2014.
Research into on-body projected interfaces has primarily focused on the fundamental question of whether or not it was technologically possible. Although considerable work remains, these systems are no longer artifacts of science fiction — prototypes have been successfully demonstrated and tested on hundreds of people. Our aim in this work is to begin shifting the question away from how, and towards where. To better understand and explore this expansive design space, we employed a mixed-methods research process involving more than two thousand individuals. This started with high-resolution, but low-detail crowdsourced data. We then combined this with rich, expert interviews, exploring aspects ranging from aesthetics to kinesthetics. Published at DIS 2014.
We propose using the face of a smartwatch as a multi-degree-of-freedom mechanical interface. This enables rich interaction without occluding the screen with fingers, and can operate in concert with touch interaction and physical buttons. We built a proof-of-concept smartwatch that supports continuous 2D panning and twist, as well as binary tilt and click. To illustrate the potential of our approach, we developed a series of example applications, many of which are cumbersome – or even impossible – on today’s smartwatch devices. Published at CHI 2014.
The average person can skillfully manipulate a plethora of tools, from hammers to tweezers. However, despite this remarkable dexterity, gestures on today’s touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-finger pinch, four-finger swipe and similar. We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction. With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps. Published at CHI 2014.
Tablet computers are often called upon to emulate classical pen-and-paper input. However, touchscreens typically lack the means to distinguish between legitimate stylus and finger touches and touches with the palm or other parts of the hand. This forces users to rest their palms elsewhere or hover above the screen, resulting in ergonomic and usability problems. We present a probabilistic touch filtering approach that uses the temporal evolution of touch contacts to reject palms. Our system improves upon previous approaches, reducing accidental palm inputs to 0.016 per pen stroke, while correctly passing 98% of stylus inputs. Published at CHI 2014.
2013 (to be posted)
2012 (to be posted)
2011 (to be posted)
TapSense is an enhancement to touch interaction that allows conventional screens to identify how the finger is being used for input. Our system can recognize different finger locations – including the tip, pad, nail and knuckle – without the user having to wear any electronics. This opens several new and powerful interaction opportunities for touch input, especially in mobile devices, where input bandwidth is limited due to small screens and fat fingers. For example, a knuckle tap could serve as a “right click” for mobile device touch interaction, effectively doubling input bandwidth. Published at UIST 2011.
2010
TeslaTouch infuses finger-driven interfaces with physical feedback. The technology is based on the electrovibration principle, which can programmatically vary the electrostatic friction between fingers and a touch panel. Importantly, there are no moving parts, unlike most tactile feedback technologies, which typically use mechanical actuators. This allows for different fingers to feel different sensations. When combined with an interactive graphical display, TeslaTouch enables the design of a wide variety of interfaces that allow the user to feel virtual elements through touch. Published at UIST 2010.
Devices with significant computational power and capability can now be easily carried with us. These devices have tremendous potential to bring the power of information, creation, and communication to a wider audience and to more aspects of our lives. However, with this potential comes new challenges for interaction design. For example, we have yet to figure out a good way to miniaturize devices without simultaneously shrinking their interactive surface area. This has lead to diminutive screens, cramped keyboards, and tiny jog wheels - all of which diminishes usability and prevents us from realizing the full potential of mobile computing. Published in IEEE Computer Magazine.
Skinput is a technology that appropriates the human body for acoustic transmission, allowing the skin to be used as a finger input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body. We collect these signals using a novel array of sensors worn as an armband. This approach provides an always-available, naturally-portable, and on-body interactive surface. To illustrate the potential of our approach, we developed several proof-of-concept applications on top of our sensing and classification system. Published at CHI 2010.
Minput is a sensing and input method that enables intuitive and accurate interaction on very small devices – ones too small for practical touch screen use and with limited space to accommodate physical buttons. We achieve this by adding two, inexpensive and high-precision optical sensors (like those found in optical mice) to the underside of the device. This allows the entire device to be used as an input mechanism, instead of the screen, avoiding occlusion by fingers. In addition to x/y translation, our system also captures twisting motion, enabling many interesting interaction opportunities typically found in larger and far more complex systems. Published at CHI 2010.
Human perception of time is fluid, and can be manipulated in purposeful and productive ways. We propose and evaluate variations on two visual designs for progress bars that alter users’ perception of time passing, and “appear” faster when in fact they are not. In a series of direct comparison tests, we are able to rank how different augmentations compare to one another. We then show that these designs yield statistically significantly shorter perceived durations than conventional progress bars. Progress bars with animated ribbing that move backwards in a decelerating manner proved to have the strongest effect. We measure the effect of this particular progress bar design and show it reduces the perceived duration by 11%. Published at CHI 2010.
A cord, although simple in form, has many interesting physical affordances that make it powerful as an input device. Not only can a length of cord be grasped in different locations, but also pulled, twisted and bent — four distinct and expressive dimensions that could potentially act in concert. Such an input mechanism could be readily integrated into headphones, backpacks, and clothing. Once grasped in the hand, a cord can be used in an eyes-free manner to control mobile devices, which often feature small screens and cramped buttons. We built a proof-of-concept cord-based sensor, which senses three of the four input dimensions we propose. Published at CHI 2010.
Although network bandwidth has increased dramatically, high-resolution images often take several seconds to load, and considerably longer on mobile devices over wireless connections. Progressive image loading techniques allow for some visual content to be displayed prior to the whole file being downloaded. In this note, we present an empirical evaluation of popular progressive image loading methods, and derive one novel technique from our findings. Results suggest a spiral variation of bilinear interlacing can yield an improvement in content recognition time. Published at CHI 2010.
Mark Weiser envisioned a third wave of computing, one with “hundreds of wireless computers in every office,” which would come about as the cost of electronics fell. Two decades later, some in the Ubiquitous Computing community point to the pervasiveness of microprocessors as a realization of this dream. Without a doubt, many of the objects we interact with on a daily basis are digitally augmented – they contain microchips, buttons and even screens. But is this the one-to-many relationship of people–to-computers that Weiser envisioned? Published in IEEE Multimedia.
Whack Gestures seeks to provide a simple means to interact with devices with minimal attention from the user – in particular, without the use of fine motor skills or detailed visual attention (requirements found in nearly all conventional interaction techniques). For mobile devices, this could enable interaction without “getting it out,” grasping, or even glancing at the device. Users can simply interact with a device by striking it with open palm or heel of the hand. Published at TEI 2010.