TWEND: Twisting and Bending as new Interaction Gesture in Mobile Devices

TWEND consists of a hardware protoype that enables users to perform bending gestures and a Mac OS X software framework that maps these gestures to system events. These events can be configured to mimick scrolling, keystrokes or Apple Script events. This enables a quick way to set up experiments to research the value of different bending gestures as input method for interactive tasks involving a computer system.
Since the software framework is extensible, even setups that can not be done with only the standard keystrokes, scrolling or Apple Script can be established.
Authors: Gero Herkenrath, Thorsten Karrer and Jan Borchers

CHI 2008 Student Research Competiton Runner-Up

This work was sent to the CHI 2008 Student Research Competition and won the second place. The paper is available here. Rumor has it that this means we will be invited to the ACM Grand Finals with this.

---

Motivation

Although the prototype framework still requires a desktop computer or at least a notebook, the goal of the project is to find new interaction methods for mobile devices. Traditional input methods suffer from certain problems in this context. First, there is a real estate problem between space used for input and space used for output. Screen sizes and number of input buttons are limited to keep the device small. Touchscreens counter this, but can lead to occlusion problems. Tilting introduces a new interaction style by facilitating not just the fingers, but sometimes it can be hard to correctly interpret the users intention with this (just try to read something on the iPhone while lying in your bed...).

A bendable device also makes use of the users wrists, i.e. just like tilting it makes use of an additional part of the dextrous system. Because the hands are already "in position" holding the device, no further space for input areas is needed.

Building the Prototype

Before the actual TWEND prototype was build, we constructed a smaller one to figure out whether our choice of sensors was feasible for our needs.
We used optical bending sensors to measure the amount of bending done to the device. These are embedded in a layer of foam between two pieces of plastic.

The Sensors

The sensors consist of an LED, a photocell and a fibre cable with an abraded section.
If bent, more or less light is lost at the abraded part of the cable which can be measured at the photocell.
The LEDs and the photocell are connected to seperate circuits to prevent cross influence.
More information on the sensors can be found in this paper: http://stacks.iop.org/MST/13/1523 (Note to readers from outside the RWTH network: This links to an online library you may not have full access to)

The Small Test Prototype

This prototype contains only one sensor and can thus only measure bending along one dimension.
Unlike the lower, the upper plastic plate is not glues to the layer of foam so bending and maintenance is easier.

The Final Prototype

These images show the larger, final prototype. It's 15 x 25 cm large and about 1 cm thick.
On the left you can see the insides, i.e. the sensors embedded in the foam and their wiring. There are eight sensors in total.
The right picture shows one of the possible gestures the prototype can recognize. This specific one we call a "wave". Note that it can only be recognized because the horizontal sensors in the prototype are actually devided in pairs of two instead of one long sensors at the upper and lower side of the device.

The Connection & the Software

The sensor measurement is done by one Bluetooth Arduino (see www.arduino.cc) that then streams the eight values of the sensors to a computer that runs the software doing the gesture recognition and system event generation. We use an Apple Macintosh so far, but aside from the parts of the code that generates the actual system events the software should run on any unix machine. However, we did not yet test that.

A First Try

As a first experiment to see how our prototype does and what users think of this new interaction technique, we configured it to control parts of the game "World of Warcraft". Participants were asked to navigate a character through the games 3D environment, one time with our prototype and one time with a standard keyboard. As expected they were faster with the keyboard, but the general feedback on the idea to use bending as interaction technique was very positive. Information on how to improve the prototype and its configuration in future experiments was also useful to us.

Next Steps

Of course a comparison with a standard keyboard is not our final goal, since as the title implies we see the future of bending gestures mainly in the mobile area. So we plan to do further comparisons, but this time with interaction techniques usually found on PDAs, cell phones etc. This will probably include touchpads, cellphone keyboards and PDA pens/pointing devices.
Since there are not yet many findings about how users perceive bending gestures as a control mechanism we probably will have to be careful how to do this comparison. Currently, our prototype relies on an external display and we'd have to eliminate this factor from any experiment with mobile devices, probably by connecting them to a larger display as well. The size of the device could also be a matter, but since we are mostly interested in some basic findings, we can overcome this problem by using more abstract forms of the usual mobile input techniques. A pointing device, e.g. could be replaced or mimicked by a tablet system, which would be about equal in size to our prototype.
A future goal, however, would be to have a smaller prototype with a mounted, flexible display that is really mobile. This will be challenging and propably cost a lot more, which is why we went with the current solution for now.