Interacting with wearable computers is the most brutal UX challenge of all time. Today we explore the unforgiving hellscape that is controlling wearables.

Wearable computers have their own special Occam’s Razor. If a wearable is ever inaccessible or unreliable, it is completely worthless. If a wearable passes the test, a user can operate it “for free” as if it were part of his or her body. If not, it’s just a weak, clumsy smartphone.

Displays and earpieces passed this threshold years ago, but input is stuck. Google Glass and HoloLens have sophisticated displays and thoughtful UI’s, but their input devices are too poor for real-world usage. As you’ll see, the current cutting-edge input devices – gesture detection and voice recognition – are wholly impractical and neutralize the device. Luckily, we have some promising input tech that just may make wearables usable.

Input devices must be usable at all times.

I’m being extremely strict here. All input devices must be usable at all times. This nullifies gesture controls, handheld controllers, and voice recognition. Body sensors are rarely inaccessible, but often compromised.

Sunlight overwhelms HoloLens’ infrared cameras, so it’s uncontrollable outdoors. Indoors, you still need to have one empty hand, which needs to be in view. A similar weakness kills handheld controls like the pictured Twiddlers – not only do you need a free hand, you need your Twiddler hand free.

The worst offender is voice recognition. The best modern speech-to-text systems rely on an Internet connection, which isn’t dependable enough. Furthermore, background noise can interfere with recognition. Even with a perfect connection and quiet, the user still needs to speak to him or herself. “Often usable” is not good enough.

Wearable computers are body parts. Our ears don’t stop working because we’re underground, and our hands keep working when we’re looking over our shoulders. If you can’t count on the wearable to be there for you, you’re usually better off with a smartphone.

This affects body sensors like data gloves, but doesn’t completely hose them. A well-designed data glove still provides interaction if the gloved hand is full by falling back on finger and motion gestures. Android Wear translates crude wrist gestures into basic controls, which recovers a lot of usability when the touchscreen is inaccessible.

The most accessible input devices are eye trackers and invasive brain-computer interfaces. High-quality eye trackers match input with output – if you can look at the display, you can interact with it. After all, if you’re holding a smartphone, you have a free thumb to tap it. Invasive brain-computer interfaces that expose thought-to-text and virtual limb interfaces are the Holy Grail, but those are unlikely to arrive this century

The interface must be extremely efficient and reliable.

This is an unforgiving metric. You likely type quite quickly, touch and click accurately, and use shortcuts without thinking. You rarely need to repeat an action, and actions usually take only a few milliseconds. This is the bare minimum for a wearable.

Mouse- and scrolling-based interfaces, gestures, and touch-typing “floating keyboard” systems are simply too slow. The user only has a few milliseconds to write down that calendar event, send that reply, read the notes, or check that fact. By the time the interface is swiped and typed through, it’s too late.

Modern voice recognition suffers here – the odds of a mistype are just too high, and correcting mistakes is too difficult. The user must be able to say the command, completely, and expect it to work so well that he or she doesn’t even bother double-checking.

For a wearable to extend the user’s mind, it must be so fast, efficient, and reliable that it can be used subconsciously. If the user can effortlessly consult the computer at will, he or she gains enhanced cognition, memory, and empathy. If there’s a cost or risk associated with using the wearable, this is nullified; the wearable is just another device to be occasionally checked and usually ignored.

The original wearable computers used a command-line interface with a handheld controller. With practice, keyboard shortcuts and typed commands are perfectly reliable, even eyes-free. Typing speed is low compared to a two-handed keyboard, but stenographic macros, chords, and other shortcuts accelerate input. These struggle with GUI interaction, which made them unsuitable for this generation’s wearables.

Assuming perfect reliability, voice recognition should be extremely fast – English speakers average 150wpm and most commands can be completed in less than 20 words. The problem is latency – even though the actual command can be completed in a second or two, it often takes 10 to 30 seconds to upload, recognize, parse, and run the command. Like typing, voice must be coupled with another input for GUI interaction.

Eye trackers are extremely efficient and precise for on-screen interaction – the human eye can rapidly snap between targets and is expressive enough to aid interaction. The caveat is that free-form input like text and sculpting could fatigue the user.

Body sensors and data gloves should also perform very well, either mapped onto the visible interface or with functions assigned to gestures and motions. The UX paradigms of these devices are still in flux, held back by a history of poor and proprietary hardware. The upside is certainly there – muscle memory is powerful and can massively simplify interfaces.

Again, invasive BCI’s should be powerhouses, limited only by the user’s focus and practice. Hopefully this potential is actualized in the future.

The future of wearable input

I strongly believe that the primary input device for near-future wearable computers will be eye tracking with lightweight voice recognition and contextual awareness. While serious and professional users will add a glove or handheld device for rapid text entry, most users will simply fall back on a smartphone.

For instance, Christine leaves work and stands at a crowded bus stop to commute home. She activates her wearable by momentarily staring at the bus sign. The wearable recognizes her stare and wakes up, following her gaze to the bus sign, and offers some commands. She glances at the “schedules” command and sees that the next bus arrives in 20 minutes.

Then a text arrives. She blinks at the notification to open it, reads it, and glances at “reply”. A list of canned responses pops up. Instead, Christine traces out a brief reply with a finger, still carrying her groceries at her hip. She looks away and the system automatically sends the reply.

20 minutes is too long in a crowd, so she stares at a “hot spot” on the lens to enter an app switcher. She glances at “Actions”, then at a taxi app. A driver is dispatched and a map is displayed. She stares at her neighborhood on the map and it zooms in. She blinks at her house and her destination is set.

In the car, she receives an emergency email that a Web site is misbehaving. It’s too long to trace out an answer, so she blinks at “reply on phone” and then blinks at the link in the email. Using the wearable as a second display, she checks the page, identifies the problem, and sends a detailed reply the old-fashioned way.

Now, Christine wouldn’t have spoken out loud at the bus stop. She couldn’t raise her arms while carrying groceries. We didn’t even need to know whether she had Internet. The wearable was always available and usable. When she had her hands free, she could do the complex text entry on a phone and use the wearable for extra power.

If she had a Google Glass, HoloLens, or probably a Magic Leap, it would probably be at home. After the fourth or fifth time she needed the wearable and couldn’t use it, she just stopped trusting it and fell back on her phone.

Wearable input is a merciless hell, but it’s the only way forward.