It’s truly mind-blowing to see virtual objects floating before our eyes, but the magic and illusion start to break down once we try to manipulate those objects. Input has always been a tricky subject in mixed reality, either because we can’t see our actual hands or we can’t feel what we’re supposed to be touching, which is physically nothing. Until the perfect haptic feedback gloves become a reality, we have to make do with tricks and workarounds to make input less awkward and more convenient. That’s especially true with typing on air, and Apple is apparently using some special techniques to offer a more usable experience on the Vision Pro mixed reality headset.
Designer: Apple (via Brian Tong)
Apple’s first teaser for the Vision Pro headset and visionOS platform didn’t show typing of any sort. It focused, instead, on icons, windows, and menus, virtual 3D objects that are easier to interact with using hand gestures. Of course, sooner or later you will be faced with the need to input text, and the usual method of voice recognition won’t always cut it. visionOS, fortunately, does include a virtual floating keyboard like other VR systems, but the way you use it is quite special and, to some extent, ingenious.
For one, you can interact with the keyboard like you would any part of the Vision Pro’s interface, which is to look at the UI element to focus on it and then use hand gestures. In this case, pinching a letter is the equivalent of selecting it, just like what you’d do for menu items or icons in visionOS. It makes the gesture grammar consistent, but it’s also an awkward way to type.
You can also “peck” at the keys with your fingers, making you feel like you’re typing on air. The difference that the Vision Pro makes, however, is that it tricks your eyes into believing you’re actually pressing down on those keys. Thanks to Apple’s flavor of spatial computing, hovering your real-world finger on a virtual key makes that key glow, and tapping on it results in an animation that looks like the key is actually moving down, just like on a real keyboard. There’s also a haptic sound, similar to the clicking sound effect you’d normally hear on an iOS virtual keyboard, to complete that audiovisual illusion.
Of course, your fingers aren’t actually hitting anything physical, so there’s still a disconnect that will probably confuse your brain. The visual effect, which is really only possible thanks to spatial computing, is still an important step forward in helping our minds believe that there’s a “real” three-dimensional object, in this case, a keyboard, right in front of us. It’s not going to be the most efficient way to input text, but fortunately, you can connect a wireless keyboard to the Vision Pro and you’ll be able to see your actual hands typing away on it.
