Apple's AR/VR headset is said to not need a controller in order to navigate virtual and augmented reality, which is somewhat rare for modern head-mounted displays. Some older systems, such as Google Cardboard, put controls directly on the device, others ship with a game controller, which sets the expectation that it is a gaming device. Apple's mixed reality headset might be compatible with a controller but it likely won't be needed, based on what is known so far.
Apple CEO Tim Cook often describes augmented reality as one of the most important technologies of the future. While the iPhone maker has never publicly stated that an AR or VR headset is being developed, it doesn't require a leap of faith to recognize that Apple has invested a great amount of time into such a project. A quick check of the U.S. Patent & Trade Office's website reveals 142 documents that mention "augmented reality" and 237 for "virtual reality." Apple has been busily filling a great number of patent applications that reference head-mounted displays and describe the user interface of these advanced devices.
According to a patent application filed earlier this year, Apple's mixed reality headset will be able to recognize hand gestures, identify locations, and objects, allowing the device to be controlled by the user with no other devices necessary. In addition to a large number of patent documents, several leaks from reliable sources reinforce the idea that an Apple AR and VR wearable is coming in 2022. MacRumors shared a note from Apple analyst Ming-Chi Kuo recently, describing a system similar to that mentioned in the patent document. Kuo explained that the headset might contain sensors similar to the TrueDepth camera used in the iPhone to detect the user's hands and fingers, allowing the use of gestures to control the device and interact in a three-dimensional environment.
Apple VR Headset Hand Gestures
Apple's mixed reality headset won't need any controllers for use. However, it might be compatible with controllers and Apple does mention in the patent document that hand gestures could either replace or augment physical controllers. The user's gaze, calculated by using eye-tracking sensors, could serve to manipulate virtual controls and select virtual objects. Voice input also reduces the need for physical input devices. While it all sounds very futuristic, the various components of this technology already exist and simply need to be assembled and refined into a usable solution. Diving back into the finer details, the patent document describes clasping the hand and touching the thumbs to different parts of the fingers to represent various commands.
A virtual overlay could generate a user interface that aligns with the hand to provide a visual reference of where to tap with the thumb for a particular function. Sliding the thumb forward, back, left and right might control the movement of a virtual object, trigger a change in the user interface similar to a swipe on an iPhone, or provide controls for a game. Apple's 84-page patent document describes many more scenarios that allow a great amount of control of virtual objects and user interface elements as well as virtual representations of real objects that can be controlled simply by using a combination of hand gestures, voice control, and gaze-tracking. This makes it clear that a separate controller might be useful but not necessary with Apple's future AR/VR headset.