Meta Quests have always relied heavily on cameras to track headsets, controllers, and world locations around users, but developers don’t have the same privileged access to headset cameras. Earlier this year, Meta was able to experiment with developers to have direct access to the headset camera in private projects. Starting this week, developers can publish apps that take advantage of the new features.
This week’s Pass-Through Camera API update for Quests means that developers will be able to publish their apps to the Horizon store, where they directly access the front cameras of Quests 3 and 3. This opens the door to third-party applications where you can scan the world around you and understand more about it. For example, developers can add computer vision capabilities to track objects and people in a scene, and build maps of their environments for analysis and interaction.
For a long time, this was impossible because the meta was placed on what developers could and could not do with headset hardware. Despite the widespread availability of computer vision capabilities by developers on smartphones, Meta was hesitant to allow the same thing with headsets.
Previously, third-party apps could learn some information about the world around the user, such as the shape of the room and the objects within it, but this information was provided by the system so that the app could not directly see what the camera could see. This allowed developers to build mixed reality applications that were somewhat aware of the space around their users. However, it has made some use cases difficult or impossible. For example, track specific objects that a user holds.
Last year, Meta announced it would unlock direct access to the headset’s camera. In March, they began offering developers the experimental version of their functionality, allowing them to build apps that access the cameras in their headsets. However, they were not allowed to make these apps public.
The company also specifies the technical capabilities and performance of the cameras that developers can access in Quests 3 and 3.
- Image capture latency: 40-60ms
- GPU Overhead: ~1-2% per streamed camera
- Memory Overhead: ~45MB
- Data Rate: 30Hz
- Maximum resolution: 1280 x 960
- Internal data format YUV420
According to the meta, developers’ use of camera data on Quest is subject to the developer data usage policy, including the “Prohibit the use of user data” section. This prohibits the use of data such as “running, promoting or providing tools for monitoring” and “uniquely identifying devices or users unless permitted.”
