I need to transform a simd_float4x4's position to screen space
Generated on 9/17/2024
1 search
To transform a simd_float4x4's position to screen space in visionOS, you need to work with the projection view matrix. This matrix is a combination of the projection matrix and the view matrix, which can be expanded into two transforms:
- Device from Origin: This transform brings the origin space to the device space.
- View from Device: This transform brings the device space to the view space.
The projection view matrix is calculated as the projection matrix times the inverse of origin from device, multiplied by device from view. You can acquire the device from view transform by calling CpView get transform in the compositor services API. This matrix brings the rendered view to the device space.
For better positioning of rendered content with real-world objects, you can use a scene-aware projection matrix, which combines camera intrinsics and real-time scene understanding factors. This is particularly useful for mixed immersion experiences.
For more detailed guidance, you can refer to the session Render Metal with passthrough in visionOS (09:52) where these concepts are discussed in depth.
Render Metal with passthrough in visionOS
Get ready to extend your Metal experiences for visionOS. Learn best practices for integrating your rendered content with people’s physical environments with passthrough. Find out how to position rendered content to match the physical world, reduce latency with trackable anchor prediction, and more.
Dive deep into volumes and immersive spaces
Discover powerful new ways to customize volumes and immersive spaces in visionOS. Learn to fine-tune how volumes resize and respond to people moving around them. Make volumes and immersive spaces interact through the power of coordinate conversions. Find out how to make your app react when people adjust immersion with the Digital Crown, and use a surrounding effect to dynamically customize the passthrough tint in your immersive space experience.
Bring your iOS or iPadOS game to visionOS
Discover how to transform your iOS or iPadOS game into a uniquely visionOS experience. Increase the immersion (and fun factor!) with a 3D frame or an immersive background. And invite players further into your world by adding depth to the window with stereoscopy or head tracking.