How is compositing done on iOS and macOS?

Compositing on iOS and macOS involves several key components and techniques, as discussed in various WWDC sessions. Here are the main points:

1. Compositor Services:

   • visionOS: Uses compositor services to blend rendered content with physical surroundings. The compositor services API is used to create a rendering session, and Metal APIs are used to render frames. The rendered content is then composited with the physical environment using ARKit for world and hand tracking (Render Metal with passthrough in visionOS).
   • CarPlay: The next generation of CarPlay uses a dedicated compositor to handle video streams from the iPhone and locally rendered UI. This compositor ensures seamless transitions and animations by synchronizing frames at a system level (Meet the next generation of CarPlay architecture).

2. Color and Depth Textures:

   • visionOS: The color pipeline functions in the P3 display color space for better consistency. Compositor services use both color and depth textures from the renderer for compositing operations. The depth texture is expected to be in reverse Z convention (Render Metal with passthrough in visionOS).

3. Rendering Techniques:

   • Metal APIs: Used extensively for rendering frames that are then composited. This includes handling pre-multiplied alpha values and ensuring correct color and depth values (Render Metal with passthrough in visionOS).
   • Projection View Matrix: In visionOS, a projection view matrix is composed using device anchors and ARKit APIs to ensure correct depth and perspective in the rendered content (Render Metal with passthrough in visionOS).

4. Synchronization:

   • CarPlay: Uses a low-latency channel called Uisync to synchronize UI elements across displays, ensuring smooth transitions and animations regardless of the rendering source (Meet the next generation of CarPlay architecture).

Relevant Sessions

• Render Metal with passthrough in visionOS
• Meet the next generation of CarPlay architecture

These sessions provide a comprehensive overview of how compositing is handled on iOS, macOS, and visionOS, leveraging Metal, ARKit, and compositor services to create seamless and immersive experiences.