What new media features are coming?
Generated on 8/1/2024
1 search
Apple introduced several new media features at WWDC 2024. Here are some of the highlights:
-
Spatial Media:
- Spatial Video and Photos: New APIs in AVFoundation and RealityKit allow for the creation of spatial videos and photos, enhancing media with stereoscopy and real-time 3D content (Build compelling spatial photo and video experiences).
- Multiview Video Playback: VisionOS now supports multiview video playback, enabling simultaneous video watching with features like spatial audio and 3D video (Explore multiview video playback in visionOS).
-
Media Performance Metrics:
- New Metrics in iOS 18: AVFoundation introduces new ways to gather media performance metrics, represented as events, to help triage issues like playback delays and stalling (Discover media performance metrics in AVFoundation).
-
HLS Interstitials:
- Integrated Timeline API: A new AVFoundation API called Integrated Timeline helps manage the timing and sequence of playback content, enhancing user navigation through interstitials (Enhance ad experiences with HLS interstitials).
-
Immersive Media Viewing:
- Custom Environments: Enhancements to media playback experiences in custom environments, including immersive playback and environment pickers (Enhance the immersion of media viewing in custom environments).
-
RealityKit Enhancements:
- Low-Level Mesh and Texture Access: New APIs for low-level mesh and texture access, enabling dynamic models and textures (Discover RealityKit APIs for iOS, macOS and visionOS).
- Animation System: New capabilities in the animation system, including creating animation timelines in Reality Composer Pro (Discover RealityKit APIs for iOS, macOS and visionOS).
These features collectively enhance the media experience across Apple's platforms, particularly with the integration of spatial computing and improved performance metrics.
Explore multiview video playback in visionOS
Learn how AVExperienceController can enable playback of multiple videos on Apple Vision Pro. Review best practices for adoption and explore great use cases, like viewing a sports broadcast from different angles or watching multiple games simultaneously. And discover how to design a compelling and intuitive multiview experience in your app.
Optimize for the spatial web
Discover how to make the most of visionOS capabilities on the web. Explore recent updates like improvements to selection highlighting, and the ability to present spatial photos and panorama images in fullscreen. Learn to take advantage of existing web standards for dictation and text-to-speech with WebSpeech, spatial soundscapes with WebAudio, and immersive experiences with WebXR.
Enhance ad experiences with HLS interstitials
Explore how HLS Interstitials can help you seamlessly insert advertisements into your HLS content. We’ll also show you how to use integrated timeline to tune your UI experience and build SharePlay for interstitials.
What’s new in USD and MaterialX
Explore updates to Universal Scene Description and MaterialX support on Apple platforms. Discover how these technologies provide a foundation for 3D content creation and delivery, and learn how they can help streamline your workflows for creating great spatial experiences. Learn about USD and MaterialX support in RealityKit and Storm, advancements in our system-provided tooling, and more.
Build compelling spatial photo and video experiences
Learn how to adopt spatial photos and videos in your apps. Explore the different types of stereoscopic media and find out how to capture spatial videos in your iOS app on iPhone 15 Pro. Discover the various ways to detect and present spatial media, including the new QuickLook Preview Application API in visionOS. And take a deep dive into the metadata and stereo concepts that make a photo or video spatial.
Platforms State of the Union
Discover the newest advancements on Apple platforms.
Enhance the immersion of media viewing in custom environments
Extend your media viewing experience using Reality Composer Pro components like Docking Region, Reverb, and Virtual Environment Probe. Find out how to further enhance immersion using Reflections, Tint Surroundings Effect, SharePlay, and the Immersive Environment Picker.
Discover RealityKit APIs for iOS, macOS and visionOS
Learn how new cross-platform APIs in RealityKit can help you build immersive apps for iOS, macOS, and visionOS. Check out the new hover effects, lights and shadows, and portal crossing features, and view them in action through real examples.
Discover media performance metrics in AVFoundation
Discover how you can monitor, analyze, and improve user experience with the new media performance APIs. Explore how to monitor AVPlayer performance for HLS assets using different AVMetricEvents, and learn how to use these metrics to understand and triage player performance issues.