How does AirPlay mirroring work?
One of the exciting features coming in iOS 5 is AirPlay mirroring which lets iPhone and iPad owners send sound and picture to an Apple TV wirelessly. But how does it actually work? With cleverness and magic.
Disclaimer: Outlandish Ideas has no insider information on this. Nor have we reverse engineered the protocol. This technical explanation is gathered from research, common sense and some informed guesswork.
My first thought when I saw a demo was that Apple had worked out a way to stream full screen uncompressed video over a wireless connection. But that’s clearly not the case. A competing technology, WirelessHD, does exactly that but it requires direct line of sight between sender and receiver and uses the 60GHz spectrum to achieve the necessary transmission rates of over 4Gbit/s.
AirPlay is more subtle than that. One clue comes from the fact that it is only possible to mirror displays between iPads, iPhones and (2nd generation) Apple TVs all of which run the same iOS operating system. That means they are capable of running exactly the same apps even if installing apps on Apple TVs is normally not permitted.
What you’re actually seeing in the video above (not ours, by the way) is the iPad desktop running on the Apple TV in parallel with the iPad. It’s not sending the video frames acros the air, it’s just sending the touch gestures and some extra stuff to keep the picture in sync. When you see it load a web page, it’s just sending the HTML over the air and rendering it on the Apple TV. When you run an app, it sends the app over the air and runs it on the Apple TV hardware. When you play a 3D game, it renders each frame independently on both the iPad and the Apple TV. When you watch an HD movie, it sends the compressed video stream, buffers for a few seconds, then plays it on both devices.
The exception to this is the camera app where the raw image data has to be sent over the air since it’s coming live from the iPad camera. And in the demo above (just over 1 minute in) there’s a very visible lag and the frame rate goes way down, as you would expect.
This is clever stuff. And it explains why AirPlay mirroring is not available on Mac laptops: the Apple TV is just not capable or running Mac OS applications.
It will be interesting to see what happens when there’s a mismatch between the processor speed of the iPad/iPhone and the Apple TV. Will it still be possible to synchronise the running of apps between an iPad 3, 4, 5 gen and the current 2nd generation Apple TV with its 1GHz A4 processor? And will the Apple TV eventually be upgraded to allow mirroring some (or all?) Mac OS applications? Time will tell.
What’s clear is that the AirPlay interaction model of sending content from a portable device to a TV is much better than the flawed idea of installing and running apps on your TV.
Comments
It’s an interesting take on how this works and I’ve yet to hear a better explanation.
I read a post on another site that said that all iOS devices still utilise the ‘oddball’ MJPEG codec which would allow anything on screen to be sent over as a 30fps stitched series of images.
For your system to work with say a 500MB app, would the iPad just send whatever it is processing over to the ATV2? I’m pretty sure it could process data faster than it could (in real time) send that same data over Wifi to the ATV2.
@Phil I too wonder what happens when attempting to mirror a very large app. I suspect it might send only the files currently being read by the app across the air and stall the iPad app until the files have fully transferred. Unfortunately I don’t have an iPad 2 with which to do the test.
Another possibility is that AirPlay Mirroring simply sends very low level GPU commands wirelessly.
GPU cards already work on server/client model (the GPU is the server, the CPU is the client). As long as data is going only one way (CPU to GPU, iPad 2 to Apple TV), as is usually the case when talking to a GPU, the transfer rate can be very fast. The moment the CPU needs to read something back from the GPU, the performance hit is very severe — you can observe this with AirPlay Mirroring too.
I dont think they ar sending the app from the ipad to the APple TV and then running it from there.
I am feeling that they are simply sending encrypted video frames tapping into the screen frame buffer. They might be really using some smart compressions and we can find this by sniffing the network packets over the air. The volume noticeably increases when run some high graphics content.
Not sure what hadware encoders they are using and what compression techniques they are using on the client and corresponding decoders on the apple TV to achive a latency around 130ms
Well done – this is the first intelligent explanation of why I cannot wirelessly mirror my macbook air desktop to an Apple TV2.
I guess its time to use an Ipad2 for my wireless presentations – that or buy some fancy transmitter / receiver