How Instagram Saved 90% of Computing Power & Improved Video Quality
Ditching this key process gave Instagram users a better video experience
With 2.5 billion active users, Instagram is one of the most popular social media platforms in the world.
And video accounts for over 80% of its total traffic.
With those numbers, it's difficult to imagine how much computation time and resources it takes to upload, encode and publish videos from all those users.
But Instagram managed to reduce that time by 94% and also improve their video quality.
Here's how.
Estimated reading time: 4 minutes 52 seconds
The Process from Upload to Publish
Here are the typical steps that take place whenever a user uploads a video on Instagram:
Pre-processing: Enhance the video’s quality like color, sharpness, frame rate, etc.
Compression/Encoding: Reduce the file size
Packaging: Splitting it into smaller chunks for streaming
For this article, we will focus on the encoding and packaging steps.
Sidenote: Video Encoding
If you were to record a 10-second 1080 video on your phone without any compression, it would be around 1.7 GB.
That’s a lot!
To make it smaller your phone uses something called a codec, that compresses the video for storage using efficient algorithms.
So efficient that it will get the file size down to 35MB, but it's in a format that’s not designed to be read by humans.
To watch the encoded video, a codec needs to decompress the file to pixels that can be displayed on your screen.
The compression process is called encoding, and the decompression process is called decoding.
Codecs have improved over time so there are many of them out there. And they’re stored in most devices, cameras, phones, computers, etc.
Instagram generated two types of encodings on upload: Advanced Encoding (AV1), and Simple Encoding (H.264).
Advanced encoding produces videos that are small in size with great quality. These kind of videos only made up 15% of Instagram’s total watch time.
Simple encoding produces videos that work on older devices, but used a less efficient method of compression, meaning that videos are small with not great quality.
To make matters worse, simple encoding alone took up more than 80% of Instagram's computing resources.
Why Simple Encoding Is Such a Resource Hog
For Simple encoding, a video is actually encoded in two formats:
Adaptive bit rate (ABR): video quality will change based on the user's connection speed.
Progressive: video quality stays the same no matter the connection. This was for older versions of Instagram that don't support ABR.
Both ABR and progressive created multiple encodings of the same video in different resolutions and bit rates.
But for progressive, the video player will only play one encoded video.
While for ABR those videos are split into small 2-10 second chunks, and the video player will change which chunk is played based on the user’s internet speed.
Sidenote: Bit rate
When a video is encoded, it stores binary data (1s and 0s) for each frame of the video, the more information each frame has, the higher its bit rate.
If I recorded a video of a still pond the compression algorithm will notice that most pixels stay blue, and store them with less data to keep the pixels the same.
If I had a recording of a fast-flowing waterfall and the compression algorithm kept pixels the same, the video would look odd.
Since pixels change a lot between frames it needs to store more information in each frame.
Bit rate is measured in megabits per second (mbps) since this is how much data is sent to the video player.
On YouTube the average bitrate for a 1080 video is 8Mbps which is 1MB of transmitted data every second.
If you had to guess which specific process was taking up the most resources, you'd correctly guess adaptive bit rate.
This is not only due to creating multiple video files, but also because the additional packaging step involves complex algorithms to figure out how to seamlessly switch between different video qualities.
The Clever Fix
Usually, progressive encoding creates just one video file. But because Instagram was creating multiple files with the same codec as ABR (H.264).
They realized they could use the same files for progressive and ABR eliminating the need to create two sets of the same videos.
If you compare the image above to the previous image, you’ll see that 4 videos are now created during the encoding stage instead of 8.
The team were able to use the same progressive files for the packaging stage of ABR which wasn’t as efficient as before resulting in poorer compression.
But they did save a lot of resources.
Instagram claims the old ABR process took 86 seconds for a 23-second video.
But the new ABR process, just packaging, took 0.36 seconds, which is a whopping 99% reduction in processing time.
With this much reduction Instagram could dedicate more resources to the advanced encoding process, which meant more users could see higher quality videos. How?
Because simple encoding took longer in the old process and used more resources, there wasn’t enough to always create advanced videos.
With the new process, there was enough resource to run both types of encoding, meaning both can be published and more users would see higher quality videos.
This resulted in an increase in views of advanced encoded video from 15% to 48%.
Sidenote: Encoding vs Transcoding
This is an optional side note for the video experts among you.
The word transcoding isn't used in this article, but technically it should have been.
Encoding is the process of compressing an uncompressed video into a smaller format.
Transcoding is the process of changing a video from one encoded format to the same, or another format.
Because all devices (phones, cameras) have a codec, when a video is recorded it is automatically encoded.
So even before you upload a video to Instagram it is already encoded, and any further encoding is called transcoding.
But because the original article mostly uses the term encoding and it’s is such a catch-all term used in the industry, I decided to stick with it.
Wrapping Things Up
After reading this you may be thinking, how did the team not spot this obvious improvement?
Well, small issues on a small scale are often overlooked. Small issues on a large scale no longer remain small issues, and I guess that's what happened here.
Besides, Instagram was always a photo app that is now focusing more on video, so I assume it's a learning process for them too.
If you want to read more about their learnings, check out the Meta Engineering Blog.
But if you enjoyed this simplified version, be sure to subscribe.
PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 16 hours.