But, LinkedIn has so much data that training these models takes up a huge amount of resources.

So the team found a way to make that process more efficient.

Here's how they did it.

Estimated reading time 5 minutes and 15 seconds.

Why Is Training So Resource Intensive?

Training a model involves giving it a large amount of data to learn from.

There are three steps that can be done for training:

• Pre-training: Teaching the model general language

• Fine-tuning: Specializing the model for specific tasks

• Alignment: Making sure the model behaves as intended, this step is optional

Pre-training is the most resource-intensive step, so we'll focus on that.

This involves giving the model large unstructured data like books, articles, and websites. The data is stored in text, JSON, or binary format.

Then, words are tokenized into a numerical representation. So a sentence like "I love programming" could be represented as [1, 23, 456]

These tokens are converted to embeddings, which capture words or phrases in a dense numerical format. This helps the model understand their meaning and relationship to other words.

After that, the training phase starts.

Here, the model learns patterns by predicting the next word in a sentence or guessing missing words. This is all done automatically (self-supervised).

The model has the answers, so it tries to predict the answer and then checks it against the correct sentence. If it's correct or incorrect, it adjusts its weights to reduce the chances of a wrong prediction (backpropagation).

A complete pass through all the data is called an epoch. But training usually stops after many epochs. At least one, but it could be up to 300.

To speed things up, predictions are made in parallel using lots of GPUs. But even so, training a model on gigabytes of data through many epochs could take days or weeks.

Sidenote: Model Weights

An LLM is made up of many nodes. These process data from inputs.

Nodes are organized into layers to process data in stages. A weight is the strength of the connection between nodes on different layers.

Weights are randomly assigned numbers at the start of training. But they change during training to produce better outputs.

For example, if you're training a model to recognize the positivity of the sentence, “Get lost”. The inputs would be the word embeddings, and the weight value could lean towards positivity. So, the higher the value, the more positive the output.

A node typically has more than one weight. So, all weight values are summed and then go through an activation function. This adds 'non-linearity' to the values, which helps the model learn complex patterns.

In LinkedIn’s case, they were experiencing performance bottlenecks during training, such as:

1. Heavy GPU memory access

2. Extra time and resources used per-operation

To address this, they built a library called Liger-Kernel.

Liger-Kernel to the Rescue

Let's jump into how Liger-Kernel addressed these bottlenecks. We'll start with GPU memory access.

The GPU has different memory types for different purposes during training.

Slower High Bandwidth Memory (HBM) stores datasets, weights, and other large data structures.

Faster Shared Memory (SRAM) stores frequently accessed data. This could be intermediate calculations such as attention scores.

Because the SRAM is small, data is regularly transferred between the HBM and SRAM. This can add latency and delay computation time.

Sidenote: Attention Scores

A way for the model to "pay attention" to the most relevant parts of the input data.

Let's take a look at these two sentences:

"Jason is much faster than Toby; he trains a lot."

"Jason is much faster than Toby, who trains a lot."

The sentences only have one different word, which completely changes their meaning. The 'trains a lot' in the first sentence is for Jason, and the second is for Toby. We humans can figure out this instantly, but a model can't.

What we see as words, models see as lots of numbers. So, attention scores exist to focus on the important parts.

In the case of the first sentence, "Jason", "he", "trains a lot", and the semicolon could have higher attention scores. This is because the semicolon indicates a new clause that refers to Jason. This means Jason is faster because he trains a lot.

For the second sentence. "Toby" and "who trains a lot" could have a higher attention score because "trains a lot" relates to Toby. This suggests Toby is the one who trains, but Jason is still faster.

Liger-Kernel is built upon a technique called FlashAttention. This can improve GPU performance by calculating things like attention scores and partial sums on the SRAM instead of the HBM.

This was a good first step. But, they further optimized GPUs by taking tasks that needed many GPUs and merging operations so tasks could run on one GPU.

This is one of the ways they dealt with the per-operation time bottleneck.

Let's explain it with an example.

Two operations are used to control the range of values during training:

1. RMSNorm: Root Mean Square Normalization. A technique used to normalize the size (magnitude) of activation outputs from a layer of nodes.

2. Scaling: Adjust the range or magnitude of input data to a common scale. This helps the learning process because scaling makes sure data is within a range that the model can learn from.

Typically, these operations are performed on different GPUs. But they can be merged together (operator fusion). How?

Training can be done with a framework like PyTorch or TensorFlow. Out of the box, PyTorch does something called eager execution. This means operations are executed immediately without a compilation step.

While it sounds good in theory, it has some performance issues. Since operations are executed one at a time (synchronously), this prevents parallel executions.

To address this, PyTorch introduced a feature called torch.compile. Which enables Just-In-Time (JIT) compilation. JIT compiles a models' computational graph to machine code for faster execution.

A computation graph represents the sequence of operations a model performs on its inputs to produce an output.

Part of JIT involves operator fusion for many operations, not just the two mentioned earlier. In some cases JIT can be 22 times faster than eager execution.

Both FlashAttention and operator fusion work in Liger-Kernel. These features addressed the bottlenecks LinkedIn had.

But Liger-Kernel is a Python library, and raw Python code cannot run on a GPU.

So they wrote it in a language called Triton.

Python on the GPU

Triton is an open source domain-specific language and compiler created by an OpenAI employee. It's designed to help write custom GPU kernels using a Python-based syntax.

Triton code complies to low-level GPU code that generates a new optimized GPU kernel.

LinkedIn wrote their own operator fusion in Triton for Liger-Kernel. But also wrote RMSNorm and other operations in Triton to take full advantage of the GPU.

So, how is Liger-Kernel deployed?

Usually training is done through a distributed setup. This involves many GPUs training different parts of the model.

There are many distributed setups, but we'll focus on Torch Distributed Elastic in AWS purely because AWS diagrams make sense to me.

This has a parameter server used to store and update parameters like weights. And a controller to manage training jobs based on the available resources.

Liger-Kernel will be added to the container image, which runs on a pod. Once the pod starts, Liger-Kernel code compiles then optimizes the GPU kernel before training begins.

With all these features and optimized post-training losses, which I won't go into in this article. LinkedIn's open-source library has achieved impressive performance.

Improved multi-GPU training throughput by 20%. A 3x reduction in end-to-end training time. Reduced memory usage by 60% and much more.

If you enjoyed this article, check out the original article from LinkedIn.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

It launched in 2012 and has grown to be one of the most popular dating platforms with around 75 million active users.

This growth has led to the development of many different tech-related services over the years. Some of these are external, so they’re open to the public and attackers.

These external services used different third-party solutions for security and routing requests. This, of course, made them very difficult to maintain.

So, the team at Tinder decided to build their own solution. A solution that would match their custom infrastructure.

Here's how they did it.

Estimated reading time: 4 minutes 48 seconds

Why Tinder Has External Services

Tinder has many external services. Some of these include;

• The Authentication service for login and session management

• Recommendations to suggest matches

• Messaging for communication between matched users

• Geolocation for location-based matching

• Media to handle all image and video uploads

And many more.

These are external because people can access them easily. They can view the API requests and see the data being received.

This also means that bad actors can also see this information. This isn't great for any app, but it's a massive problem for a dating app like Tinder.

Someone could track a user's exact location via the geolocation service. Access phone numbers, images, or other personal information, hijack accounts, and so on.

Not to mention, if a hacker got access to an external service, they could also get access to internal services.

So it's important the team at Tinder keep these external services secure.

The best way to do this is to use an API gateway which enables authorization and security.

The problem Tinder had was that they were using many different third-party API gateways. These used different tech stacks and were difficult to manage.

They needed a single API gateway solution that:

1. Would have a consistent use of session management across different services

2. Could be a framework teams could take and use to scale their application independently

3. Could be customized using a configuration file instead of writing code

4. Would integrate with their Envoy service mesh

Sidenote: Service Mesh

A service mesh is a piece of software that manages communication between services.

Service meshes are usually added to a Kubernetes cluster and provide features like:

• mTLS: Mutual TLS which makes sure that services verify their identities before they can talk

• Retries: Automatically retry a connection if a service is down

• Observability: Collect metrics and traces from a service

• Traffic splitting: Control what percentage of traffic goes to each version of a service if there are many versions

These features and more work without making any changes to the application code.

A service mesh works by adding two things. A sidecar proxy to each pod. This intercepts all the application network calls. And a control plane that adds and manages the proxies.

So, configuration is done via the control plane, which updates the proxies. But the proxies don't need to communicate with the control plane.

Popular service mesh proxies include Envoy, Linkerd Proxy, and Consul Connect Proxy. Popular control planes include Istio, Linkerd, and Consul.

In fact, Envoy is the most popular service mesh proxy with first class support for gRPC.

It's important to note the difference between a service and a pod in Kubernetes.

A pod runs the application code. A service is a virtual layer that can be applied to a single or many pods.

It provides a stable DNS name and IP address. This is used to make sure that internal or external apps can access a pod or pods, even if the pod gets replaced or restarted.

Although you don't need services to use a service mesh, they provide stable names for pods that can complement a service mesh.

The team at Tinder looked into existing solutions like Amazon AWS Gateway, Kong, Apigee, and Tyk.io, but none met all their requirements.

So they built their own solution called TAG, Tinder API Gateway.

How TAG Works

Before TAG can receive traffic, it needs to be configured with a list of routes.

A developer would first create a route as configuration file. This would be a YAML file that could include information like an API endpoint or route, as well as its service and filters.

Filters are bits of logic that can be applied to requests as they come in, and responses as they go out. There are three types of filters.

• Pre-Filters: applied to requests before they reach the service (e.g., converting HTTP to gRPC).

• Post filters: applied to responses after they leave the service (e.g., adding headers like location).

• Global filters: applied to all requests and responses. (e.g., request and response scanning).

These three types of filters have predefined logic inside of TAG, but it's also possible to add custom filters.

Sidenote: Request and Response Scanning

This is one of the ways TAG can prevent attacks on the system, and it does this very cleverly.

When a request or response is sent to TAG, an async event is sent to an event streaming platform. This event contains details like the type of request and endpoint being access.

This is async, so it doesn't block other processes.

The data is securely streamed to other applications using Amazon MSK (Managed Streaming for Apache Kafka).

These applications can check for unusual patterns, bots, or other attacks.

If TAG detects any issues, it can trigger some global filters, such as rate limiting, or just block the request.

When TAG gets an update to the route configuration or starts up, it does a few things.

It creates internal objects that represent each route. Associates the correct filters with each route object. Then, sets up rules for how each request should be matched to a service.

After that, TAG is ready to receive traffic. Here's a step-by-step, high-level overview of how it does this.

1. A client sends a HTTP request to the backend

2. The request hits TAG, which applies global filters

3. A route in TAG is matched to the request

4. TAG uses Envoy to discover which service should handle the request

5. TAG applies pre-filters, then forwards the request to the service

6. Once the service has responded, TAG applies post-filters

7. Then the response is sent back to the client

Teams at Tinder use TAG as a framework for building their own API gateways just by writing configuration files.

TAG is also used by other companies like Hinge, OkCupid, and PlentyOfFish.

Wrapping Things Up

If I'm being honest, I'm amazed to see how much effort went into building TAG for an app like Tinder. It's easy not to think much of something like a dating platform. How complicated can it be?

But doing the research for this article was a great insight into how problems at scale, no matter the app, can be really complicated to solve.

Check out the original article if you want more details about how Tinder's API gateway works.

And as usual, if you want the next article sent to your inbox as soon as it's released, go ahead and subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

Founded in 2013 by four US students. It has grown to over 19 thousand employees worldwide, with 550,000 restaurants in 2023, and made over 8 billion dollars in revenue that same year.

That’s a lot of growth in 10 years. But there’s more.

In 2022, DoorDash acquired Wolt Enterprises, taking the total number of countries it operates in from 6 to more than 30.

With so many restaurants, giving users an excellent search and recommendation experience is important.

So to do this, the team at DoorDash built a machine learning model that used Redis to store data.

But Redis wasn't coping well with the amount of reads to the data.

So here's how they improved it.

Estimated reading time: 4 minutes 51 seconds

Why Does DoorDash Use ML?

Not all online services use machine learning for their search and recommendations. So why does DoorDash?

The team used traditional methods in the past to suggest restaurants based on a user's location and preferences. Most likely using a search pipeline with Elasticsearch.

But this didn't have the level of personalization users have come to expect. The search and recommendations didn't update dynamically based on user behavior.

So, the team at DoorDash built a machine learning model to learn from its users and make better predictions.

But to do that, they would need to store a lot of data somewhere for fast and easy access. And that somewhere for DoorDash was Redis.

Sidenote: Redis

Redis (Remote dictionary server) is an in-memory data store. In-memory means data is read and modified from computer memory (RAM), not the disk. This makes it incredibly fast.

Redis reads 12x faster than MongoDB and 500x faster than Elasticsearch.

It stores data as key-value pairs where keys are always strings, and values can be any data type.

But, because Redis stores data in memory, all the data must be stored in RAM, which can get expensive for a lot of data. This also means if the server crashes, data not yet written to disk is lost.

Because of that, Redis is commonly used as a cache for data to be retrieved quickly. But, is often paired with other databases for long-term storage.

The team tried using different databases: Cassandra, CockroachDB and Scylla. But they settled on Redis for its performance and cost.

An ML model capable of the predictions DoorDash wanted would need to make tens of millions of reads per second.

As performant as Redis is, it wasn't able to handle that many reads out of the box.

So they needed to massively improve it.

Sidenote: ML Predictions

Why does a machine learning model need to make tens of millions of reads per second?

A machine learning model is essentially a program that finds patterns in data and uses them to make predictions.

So if someone types 'best-running shoes' into a model for recommendations. The model would search for data, like shoe ratings, user's purchase history, shoe specifications, etc.

These pieces of data are called features. This is the input data the model needs to analyze. Features start out as raw data, like shoe data from an application database.

It's then cleaned up and transformed into a format that the model can be trained on and used to make predictions.

This includes creating categories or buckets for data, combining buckets to make new data, and removing redundant data. Things that can help the model find patterns.

All this data is stored in a feature store.

A feature store itself contains two main components: offline and online stores.

Offline stores contains historical data used to train the model. Usually stored on disk-based databases.

Online stores contain the most current data from real-time events used for real-time predictions. This data is often streamed via CDC and stored in memory for quick access.

New Data from online storage is often transferred to offline storage so the model can be trained on it. This is called feature ingestion.

So, if a prediction needs to be made, the model will read the online feature store to get data.

If many predictions need to be made from different users that require lots of feature data, thousands or tens of thousands of reads could be made simultaneously.

How DoorDash Improved Redis

Without modifications, Redis can handle a few hundred thousand reads per second. Which is more than enough for the average company.

But for DoorDash to use it as its feature store, it needed to handle a few million reads per second, which it struggled with.

So to improve Redis, the team needed to make it use less memory and use the CPU more efficiently. These were some of the bottlenecks they encountered.

Let's go through how they did that.

The first thing they did was to use Redis Hashes.

Sidenote: Redis Hashes

Redis Hashes are a data structure that allows you to store many values with a single key.

By default, Redis uses strings to store values, which weren't designed for many related values.

But hashes are designed to do that. They are more memory efficient for storing many values because Redis can optimize them.

You could also use the HGET command to get a single value and HMGET to get multiple values.

Hashes alone reduced CPU usage by 82%. But there were more optimizations the team could make.

Next, they compressed feature names and values.

They compressed feature names with a fast hashing algorithm called xxHash.

Feature names were typically very long for human readability.

But they took up 27 bytes of memory. Putting that exact text through xxHash would reduce it to 32 bits.

Considering 27 bytes (B) is 216 bits (b), that's an 85% reduction in size. Doing this on a large scale reduced a lot of memory.

The team likely had a separate mapping or table that linked each feature name to the hashed feature name.

When it came to compressing feature values, they used a more complicated approach.

They first converted values to Protocol buffers (protobufs). A data format developed by Google to store and transmit data in a compact form. It is a way to convert structured data to a binary format and is heavily used in gRPC.

Then, they compressed the protobufs using Snappy. Another Google-developed library that focuses on speed over compression size.

Snappy doesn't have the highest compression ratio and doesn't have the lowest CPU usage. But it was chosen over other options because it could compress Redis hashes and decompress feature values well.

With all these changes, DoorDash saw a 62% reduction in overall memory usage, from 298 GB of RAM to 112GB.

And a 65% reduction in CPU use from 208 CPUs to 72 per 10 million reads per second.

That’s incredible.

Wrapping things up

If you thought the efforts of the DoorDash team weren't impressive enough, check this out.

They added CockroachDB to their feature store because Redis' memory costs were too high.

They used CockroachDB as an offline feature store and kept Redis as their online feature store. But that's a topic for another article.

As usual, if you liked this post and want more details, check out the original article.

And if you want the next article sent straight to your inbox, be sure to subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 22 hours.

Download the app and drag your files into the newly created folder. That's it; your files are in the cloud and can be accessed from anywhere.

It sounds like a simple idea, but back in 2007, when it was released, there wasn't anything like it.

Today, Dropbox has around 700 million users and stores over 550 billion files.

All these files need to be organized, backed up, and accessible from anywhere. Dropbox uses virtual servers for this. But they often got overloaded and sometimes crashed.

So, the team at Dropbox built a solution to manage server loads.

Here's how they did it.

Estimated reading time: 5 minutes 15 seconds

Why Dropbox Servers Were Overloaded

Before Dropbox grew in scale, they used a traditional system to balance load.

This likely used a round-robin algorithm with fixed weights.

So, a user or client would upload a file. The load balancer would forward the upload request to a server. Then, that server would upload the file and store it correctly.

Sidenote: Weighted Round Robin

A round-robin is a simple load-balancing algorithm. It works by cycling requests to different servers so they get an equal share of the load.

If there are three servers, A, B, C, and three requests come in. A gets the first, B gets the second, and C gets the third.

Weighted round robin is a level up from round robin. Each server is given a weight based on its processing power and capacity.

Static weights are assigned manually by a network admin. Dynamic weights are adjusted in real time by a load balancer.

The higher the weight, the more load the server gets.

So if A has a weight of 3, B has 2, C has 1, and there were 12 requests. A would get 6, B would get 4, and C would get 2.

But there was an issue with their traditional load balancing approach.

Dropbox had many virtual servers with vastly different hardware. This made it difficult to distribute the load evenly between them with static weights.

This difference in hardware could have been caused by Dropbox using more powerful servers as it grew.

They may have started with an average server. As it grew, the team acquired more powerful servers. As it grew more, they acquired even more powerful ones.

At the time, there was no off-the-shelf load-balancing solution that could help. Especially one that used a dynamic weighted round-robin with gRPC support.

So, they built their own, which they called Robinhood.

Sidenote: gRPC

Google Remote Procedure Call (gRPC) is a way for different programs to talk to each other. It's based on RPC, which allows a client to run a function on the server simply by calling it.

This is different from REST, which requires communication via a URL. REST also focuses on the resource being accessed instead of the action that needs to be taken.

But gRPC has many more differences between REST and regular RPC.

The biggest one is the use of protobufs. This file format developed by Google is used to store and send data.

It works by encoding structured data into a binary format for fast transmission. The recipient then decodes it back to structured data. This format is also much smaller than something like JSON.

Protobufs are what make gRPC fast, but also more difficult to set up since the client and server need to support it.

gRPC isn't supported natively by browsers. So, it's commonly used for internal server communication.

The Custom Load Balancer

The main component of RobinHood is the load balancing service or LBS. This manages how requests are distributed to different servers.

It does this by continuously collecting data from all the servers. It uses this data to figure out the average optimal resource usage for all the servers.

Each server is given a PID controller, a piece of code to help with resource regulation. This has an upper and lower server resource limit close to the average.

Say the average CPU limit is 70%. The upper limit could be 75%, and the lower limit could be 65%. If a server hits 75%, it is given fewer requests to deal with, and if it goes below 65%, it is given more.

This is how the LBS gives weights to each server. Because the LBS uses dynamic weights, a server that previously weighted 5 could become 1 if its resources go above the average.

In addition to the LBS, Robinhood had two other components: the proxy and the routing database.

The proxy sends server load data to the LBS via gRPC.

Why doesn't the LBS collect this itself? Well, the LBS is already doing a lot.

Imagine there could be thousands of servers. It would need to scale up just to collect metrics from all of them.

So, the proxy has the sole responsibility of collecting server data to reduce the load on the LBS.

The routing database stores server information. Things like weights generated by the LBS, IP addresses, hostname, etc.

Although the LBS stores some data in memory for quick access, an LBS itself can come in and out of existence; sometimes, it crashes and needs to restart.

The routing database keeps data for a long time, so new or existing LBS instances can access it.

Routing databases can either be Zookeeper or etcd based. The decision to choose one or the other may be to support legacy systems.

Sidenote: Zookeeper vs etcd

Both Zookeeper and etcd are what's called a distributed coordination service.

They are designed to be the central place where config and state data is stored in a distributed system.

They also make sure that each node in the system has the most up-to-date version of this data.

These services contain multiple servers and elect a single server, called a leader, that takes all the writes.

This server copies the data to other servers, which then distribute the data to the relevant clients. In this case, a client could be an LBS instance.

So, if a new LBS instance joins the cluster, it knows the exact state of all the servers and the average that needs to be achieved.

There are a few differences between Zookeeper and etcd.

After Dropbox deployed RobinHood to all their data centers, here is the difference it made.

The X axis shows date in MM/DD and the Y axis shows the ratio of CPU usage compared to the average. So, a value of 1.5 means CPU usage was 1.5 times higher than the average.

You can see that at the start, 95% of CPUs were operating at around 1.17 above the average.

It takes a few days for RobinHood to regulate everything, but after 11/01, the usage is stabilized, and most CPUs are operating at the average.

This shows a massive reduction in CPU workload, which indicates a better-balanced load.

In fact, after using Robinhood in production for a few years, the team at Dropbox has been able to reduce their server size by 25%. This massively reduced their costs.

It isn't stated that Dropbox saved millions annually from this change. But, based on the cost and resource savings they mentioned from implementing Robinhood, as well as their size.

It can be inferred that they saved a lot of money, most likely millions from this change.

Wrapping Things Up

It's amazing everything that goes on behind the scenes when someone uploads a file to Dropbox. I will never look at the app in the same way again.

I hope you enjoyed reading this as much as I enjoyed writing it. If you want more details, you can check out the original article.

And as usual, be sure to subscribe to get the next article sent straight to your inbox.

Newsletter Spotlight

If you want to become a more productive tech professional, check out the Tech Productivity newsletter.

It’s full of links and tools you wish you had known about earlier.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

Yes, there are lots of other payment platforms like PayPal and Square. But what makes Stripe so popular is its developer-friendly approach.

It can be set up with just a few lines of code, has excellent documentation and support for lots of programming languages.

Stripe is now used on 2.84 million sites and processed over $1 trillion in total payments in 2023. Wow.

But what makes this more impressive is they were able to process all these payments with virtually no downtime.

Here's how they did it.

Estimated reading time: 4 minutes 58 seconds.

The Resilient Database

When Stripe was starting out, they chose MongoDB because they found it easier to use than a relational database.

But as Stripe began to process large amounts of payments. They needed a solution that could scale with zero downtime during migrations.

MongoDB already has a solution for data at scale which involves sharding. But this wasn't enough for Stripe's needs.

Sidenote: MongoDB Sharding

Sharding is the process of splitting a large database into smaller ones. This means all the demand is spread across smaller databases.

Let's explain how MongoDB does sharding. Imagine we have a database or collection for users.

Each document has fields like userID, name, email, and transactions.

Before sharding takes place, a developer must choose a shard key. This is a field that MongoDB uses to figure out how the data will be split up. In this case, userID is a good shard key.

If userID is sequential, we could say users 1-100 will be divided into a chunk. Then, 101-200 will be divided into another chunk, and so on. The max chunk size is 128MB.

From there, chunks are distributed into shards, a small piece of a larger collection.

MongoDB creates a replication set for each shard. This means each shard is duplicated at least once in case one fails. So, there will be a primary shard and at least one secondary shard.

It also creates something called a Mongos instance, which is a query router. So, if an application wants to read or write data, the instance will route the query to the correct shard.

A Mongos instance works with a config server, which keeps all the metadata about the shards. Metadata includes how many shards there are, which chunks are in which shard, and other data.

Stripe wanted more control over all this data movement or migrations. They also wanted to focus on the reliability of their APIs.

So, the team built their own database infrastructure called DocDB on top of MongoDB.

MongoDB managed how data was stored, retrieved, and organized. While DocDB handled sharding, data distribution, and data migrations.

Here is a high-level overview of how it works.

Aside from a few things the process is similar to MongoDB's. One difference is that all the services are written in Go to help with reliability and scalability.

Another difference is the addition of a CDC. We'll talk about that in the next section.

The Data Movement Platform

The Data Movement Platform is what Stripe calls the 'heart' of DocDB. It's the system that enables zero downtime when chunks are moved between shards.

But why is Stripe moving so much data around?

DocDB tries to keep a defined data range in one shard, like userIDs between 1-100. Each chunk has a max size limit, which is unknown but likely 128MB.

So if data grows in size, new chunks need to be created, and the extra data needs to be moved into them.

Not to mention, if someone wants to change the shard key for a more even data distribution. Then, a lot of data would need to be moved.

This gets really complex if you take into account that data in a specific shard might depend on data from other shards.

For example, if user data contains transaction IDs. And these IDs link to data in another collection.

If a transaction gets deleted or moved, then chunks in different shards need to change.

These are the kinds of things the Data Movement Platform was created for.

Here is how a chunk would be moved from Shard A to Shard B.

1. Register the intent. Tell Shard B that it's getting a chunk of data from Shard A.

2. Build indexes on Shard B based on the data that will be imported. An index is a small amount of data that acts as a reference. Like the contents page in a book. This helps the data move quickly.

3. Take a snapshot. A copy or snapshot of the data is taken at a specific time, we'll call this T.

4. Import snapshot data. The data is transferred from the snapshot to Shard B. But during the transfer, the chunk on Shard A can accept new data. Remember, this is a zero-downtime migration.

5. Async replication. After data has been transferred from the snapshot, all the new or changed data on Shard A after T is written to Shard B.

But how does the system know what changes have taken place? This is where the CDC comes in.

Sidenote: CDC

Change Data Capture, or CDC, is a technique that is used to capture changes made to data. It's especially useful for updating different systems in real-time.

So when data changes, a message containing before and after the change is sent to an event streaming platform, like Apache Kafka. Anything subscribed to that message will be updated.

In the case of MongoDB, changes made to a shard are stored in a special collection called the Operation Log or Oplog. So when something changes, the Oplog sends that record to the CDC.

Different shards can subscribe to a piece of data and get notified when it's updated. This means they can update their data accordingly.

Stripe went the extra mile and stored all CDC messages in Amazon S3 for long term storage.

6. Point-in-time snapshots. These are taken throughout the async replication step. They compare updates on Shard A with the ones on Shard B to check they are correct.

Yes, writes are still being made to Shard A so Shard B will always be behind.

7. The traffic switch. Shard A stops being updated while the final changes are transferred. Then, traffic is switched, so new reads and writes are made on Shard B.

This process takes less than two seconds. So, new writes made to Shard A will fail initially, but will always work after a retry.

8. Delete moved chunk. After migration is complete, the chunk from Shard A is deleted, and metadata is updated.

Wrapping Things Up

This has to be the most complicated database system I have ever seen.

It took a lot of research to fully understand it myself. Although I'm sure I'm missing out some juicy details.

If you're interested in what I missed, please feel free to run through the original article.

And as usual, if you enjoy reading about how big tech companies solve big issues, go ahead and subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

That's almost as much as the population of Singapore.

But with so many stores, it's essential to ensure they feel quick to navigate through and don't go down.

So, the team at Shopify created a system from scratch to monitor their infrastructure.

Here's exactly how they did it.

Estimated reading time: 4 minutes 53 seconds

Shopify's Bespoke System

Shopify didn't always have its own system. Before 2021, it used different third-party services for logs, metrics, and traces.

But as it scaled, things started to get very expensive. The team also struggled to collect and share data across the different tools.

So they decided to build their own observability tool, which they called Observe.

As you can imagine, a lot of work from many different teams went into building the backend of Observe. But the UI was actually built on top of Grafana.

Sidenote: Grafana

Grafana is an open-source observability tool. It focuses on visualizing data from different sources using interactive dashboards.

Say you have a web application that stores its log data in a database. You give Grafana access to the data and create a dashboard to visually understand the data.

Of course, you would have to host Grafana yourself to share the dashboard. That's the advantage, or disadvantage, of open-source software.

Although Grafana is open-source, it allows users to extend its functionality with plugins. This works without needing to change the core Grafana code.

This is how Shopify was able to build Observe on top of it. And use its visualization ability to display their graphs.

Observe is a tool for monitoring and observability. This article will focus on the metrics part.

Although it has 5.6 million active stores, at most, Shopify collects metrics from 1 million endpoints. An endpoint is a component that can be monitored, like a server or container. Let me explain.

Like many large-scale applications, Shopify runs on a distributed cloud infrastructure. This means it uses servers in many locations around the world. This makes the service fast and reliable for all users.

The infrastructure also scales based on traffic. So if there are many visits to Shopify, more servers get added automatically.

All 5.6 million stores share this same infrastructure.

Shopify usually has around a hundred thousand monitored endpoints. But this could grow up to one million at peak times. Considering a regular company would have around 100 monitored endpoints, 1 million is incredibly high.

Even after building Observe the team struggled to handle this many endpoints.

More Metrics, More Problems

The Shopify team used an architecture for collecting metrics that was pretty standard.

Kubernetes to manage their applications and Prometheus to collect metrics.

In the world of Prometheus, a monitored endpoint is called a target. And In the world of Kubernetes, a server runs in a container that runs within a pod.

Sidenote: Prometheus

Prometheus is an open-source, metrics-based monitoring system.

It works by scraping or pulling metrics data from an application instead of the application pushing or giving data to Prometheus.

To use Prometheus on a server, you'll need to use a metrics exporter like prom-client for Node.

This will collect metrics like memory and CPU usage and store them in memory on the application server.

The Prometheus server pulls the in-memory metrics data every 30 seconds and stores it in a time series database (TSDB).

From there, you can view the metrics data using the Prometheus web UI or a third-party visualization tool like Grafana.

There are two ways to run Prometheus: server mode and agent mode.

Server mode is the mode explained above that has the Prometheus server, database, and web UI.

Agent mode is designed to collect and forward the metrics to any storage solution. So a developer can choose any storage solution that accepts Prometheus metrics.

The team had many Prometheus agent pods in a replication set. A replication set makes sure a specific number of pods are running at any given time.

Each Prometheus agent would be assigned a percentage of total targets. They use the Kubernetes API to check which targets are assigned to them.

Then search through all the targets to find theirs.

You can already see what kind of problems would arise with this approach when it comes to scaling.

1. Lots of new targets could cause an agent to run out of memory and crash.

2. Distributing targets by percentage is uneven. One target could be a huge application with 100 metrics to track. While another could be small and have just 4.

But these are nothing compared to the big issue the team discovered.

Around 50% of an agent's resources were being used just to discover targets.

Each agent had to go through up to 1 million targets to find the ones assigned to them. So, each pod is doing the exact same piece of work which is wasteful.

To fix this, the team had to destroy and rebuild Prometheus.

Breaking Things Down

Since discovery was taking up most of the resources, they removed it from the agents. How?

They went through all the code for a Prometheus agent. Took out the code related to discovery and put it in its own service.

But they didn't stop there.

They gave these discovery services the ability to scrape all targets every two minutes.

This was to check exactly how many metrics targets had so they could be shared evenly.

They also built an operator service. This managed the Prometheus agents and received scraped data from discovery pods.

The operator will check if an agent has the capacity to handle the targets; if it did, it will distribute them. If not, it will create a new agent.

These changes alone reduced resource usage by 33%. A good improvement, but they did better.

The team had many discovery pods to distribute the load and for the process to keep running if one pod crashed. But they realized each pod was still going through all the targets.

So they reduced it to just one pod but also added what they called discovery workers. These were responsible for scraping targets.

The discovery pod will discover targets then put the target in a queue to be scraped. The workers pick a target from the queue and scrape its metrics.

The worker then sends the data to the discovery pod, which then sends it to the operator.

Of course, the number of workers could be scaled up or down as needed.

The workers could also filter out unhealthy targets. These are targets that are unreachable or do not respond to scrape requests.

This further change reduced resource use by a whopping 75%.

Wrapping Things Up

This is a common pattern I see when it comes to solving issues at scale. Break things down to their basic pieces, then build them back up.

All the information from this post was from a series of internal YouTube videos about Observe that were made public. I'm glad Shopify did this so others can learn from it.

Of course, there is more information in this video than what this article provides, so please check it out.

And if you want the next Hacking Scale article sent straight to your inbox, go ahead and subscribe. You won't be disappointed.

One final thing. If you’re thinking of self-hosting Prometheus, it’s actually cheaper and easier to use Better Stack. Don’t believe me? Check this out.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

Reading this email in gmail? Drag it to the Primary tab and never miss the next one.

It stores almost 19 petabytes of data.

You can store 3 billion songs with one petabyte, so we're talking about a lot of data.

And much of that data is unreachable; it's just taking up space unnecessarily.

But with some clever engineering, GitHub was able to fix that and reduce the size of specific projects by more than 90%.

Here's how they did it.

Estimated reading time: 4 minutes 25 seconds.

Why GitHub has Unreachable Data

The Git in GitHub comes from the name of a version control system called Git, which was created by the founder of Linux.

It works by tracking changes to files in a project over time using different methods.

A developer typically installs Git on their local machine. Then, they push their code to GitHub, which has a custom implementation of Git on its servers.

Although Git and GitHub are different products, the GitHub team adds features to Git from time to time.

So, how does it track changes? Well, every piece of data Git tracks is stored as an object.

Sidenote: Git Objects and Branches

A Git object is something Git uses to keep track of a repository's content over time.

There are three main types of objects in Git.

1. BLOB -  Binary large object. This is what stores the contents of a file, not the filename, location, or any other metadata.

2. Tree - How Git represents directories. A tree lists blobs and other trees that exist in a directory.

3. Commit - A snapshot of the files (blobs) and directories (trees) at a point in time. It also contains a parent commit, a hash of the previous commit.

A developer manually creates a commit containing hashes of just the blobs and trees that have changed.

Commit names are difficult for humans to remember, so this is where branches come in.

A branch is just a named reference to a commit, like a label. The default branch is called main or master, and it points to the most recent commit.

If a new branch is created, it will also point to the most recent commit. But if a new commit is made on the new branch, that commit will not exist on main.

This is useful for working on a feature without affecting the main branch.

Based on how Git keeps track of a project, it is possible to do things that will make objects unreachable.

Here are three different ways this could happen:

1. Deleting a branch: Deleting doesn't immediately remove it but removes the reference to it.

Reference is like a signpost to the branch. So the objects in the deleted branch still exist.

2. Force pushing. This replaces a remote branch's commit history with a local branch's history.

A remote branch could be a branch on GitHub, for example. This means the old commits lose their reference.

3. Removing sensitive data. Sensitive data usually exists in many commits. Removing the data from all those commits creates lots of new hashes. This makes those original commits unreachable.

There are many other ways to make unreachable objects, but these are the most common.

Usually, unreachable objects aren't a big deal. They typically get removed with Git's garbage collection.

Sidenote: Git's Garbage Collection

Garbage collection exists to remove unreachable objects.

It can be triggered manually using the git gc command. But it also happens automatically during operations like git commit, git rebase, and git merge.

Git only removes an object if it's old enough to be considered safe for deletion. This is typically 2 weeks. In case a developer accidentally deletes objects and they need to be retrieved.

Objects that are too recent to be removed are kept in Git's objects folder. These are known as loose objects.

Garbage collection also compresses loose, reachable objects into packfiles. These have a .pack extension.

Like most files, packfiles have a single modification time (mtime). This means the mtime of individual objects in a packfile would not be known until it’s uncompressed.

Unreachable loose objects are not added to packfiles. They are left loose to expose their modification time.

But garbage collection isn't great with large projects. This is because large projects can create a lot of loose, unreachable objects, which take up a lot of storage space.

To solve this, the team at GitHub introduced something called Cruft Packs.

Cruft Packs to the Rescue

Cruft packs, as you might have guessed, are a way to compress loose, unreachable objects.

The name "cruft" comes from software development. It refers to outdated and unnecessary data that accumulates over time.

What makes cruft packs different from packfiles is how they handle modification times.

Instead of having a single modification time, cruft packs have a separate .mtimes file.

This file contains the last modification time of all the objects in the pack. This means Git will be able to remove just the objects over 2 weeks old.

As well as the .pack file and the .mtimes file, a cruft pack also contains an index file with an `.idx` extension.

This includes the ID of the object as well as its exact location in the packfile, known as the offset.

Each object, index, and mtime entry matches the order in which the object was added.

So the third object in the pack file will match the third entry in the idx file and the third entry in the mtimes file.

The offset helps Git quickly locate an object without needing to count all the other objects.

Cruft packs were introduced in Git version 2.37.0 and can be generated by adding the --cruft flag to git gc, so git gc --cruft.

With this new Git feature implemented, GitHub enabled it for all repositories.

By applying a cruft pack to the main GitHub repo, they were able to reduce its size from 57GB to 27GB, a reduction of 52%.

And in an extreme example, they were able to reduce a 186GB repo to 2GB. That's a 92% reduction!

Wrapping things up

As someone who uses GitHub regularly I'm super impressed by this.

I often hear about their AI developments and UI improvements. But things like this tend to go under the radar, so it's nice to be able to give it some exposure.

Check out the original article if you want a more detailed explanation of how cruft packs work.

Otherwise, be sure to subscribe so you can get the next Hacking Scale article as soon as it's published.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

A tool for note-taking, documentation, project management, and more. All wrapped up with great collaboration features and a beautiful UI.

It has over 30 million users, 4 million of whom are paying for it. Not bad.

But if there was one common criticism of Notion, it was that it felt slow. Specifically when navigating between pages.

The team managed to make it faster with a few interesting techniques.

Let's go through it.

Estimated reading time: 4 minutes 35 seconds

What Made Notion Slow?

If you created a few blank pages in Notion, navigating between them would feel lightning fast.

But, if you added some images, tables, charts, and other complex widgets. Navigation would feel very slow.

There are no articles saying what caused this slowness. But we can make some assumptions from technical posts they've written:

1. Notion depended on many third-party scripts. Possibly for collaboration features, analytics, and third-party assets like images.

2. Frequent API calls being made to Notion's servers. This was because there was no or very limited caching in the browser.

3. CPU cores not being used for processing tasks. Most people have between 4 and 8 cores, which means 4 to 8 tasks could be processed at the same time. Notion wasn’t taking advantage of this.

Some previous attempts were made to fix these issues. The team used LocalStorage to cache data in the browser. But this had limited storage, which wasn't great for users with lots of pages.

They then tried using IndexedDB for caching. This was never shipped because it didn't improve performance. In fact, on certain devices it was even slower than LocalStorage.

Sidenote: LocalStorage vs IndexedDB

Both LocalStorage and IndexedDB are ways to store data in the browser.

This also means the data will get saved on a user's device. Meaning it will exist after closing a tab or restarting the browser.

But there are a few differences between them.

Because IndexedDB is different across browsers, fixing browser-specific bugs .

Also, some users would have Notion open on different tabs in the same browser. Notion had fine-grained data. This means if a page had a wall of text, each paragraph would have its own database row.

So lots of data being changed between tabs with IndexedDB would cause major performance issues.

The team improved performance for the desktop and mobile apps by using an SQLite database to cache data. So it made sense to try it on the browser.

To their surprise, it worked really well.

Why SQLite Worked

SQLite is a database like MySQL and Postgres, using SQL as its query language.

But it's different from them because it holds all its data in a single file and doesn't have a server.

Databases tend to use servers to manage data access, prevent conflicts, and control user permissions.

The lack of a server limits SQLite compared to other databases. But it was ideal for Notion's caching needs.

SQLite isn't natively supported in browsers. But it does have a WebAssembly version.

Sidenote: WebAssembly (WASM)

WebAssembly allows you to run code written in languages other than JavaScript in the browser.

If I wrote a really fast complex calculation in C++ and wanted to run it in the browser.

Instead of rewriting it in JavaScript, I could keep the C++ code, compile it to WebAssembly, and run it in the browser.

Because SQLite is written in C, it can be compiled into WebAssembly.

But a user will have to download all of SQLite before it can be used.

Unfortunately, Notion couldn't just drop SQLite into their project and call it a day. They had to make a bunch of changes first.

Problems with SQLite

As well as WebAssembly, SQLite uses a few other web technologies.

It uses a Web Worker to handle reading and writing to the database.

Web Workers allow code to run in the background, meaning they won't block actions on the main site.

The SQLite file was stored on the Origin Private File System (OPFS). Browsers cannot access a user's file system without their permission.

So OPFS provides an isolated file system only for the browser. This is separate from the main file system.

But OPFS has a crucial limitation. If one tab is reading or writing to a file, it locks the file to that tab, meaning changes made by other tabs will not work.

To fix this, Notion created a system where changes made by other tabs went to a single worker that had access to the database file. This was the Active Worker.

A SharedWorker was created to figure out which tab would have the active worker.

So if the active worker tab was closed. The SharedWorker would make another web worker active.

Sidenote: The two types of WASM SQLite

SQLite can interact with the OPFS virtual file system (VFS) in two different ways.

1. OPFS sqlite3_vfs

2. OPFS SyncAccessHandle Pool VFS

Note, OPFS isn't really a virtual file system; it's just an isolated environment which is where the term virtual comes from.

The first one, sqlite3_vfs, does support running on many tabs. But only works with cross-origin isolation. Cross-origin isolation puts the browser in a 'protective bubble' that gives it extra security.

But this restricts it from sharing data with other websites.

This didn't work for Notion because they depended on third-party scripts.

So they chose OPFS SyncAccessHandle Pool VFS.

This can only run in one tab. But is supported in all major browsers and has slightly better performance than sqlite3_vfs.

Another issue the team had with this approach was that pages loaded slower at first.

This was because a user would have to download SQLite if they didn't have it. It wasn't huge, under 1 MB. But on slower connections, it was noticeable.

To fix this, the team changed the way SQLite was loaded. Instead of loading it together with the site, they would wait for the page to finish loading first before downloading SQLite.

This meant that the initial page data wasn't coming from the cache. But the slight speed increase from loading initial data with SQLite wasn't worth the complication.

In general, the move to SQLite in the browser was a success.

The Notion site in certain parts of the world benefited from a 33% speed increase when navigating between pages.

Wrapping Things Up

I would love to have seen if this improved signups or kept users on the site for longer. Maybe Notion is holding off these metrics for another article.

Anyway, I hope you enjoyed this and learned something new. I certainly did.

If you want more details on this topic, you can check out the original article.

Until then, be sure to subscribe to get the next article as soon as it’s released.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 22 hours.

It's also insanely popular, with over 170 million users worldwide creating over 180 designs every second.

Canva's search is 'foundational' to its success. It allows users to search for templates, design assets, and other media.

It handles over 20,000 requests every second and 1 million every minute. But it has significant architectural issues that causes downtime and performance problems.

Here's how they addressed it.

Estimated reading time: 4 minutes 35 seconds

Why Does Canva Have Search?

If you have never used Canva, you may be wondering why its search can be so pivotal.

Well, it has a huge content library. With over 100 million stock images, video and graphics elements, as well as 600,000 templates.

It's users commonly search for media assets to help with designs. For example balloon or assets would help with a birthday card design.

The search functionality had 4 different servers and search indexes for each category. Media (images, videos, graphics), templates, fonts and audio.

There was some shared code between the servers but for the most part, they were completely separate.

This meant any updates to the search would have to be done 4 times. It was also difficult to do things like A/B testing without lots of duplication.

The team needed to find a way to reduce repeated code.

Sidenote: Search Index

Imagine you have a database of sentences, and you wanted to search the text "brown dog."

By default, a traditional database search would return an exact match. So, the sentence "The brown dog jumped over the fence" would be returned. But, "The dog brown jumped over the fence" would not be returned.

A search index is a data structure designed to help with this problem.

It works by breaking the text into individual words or tokens, then returns all the sentences that contain those words.

It also stores things like word frequency, the position of the word in the sentence, and caches these results for faster searches.

This of course, takes up more storage space and processing power. But it makes searching much faster.

A popular piece of software for creating search indexes is Apache Lucene. We will talk more about this later.

How Search Actually Worked

Even though Canva's search wasn't as complex as Google's, it still had many steps to go through.

These are the steps that would take place if a user searched for "brown dog."

1. Rewriting: Transform the query into standardized text. In the case of "brown dog," not much would be done. But if it had any spelling mistakes, was in another language or had uppercase letters, it would be rewritten.

2. Tokenization: Split the text into individual words or tokens.

3. Annotation: Add extra information (metadata) to expand the tokens and the search query. For example, it figures out that "dog" is an animal and "brown" is a color.

   It can also be configured to find synonyms, like chocolate or auburn for "brown," and canine or hound for "dog."

4. Candidate generation: This step reduces a large amount of data using various techniques. The goal is to create a smaller set of results based on the annotated query.

5. Re-ranking: Reorder the narrowed results based on relevance. For example, if a user had searched for a vet before, brown dogs with vets might be shown first.

After these five steps, the results are returned to the user. This is known as a search pipeline, and these were the core steps that were repeated across the four different search servers.

The team planned to separate these steps into their own components. This meant individuals could contribute without having to understand the entire system.

They also wanted to change the candidate generation step to use Elasticsearch instead of Solr.

Sidenote: Elasticsearch vs. Solr

Both Elasticsearch and Solr are platforms designed to efficiently search through search indexes.

They're both built on top of Apache Lucene, the Java-based open-source software used to create search indexes.

But, they do have some differences, which mostly favor Elasticsearch.

• Architecture: Elasticsearch is more optimized for scale because it uses a distributed architecture. Solr uses a more traditional client/server architecture. This requires more manual configuration for scaling.

• Querying: Solr is great for text search, but Elasticsearch is better at filtering, grouping data, and real-time indexing.

• Ease of Use: Solr can be complex to set up for beginners, but Elasticsearch is much easier. It has better documentation and a more user-friendly interface. It also has a larger active community with many plugins and extensions.

You can easily see why Canva switched from Solr to Elasticsearch. For more details, the team has put together a detailed article on their reasons.

The Migration

The decision to create components wasn't made overnight. There were many prototypes and experiments that led to this.

The team also paid special attention to creating a stable and clean interface. Making sure they followed good software design practices.

They came up with three criteria for each component:

1. Transient: Each component can be removed and reintroduced without any issues.

2. Stateless: Components should manage their own state and not share it with others. For example, the candidate generator doesn't need to know how many annotations have been cached.

3. Ordered: Each component processes data in the search pipeline order. Apart for the Annotation and Candidate generators. These steps can occur simultaneously. Why?

   As soon as the first set of annotations are processed—such as 'brown dog,' 'chocolate,' and 'canine'. They can be sent to the candidate generator to fetch some basic results.

   The annotation generator can continue producing more synonyms. These can then be sent to the candidate generator to refine the initial set of results.

After lots and lots of work, a component library that each search server could use was created.

The team wrote very detailed articles on the problems. This includes details about their process for creating the component library from the search pipeline. But, there wasn't much in the way of an updated architectural diagram.

So I've taken some creative freedom in drawing this new diagram based on my understanding of their solution.

The new architectural model still has many servers, but each has identical search pipeline components.

This change not only helped Canva release search updates quicker. It also allowed components to scale horizontally in the server based on the load. And allowed them to add observability and monitoring to each component of their search.

Wrapping Things Up

When I first looked at this article on the Canva blog, which has a part 1 and a part 2, I wasn't sure if it would be interesting enough to write about.

But I'm surprised at how much I could get out of it. Who knew search engines for design tools could be so complicated?

As usual, if you enjoyed this article, go ahead and subscribe to get the next one as soon as it's written.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 25 hours.

With over 150 million monthly active users and 28 million trips per day, Uber isn't going anywhere anytime soon.

The company has had its fair share of challenges, and a surprising one has been log messages.

Uber generates around 5PB of just INFO-level logs every month. This is when they're storing logs for only 3 days and deleting them afterward.

But somehow they managed to reduce storage size by 99%.

Here is how they did it.

Estimated reading time: 4 minutes 56 seconds

Why Uber generates so many logs?

Uber collects a lot of data: trip data, location data, user data, driver data, even weather data.

With all this data moving between systems, it is important to check, fix, and improve how these systems work.

One way they do this is by logging events from things like user actions, system processes, and errors.

These events generate a lot of logs—approximately 200 TB per day.

Instead of storing all the log data in one place, Uber stores it in a Hadoop Distributed File System (HDFS for short), a file system built for big data.

Sidenote: HDFS

A HDFS works by splitting large files into smaller blocks, around 128MB by default. Then storing these blocks on different machines (nodes).

Blocks are replicated three times by default across different nodes. This means if one node fails, data is still available.

This impacts storage since it triples the space needed for each file.

Each node runs a background process called a DataNode that stores the block and talks to a NameNode, the main node that tracks all the blocks.

If a block is added, the DataNode tells the NameNode, which tells the other DataNodes to replicate it.

If a client wants to read a file, they communicate with the NameNode, which tells the DataNodes which blocks to send to the client.

A HDFS client is a program that interacts with the HDFS cluster. Uber used one called Apache Spark, but there are others like Hadoop CLI and Apache Hive.

A HDFS is easy to scale, it's durable, and it handles large data well.

To analyze logs well, lots of them need to be collected over time. Uber’s data science team wanted to keep one months worth of logs.

But they could only store them for three days. Storing them for longer would mean the cost of their HDFS would reach millions of dollars per year.

There also wasn't a tool that could manage all these logs without costing the earth.

You might wonder why Uber doesn't use ClickHouse or Google BigQuery to compress and search the logs.

Well, Uber uses ClickHouse for structured logs, but a lot of their logs were unstructured, which ClickHouse wasn't designed for.

Sidenote: Structured vs. Unstructured Logs

Structured logs are typically easier to read and analyze than unstructured logs.

Here's an example of a structured log.

{
  "timestamp": "2021-07-29 14:52:55.1623",
  "level": "Info",
  "message": "New report created",
  "userId": "4253",
  "reportId": "4567",
  "action": "Report_Creation"
}

And here's an example of an unstructured log.

2021-07-29 14:52:55.1623 INFO New report 4567 created by user 4253

The structured log, typically written in JSON, is easy for humans and machines to read.

Unstructured logs need more complex parsing for a computer to understand, making them more difficult to analyze.

The large amount of unstructured logs from Uber could be down to legacy systems that were not configured to output structured logs.

Uber needed a way to reduce the size of the logs, and this is where CLP came in.

What is CLP?

Compressed Log Processing (CLP) is a tool designed to compress unstructured logs. It's also designed to search the compressed logs without decompressing them.

It was created by researchers from the University of Toronto, who later founded a company around it called YScope.

CLP compresses logs by at least 40x. In an example from YScope, they compressed 14TB of logs to 328 GB, which is just 2.26% of the original size. That's incredible.

Let's go through how it's able to do this.

If we take our previous unstructured log example and add an operation time.

2021-07-29 14:52:55.1623 INFO New report 4567 created by user 4253, operation took 1.23 seconds

CLP compresses this using these steps.

1. Parses the message into a timestamp, variable values, and log type.

2. Splits repetitive variables into a dictionary and non-repetitive ones into non-dictionary.

3. Encodes timestamps and non-dictionary variables into a binary format.

4. Places log type and variables into a dictionary to deduplicate values.

5. Stores the message in a three-column table of encoded messages.

The final table is then compressed again using Zstandard. A lossless compression method developed by Facebook.

Sidenote: Lossless vs. Lossy Compression

Imagine you have a detailed painting that you want to send to a friend who has slow internet.

You could compress the image using either lossy or lossless compression. Here are the differences:

Lossy compression removes some image data while still keeping the general shape so it is identifiable. This is how .jpg images and .mp3 audio works.

Lossless compression keeps all the image data. It compresses by storing data in a more efficient way.

For example, if pixels are repeated in the image. Instead of storing all the color information for each pixel. It just stores the color of the first pixel and the number of times it's repeated.

This is what .png and .wav files use.

Unfortunately, Uber were not able to use it directly on their logs; they had to use it in stages.

How Uber Used CLP

Uber initially wanted to use CLP entirely to compress logs. But they realized this approach wouldn't work.

Logs are streamed from the application to a solid state drive (SSD) before being uploaded to the HDFS.

This was so they could be stored quickly, and transferred to the HDFS in batches.

CLP works best by compressing large batches of logs which isn't ideal for streaming.

Also, CLP tends to use a lot of memory for its compression, and Uber's SSDs were already under high memory pressure to keep up with the logs.

To fix this, they decided to split CLPs 4-step compression approach into 2 phases doing 2 steps:

Phase 1: Only parse and encode the logs, then compress them with Zstandard before sending them to the HDFS.

Phase 2: Do the dictionary and deduplication step on batches of logs. Then create compressed columns for each log.

After Phase 1, this is what the logs looked like.

The <H> tags are used to mark different sections, making it easier to parse.

From this change the memory-intensive operations were performed on the HDFS instead of the SSD.

With just Phase 1 complete (just using 2 out of the 4 of CLPs compression steps). Uber was able to compress 5.38PB of logs to 31.4TB, which is 0.6% of the original size—a 99.4% reduction.

They were also able to increase log retention from three days to one month.

And that's a wrap

You may have noticed Phase 2 isn’t in this article. That’s because it was already getting too long, and we want to make them short and sweet for you.

Give this article a like if you’re interested in seeing part 2! Promise it’s worth it.

If you really can’t wait, here’s the original article, which funnily enough, is also written in two parts.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

Today it has a much broader audience and is used by millions of people every day—29 million, to be exact.

Like many other chat apps, Discord stores and analyzes every single one of its 4 billion daily messages.

Let's go through how and why they do that.

Estimated reading time: 4 minutes 56 seconds

Why Does Discord Analyze Your Messages?

Reading the opening paragraphs you might be shocked to learn that Discord stores every message, no matter when or where they were sent.

Even after a message is deleted, they still have access to it.

Here are a few reasons for that:

1. Identify bad communities or members: scammers, trolls, or those who violate their Terms of Service.

2. Figuring out what new features to add or how to improve existing ones.

3. Training their machine learning models. They use them to moderate content, analyze behavior, and rank issues.

4. Understanding their users. Analyzing engagement, retention, and demographics.

There are a few more reasons beyond those mentioned above. If you're interested, check out their Privacy Policy.

But, don't worry. Discord employees aren't reading your private messages. The data gets anonymized before it is stored, so they shouldn't know anything about you.

And for analysis, which is the focus of this article, they do much more.

When a user sends a message, it is saved in the application-specific database, which uses ScyllaDB.

This data is cleaned before being used. We’ll talk more about cleaning later.

But as Discord began to produce petabytes of data daily.

Yes, petabytes (1,000 terabytes)—the business needed a more automated process.

They needed a process that would automatically take raw data from the app database, clean it, and transform it to be used for analysis.

This was being done manually on request.

And they needed a solution that was easy to use for those outside of the data platform team.

This is why they developed Derived.

Sidenote: ScyllaDB

Scylla is a NoSQL database written in C++ and designed for high performance.

NoSQL databases don't use SQL to query data. They also lack a relational model like MySQL or PostgreSQL.

Instead, they use a different query language. Scylla uses CQL, which is the Cassandra Query Language used by another NoSQL database called Apache Cassandra.

Scylla also shards databases by default based on the number of CPU cores available.

For example, an M1 MacBook Pro has 10 CPU cores. So a 1,000-row database will be sharded into 10 databases containing 100 rows each. This helps with speed and scalability.

Scylla uses a wide-column store (like Cassandra). It stores data in tables with columns and rows. Each row has a unique key and can have a different set of columns.

This makes it more flexible than traditional rows, which are determined by columns.

What is Derived?

You may be wondering, what's wrong with the app data in the first place? Why can't it be used directly for analysis?

Aside from privacy concerns, the raw data used by the application is designed for the application, not for analysis.

The data has information that may not help the business. So, the cleaning process typically removes unnecessary data before use. This is part of a process called ETL. Extract, Transform, Load.

Discord used a tool called Airflow for this, which is an open-source tool for creating data pipelines. Typically, Airflow pipelines are written in Python.

The cleaned data for analysis is stored in another database called the Data Warehouse.

Temporary tables created from the Data Warehouse are called Derived Tables.

This is where the name "Derived" came from.

Sidenote: Data Warehouse

You may have figured this out based on the article, but a data warehouse is a place where the best quality data is stored.

This means the data has been cleaned and transformed for analysis.

Cleaning data means anonymizing it. So remove personal info and replace sensitive data with random text. Then remove duplicates and make sure things like dates are in a consistent format.

A data warehouse is the single source of truth for all the company's data, meaning data inside it should not be changed or deleted. But, it is possible to create tables based on transformations from the data warehouse.

Discord used Google's BigQuery as their data warehouse, which is a fully managed service used to store and process data.

It is a service that is part of Google Cloud Platform, Google's version of AWS.

Data from the Warehouse can be used in business intelligence tools like Looker or Power BI. It can also train machine learning models.

Before Derived, if someone needed specific data like the number of daily sign ups. They would communicate that to the data platform team, who would manually write the code to create that derived table.

But with Derived, the requester would create a config file. This would contain the needed data, plus some optional extras.

This file would be submitted as a pull request to the repository containing code for the data transformations. Basically a repo containing all the Airflow files.

Then, a continuous integration process, something like a GitHub Action, would create the derived table based on the file.

One config file per table.

This approach solved the problem of the previous system not being easy to edit by other teams.

To address the issue of data not being updated frequently enough, they came up with a different solution.

The team used a service called Cloud Pub/Sub to update data warehouse data whenever application data changed.

Sidenote: Pub/Sub

Pub/Sub is a way to send messages from one application to another.

"Pub" stands for Publish, and "Sub" stands for Subscribe.

To send a message (which could be any data) from app A to app B, app A would be the publisher. It would publish the message to a topic.

A topic is like a channel, but more of a distribution channel and less like a TV channel. App B would subscribe to that topic and receive the message.

Pub/Sub is different from request/response and other messaging patterns. This is because publishers don’t wait for a response before sending another message.

And in the case of Cloud Pub/Sub, if app B is down when app A sends a message, the topic keeps it until app B is back online.

This means messages will never be lost.

This method was used for important tables that needed frequent updates. Less critical tables were batch-updated every hour or day.

The final focus was speed. The team copied frequently used tables from the data warehouse to a Scylla database. They used it to run queries, as BigQuery isn't the fastest for that.

With all that in place, this is what the final process for analyzing data looked like:

Wrapping Things Up

This topic is a bit different from the usual posts here. It's more data-focused and less engineering-focused. But scale is scale, no matter the discipline.

I hope this gives some insight into the issues that a data platform team may face with lots of data.

As usual, if you want a much more detailed account, check out the original article.

If you would like more technical summaries from companies like Uber and Canva, go ahead and subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 25 hours.

What makes it so unique is its amazing collaboration features.

Users can work on the same file simultaneously. Seeing exactly what everyone is doing as they edit and comment.

This is one reason Figma has 4 million active users. And also why they focus on making collaboration fast, even as their user base grows.

They do this in many ways, but one important way is by developing a system called LiveGraph.

Estimated reading time: 4 minutes 40 seconds

LiveGraph?

When Figma first launched its collaboration feature back in 2016, it ran on a simple tech stack.

React on the frontend, Ruby on the backend. Redux for state management, and WebSockets for real-time communication.

Collaboration worked like this. When a client changed something, like renaming a file. The database was updated. And then that event was broadcasted to other clients.

Events were hand-coded. The process was time-consuming and became complicated as Figma added more features.

This is why LiveGraph was created.

It was inspired by GraphQL, built in-house, and serves as a data store or cache for all clients. It also manages database queries and updates.

This means that instead of having to manually write an event to broadcast. Clients subscribe to LiveGraph.

So clients know whenever something changes. Like subscribing to a newsletter or your favorite YouTube channel.

This approach was easier to manage.

Sidenote: GraphQL

GraphQL is a query language that makes it easy to request data from a server.

The reason for using it over REST is simple. With REST, you need many endpoints to get different data. With this, you can write a single endpoint to get all the data you need.

Using JSON-like syntax for the query, a single or many HTTP POST requests are made to the server.

With GraphQL however, it's difficult to get real-time updates, especially with event streams.

This is why Figma built LiveGraph.

Problems with LiveGraph

LiveGraph worked well for the user base Figma had at the time. But since its release, Figma users tripled and page views increased by 5x.

This all ran on a single Postgres database which had tables several terabytes in size.

A technique called sharding solved this giving the database near-infinite scalability.

LiveGraph wasn't designed for such a large database; it also needed to be scaled.

This started a project called LiveGraph 100x.

This article will focus on two techniques Figma used to scale LiveGraph.

1. Improving caching

2. Creating an invalidator.

Sidenote: Sharding

For most projects, it's perfectly fine to have a single database.

But for databases with millions or even billions of rows, a single database would consume lots of resources.

Sharding is a way to split a large database into smaller databases, each containing a subset of the data.

There is horizontal sharding, which splits the database by rows. And there is vertical sharding, which splits it by columns.

Figma implemented a horizontal sharding solution.

As a simple example, if a database had 1,000 rows, it could be sharded into 10 databases, each containing 100 rows.

Improving The Caching

The obvious first step to scaling LiveGraph was to increase its number of instances. This would allow the workload to be shared.

The problem with this is that it would create many caches. The database wouldn't know which cache requested the data, so it would send the data to all of them.

To solve this, the cache was taken out and moved to a single centralized cache.

Now all LiveGraph instances would use one cache. And data would go to the same place.

But this raised another issue.

The large number of users and their Figma activities could make the cache very large.

So they decided to shard the cache, just as they did with the database.

The cache was a key-value store containing the query as the key and the results as the value. Sharding worked by hashing the key, then storing both the key and the value.

This meant a LiveGraph instance would only use its needed cache shards, not the entire cache.

But this solution introduces two problems.

1. The database wouldn't know which cache shard to send the data to.

2. Changing one shard might require changes to another shard.

For example, if a file is moved to a different folder, two shards would need to be updated. The shard that stores files from the old folder and the shard that stores files from the new folder.

To solve these problems, the team created an Invalidator.

Sidenote: Hashing

Hashing is the process of converting data into a fixed-length of letters and numbers.

As a simple example, imagine we want to hash the word "hello."

We could take the position of each letter in the alphabet, then add them all together. So "hello" would become 52.

Of course, modern hashing algorithms are much more complex.

Using the MD5 algorithm, "hello" would be:

5d41402abc4b2a76b9719d911017c592.

Hashing is very fast. It provides a unique ID for data, and the same input with the same algorithm will always produce the same output.

They are commonly used to store passwords and create indexes for things like databases.

Creating an Invalidator

The system now has many cache shards. A Figma user typically only uses a specific part of the app at a time. So a LiveGraph instance doesn't need to subscribe to all the shards.

But sometimes an update in one shard would trigger a different shard to update. And if LiveGraph isn't subscribed to a shard, it wouldn't display the update.

Think back to the file moving example.

To save resources, instead of automatically updating shards, they were marked as invalid.

Meaning if in the future, a LiveGraph instance subscribes to an invalid shard. It knows it needs to get updated data from the database.

The job of the invalidator was to mark the correct shards as invalid.

It was a server that was sharded like the database. And worked by reading the logs of that specific database shard.

It would then figure out what cache shards to mark as invalid based on mutations in the DB.

With both cache sharding and invalidators in place, here is the new LiveGraph.

It is truly impressive how the engineering team at Figma came up with these solutions.

Wrapping Things Up

It's important to remember that Figma built all these technologies in-house. This means there are a lot of details not covered in this article; for example,

• Specifics of subscribing to cache shards.

• How the invalidator creates queries by comparing data before and after a change.

• And how queries are routed to the correct database or cache shard.

This is all covered in the original article. As well as a database scaling article and a talk given by one of Figma's engineers.

But if you're satisfied with this information and can't wait for the next one. Go ahead and subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 25 hours.

The event was caused by the cybersecurity software, CrowdStrike Falcon. I mean, you couldn't make this up.

The software designed to protect against attacks was what caused the problem.

Although 8.5 million computers is less than 1% of Windows machines worldwide. The last major cyber event in 2017 affected 300,000 computers. So this is a massive step up.

But what exactly happened?

Estimated reading time: 4 minutes 55 seconds

What is CrowdStrike Falcon?

You've most likely heard of antivirus software like Norton or McAfee.

Falcon is like that, but it focuses on protecting a large network instead of a single computer. This is known as endpoint security.

It's cloud-based, but each machine in the network (endpoint) installs a small piece of software called a Falcon Sensor.

Once installed, the sensor constantly monitors and sends information to Falcon's servers.

These servers analyze all data from sensors using machine learning and threat intelligence. Basically, CrowdStrike uses its vast knowledge of cyber threats and hackers to check if a machine is infected.

To add to that, CrowdStrike has a team of researchers checking for new threats and adding their findings back to Falcon's database.

The sensors themselves have a 'detection engine' that also uses machine learning. This analyzes files and system processes.

Security engineers using Falcon have access to a web interface. This shows all monitored endpoints and alerts them if any threats are detected.

Sounds impressive.

If I owned a large organization with hundreds or thousands of machines to protect, I'd definitely buy Falcon.

But what's interesting about the sensors is that they operate at the kernel level. This is something few other programs do.

It has complete access to a machine. Meaning it can check all system activities, including hardware info such as memory and disk usage.

Sidenote: The Kernel

The kernel is a program in the operating system. It sits between hardware and software, managing their communication.

If your browser needs more memory, it doesn't need to know the type or amount of memory available; it just asks the kernel.

The kernel also stops software from accessing core system functions. These include CPU control, system configuration, and power management.

Third-party software installed at the kernel level sits between the kernel and the software. So it can access core system functions.

They can also be installed and updated from the internet in the background, without the user knowing.

If the kernel encounters a critical error that it doesn't know how to handle, known as a kernel panic, it will typically cause a blue screen on Windows.

As well as being able to receive data from sensors, Falcon servers can also send data to sensors (such as updates).

In this case, a malformed configuration update was sent to all Windows sensors, which caused those machines to stop working.

Because the issue was at the kernel level, it prevented Windows from starting correctly.

Who would have thought that such a small change would cause such a big problem?

The Configuration File

As mentioned before, the cause of the mass outage was a configuration update sent to all Falcon sensors for Windows.

Configuration updates are placed inside ‘channel files.’ In this case, the affected file was channel file 291, or C-00000291.sys. These are located in the Windows critical systems folder.

This specific update changed how Falcon analyzes 'named pipes' in Windows. A way for different processes, or programs, to communicate with each other even if they are on completely different machines.

Sidenote: Named Pipes

Imagine you have two small programs, one for adding two numbers (add) and one for doubling a number (double).

If you wanted to add a number and then double the result of that addition, you could do it like this:

add 1 2 | double  # result 6 (3 x 2)

The | character is called a pipe and it's used to connect the output of one program to the input of another.

This connection is temporary, so it closes when the program finishes. But if you want a more permanent connection, you could use a named pipe.

This is a special file that acts as a communication channel between two programs. So using a named pipe for the previous example would look like this:

add 1 2 > myNamedPipe
double < myNamedPipe # result 6

This is a very simple example, but the same concept applies to most of the programs on your computer.

For example, a browser extension wanting to communicate with a local password manager would use a named pipe.

The team discovered that hackers could use named pipes as an attack technique, so the configuration update was sent to detect this.

What's interesting is that this update was first released at the end of February 2024 and was stress-tested at the beginning of March.

Which means it was available in a customer’s sensor, but was not used until after it was stress tested.

A confusing way to release something, but it wasn’t breaking Windows when it was first released.

The July 19th update was a bug fix to what was tested in March. The main feature was already stress-tested, so the team saw no need to heavily test this specific fix.

Although light testing was conducted, there was a bug in their testing system that prevented an issue in the bug fix from being caught.

It was only after they released it that they noticed channel file 291 was causing errors when trying to access some memory that didn't exist.

Which was something the kernel didn't know how to handle, resulting in a Windows crash.

Some speculate that this crash was caused by a file full of null bytes or characters, which means the update file was full of zeros.

But CrowdStrike has responded saying, that's just how Windows works.

When a program is creating a new file, Windows first fills the new file with null bytes (a bunch of zeros) before adding data to that file.

Meaning the crash could have happened while a new file was being created.

What Happens Now?

Thankfully, it took CrowdStrike 78 minutes to release a fix.

Meaning affected machines should automatically download and install the update after a good old restart.

But if the machine crashes again after being restarted, the fix would need to be applied the hard way.

Someone would need to download the fix from a working computer to a USB drive.

Then put that USB into the broken machine before turning it on. When it does turn on, a menu should show, allowing the fix to be selected.

CrowdStrike has outlined steps they will take to prevent this from happening again, which include:

• improved developer testing

• improved stress testing

• improving their light testing stack

• and improving their error handling so that future errors do not cause crashes

Wrapping Things Up

I truly sympathize with those affected by this outage. It caused missed or canceled flights, delayed mail, and failed 911 calls.

It goes to show just how much damage the code we write can cause when it's not properly reviewed and tested.

I wish I could say this will never happen again, but we're human; we make mistakes, so it’s only a matter of time.

But hopefully not on this scale.

Anyway, I hope you enjoyed this article.

If you would like more of these, then be sure to subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.

With around 240 million paid subscribers, Netflix has to be the world's most popular streaming service. And it’s well deserved.

Wherever you are in the world, no matter the time or device, you can press play on any piece of Netflix content and it will work.

Does that mean that Netflix never has issues? Nope, things go wrong quite often. But they guarantee you'll always be able to watch your favorite show.

Here's how they can do that.

Estimated reading time: 4 minutes 13 seconds

What Goes Wrong?

Just like with many other services, there are lots of things that could affect a Netflix user's streaming experience.

1. Network Blip: A user's network connection temporarily goes down or has another issue.

2. Under Scaled Services: Cloud servers have not scaled up or do not have enough resources (CPU, RAM, Disk) to handle the traffic.

3. Retry Storms: A backend service goes down, meaning client requests fail, so it retries and retries, causing requests to build up.

4. Bad Deployments: Features or updates that introduce bugs.

This is not an exhaustive list, but remember that the main purpose of Netflix is to provide great content to its users. If any of these issues prevent a user from doing that, then Netflix is not fulfilling its purpose.

Considering most issues affect Netflix's backend services. The solution must 'shield' content playback from any potential problems.

Sidenote: API Gateway

Netflix has many backend services, as well as many clients that all communicate with them.

Imagine all the connection lines between them; it would look a lot like spaghetti.

An API Gateway is a server that sits between all those clients and the backend services. It's like a traffic controller routing requests to the right service. This results in cleaner, less confusing connections.

It can also check that the client has the authority to make requests to certain services and monitor requests, more about that later.

The Shield

If Netflix had a problem and no users were online, it could be resolved quickly without anyone noticing.

But if there's a problem, like not being able to favorite a show, and someone tries to use that feature, this would make the problem worse. Their attempts would send more requests to the backend, putting more strain on its resources.

It wouldn't make sense to block this feature because Netflix doesn’t want to scare its users.

But what they could do is ‘throttle’ those requests using the API Gateway.

Sidenote: Throttling

If you show up at a popular restaurant without booking ahead, you may be asked to come back later when a table is available.

Restaurants can only provide a certain number of seats at a time, or they would get overcrowded. This is how throttling works.

A service can usually handle only a certain number of requests at a time. A request threshold can be set, say 5 requests per minute.

If 6 requests are made in a minute, the 6th request is either held for a specified amount of time before being processed (rate limiting) or rejected.

How It Worked

Because Netflix's API Gateway was configured to track CPU load, error rates, and a bunch of other things for all the backend services.

It knew how many errors each service had and how many requests were being sent to them.

So if a service was getting a lot of requests and had lots of errors, this was a good indicator that any further requests would need to be throttled.

Sidenote: Collecting Request Metrics

Whenever a request is sent from a client to the API Gateway, it starts collecting metrics like response time, status code, request size, and response size.

This happens before the request is directed to the appropriate service.

When the service sends back a response, it goes through the gateway, which finishes collecting metrics before sending it to the client.

Of course, there are some services that if throttled, would have more of an impact on the ability to watch content than others. So the team prioritized requests based on:

1. Functionality: What will be affected if this request is throttled? If it's important to the user, then it's less likely to be throttled.

2. Point of origin: Is this request from a user interaction or something else, like a cron job? User interactions are less likely to be throttled.

3. Fallback available: If a request gets throttled, does it have a reasonable fallback? For example, if a trailer doesn’t play on hover, will the user see an image? If there's a good fallback, then it's more likely to be throttled.

4. Throughput: If the backend service tends to receive a lot of requests, like logs, then these requests are more likely to be throttled.

Based on these criteria, each request was given a score between 0 and 100 before being routed. With 0 being high priority (less likely to be throttled) and 100 being low priority (more likely to be throttled).

The team implemented a threshold number, for example 40, and if a request's score was above that number, it would be throttled.

This threshold was determined by the health of all the backend services which again, was monitored by the API Gateway. The worse the health, the lower the threshold and vice versa.

There are no hard numbers in the original article on how much resource, or time this technique saved the company (which is a shame).

But the gif below is a recording of what a potential user would experience if the backend system was recovering from an issue.

As you can see, they were able to play their favorite show without interruption, oblivious to what was going on in the background.

Let's Call It

I could go on, but I think this is a good place to stop.

The team must have put a huge amount of effort into getting this across the line. I mean, the API gateway is written in Java, so bravo to them.

If you want more information about this there's plenty of it out there.

I recommend reading the original article, watching this video, and reading this article as well.

But if you don't have time to do all that and are enjoying these simplified summaries, you know what to do.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 19 hours.

And video accounts for over 80% of its total traffic.

With those numbers, it's difficult to imagine how much computation time and resources it takes to upload, encode and publish videos from all those users.

But Instagram managed to reduce that time by 94% and also improve their video quality.

Here's how.

Estimated reading time: 4 minutes 52 seconds

The Process from Upload to Publish

Here are the typical steps that take place whenever a user uploads a video on Instagram:

1. Pre-processing: Enhance the video’s quality like color, sharpness, frame rate, etc.

2. Compression/Encoding: Reduce the file size

3. Packaging: Splitting it into smaller chunks for streaming

For this article, we will focus on the encoding and packaging steps.

Sidenote: Video Encoding

If you were to record a 10-second 1080 video on your phone without any compression, it would be around 1.7 GB.

That’s a lot!

To make it smaller your phone uses something called a codec, that compresses the video for storage using efficient algorithms.

So efficient that it will get the file size down to 35MB, but it's in a format that’s not designed to be read by humans.

To watch the encoded video, a codec needs to decompress the file to pixels that can be displayed on your screen.

The compression process is called encoding, and the decompression process is called decoding.

Codecs have improved over time so there are many of them out there. And they’re stored in most devices, cameras, phones, computers, etc.

Instagram generated two types of encodings on upload: Advanced Encoding (AV1), and Simple Encoding (H.264).

Advanced encoding produces videos that are small in size with great quality. These kind of videos only made up 15% of Instagram’s total watch time.

Simple encoding produces videos that work on older devices, but used a less efficient method of compression, meaning that videos are small with not great quality.

To make matters worse, simple encoding alone took up more than 80% of Instagram's computing resources.

Why Simple Encoding Is Such a Resource Hog

For Simple encoding, a video is actually encoded in two formats:

• Adaptive bit rate (ABR): video quality will change based on the user's connection speed.

• Progressive: video quality stays the same no matter the connection. This was for older versions of Instagram that don't support ABR.

Both ABR and progressive created multiple encodings of the same video in different resolutions and bit rates.

But for progressive, the video player will only play one encoded video.

While for ABR those videos are split into small 2-10 second chunks, and the video player will change which chunk is played based on the user’s internet speed.

Sidenote: Bit rate

When a video is encoded, it stores binary data (1s and 0s) for each frame of the video, the more information each frame has, the higher its bit rate.

If I recorded a video of a still pond the compression algorithm will notice that most pixels stay blue, and store them with less data to keep the pixels the same.

If I had a recording of a fast-flowing waterfall and the compression algorithm kept pixels the same, the video would look odd.

Since pixels change a lot between frames it needs to store more information in each frame.

Bit rate is measured in megabits per second (mbps) since this is how much data is sent to the video player.

On YouTube the average bitrate for a 1080 video is 8Mbps which is 1MB of transmitted data every second.

If you had to guess which specific process was taking up the most resources, you'd correctly guess adaptive bit rate.

This is not only due to creating multiple video files, but also because the additional packaging step involves complex algorithms to figure out how to seamlessly switch between different video qualities.

The Clever Fix

Usually, progressive encoding creates just one video file. But because Instagram was creating multiple files with the same codec as ABR (H.264).

They realized they could use the same files for progressive and ABR eliminating the need to create two sets of the same videos.

If you compare the image above to the previous image, you’ll see that 4 videos are now created during the encoding stage instead of 8.

The team were able to use the same progressive files for the packaging stage of ABR which wasn’t as efficient as before resulting in poorer compression.

But they did save a lot of resources.

Instagram claims the old ABR process took 86 seconds for a 23-second video.

But the new ABR process, just packaging, took 0.36 seconds, which is a whopping 99% reduction in processing time.

With this much reduction Instagram could dedicate more resources to the advanced encoding process, which meant more users could see higher quality videos. How?

Because simple encoding took longer in the old process and used more resources, there wasn’t enough to always create advanced videos.

With the new process, there was enough resource to run both types of encoding, meaning both can be published and more users would see higher quality videos.

This resulted in an increase in views of advanced encoded video from 15% to 48%.

Sidenote: Encoding vs Transcoding

This is an optional side note for the video experts among you.

The word transcoding isn't used in this article, but technically it should have been.

Encoding is the process of compressing an uncompressed video into a smaller format.

Transcoding is the process of changing a video from one encoded format to the same, or another format.

Because all devices (phones, cameras) have a codec, when a video is recorded it is automatically encoded.

So even before you upload a video to Instagram it is already encoded, and any further encoding is called transcoding.

But because the original article mostly uses the term encoding and it’s is such a catch-all term used in the industry, I decided to stick with it.

Wrapping Things Up

After reading this you may be thinking, how did the team not spot this obvious improvement?

Well, small issues on a small scale are often overlooked. Small issues on a large scale no longer remain small issues, and I guess that's what happened here.

Besides, Instagram was always a photo app that is now focusing more on video, so I assume it's a learning process for them too.

If you want to read more about their learnings, check out the Meta Engineering Blog.

But if you enjoyed this simplified version, be sure to subscribe.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 16 hours.

To put that into perspective, the average enterprise company will do fine with around 50-100 nodes.

This enormous increase in computing power worked because of lots of changes.

But the most important one was removing a plugin called Flannel.

Why? Let’s dive into it.

Estimated reading time: 3 minutes 59 seconds

What is Flannel?

Imagine we have a Kubernetes cluster that contains two nodes (virtual computers), one for a Node.js web server, another for a Postgres database.

Inside each of these nodes we'll have two pods (containers), one for the web server/database and something to collect logs like Vector.

Sidenote: Node vs Pod

Pods usually contain a single application (multiple if needed), and a node provides resources for the pod to run, CPU, Memory etc.

Nodes get assigned an IP address by the cloud provider. Pods do get assigned IPs by default using kubnet, but are used to communicate only within the same node.

If you want pods in different nodes to communicate, HTTP is a good approach.

But before you can even do that, you'll need to give each pod a unique IP address so they know where to send data.

A bit like giving someone a unique postal address so you're able to send letters to them.

Pod-to-pod communication between nodes out-of-the-box with Kubernetes is not straightforward, and this is where Flannel comes in.

Flannel exists to make pod-to-pod communication really easy. It configures the network for each pod and does things like assigns each pod an IP address.

In the world of Kubernetes, Flannel is called a CNI plugin (Container Network Interface). It's not the only one; there are others like Calico and Cilium.

But Flannel is the easiest to setup.

Why Was it Removed?

Flannel is a great solution, but it wasn't designed for thousands of nodes.

OpenAI was already seeing slow speeds with Flannel for 2,500 nodes, so 7,500 nodes would have definitely been a struggle.

To fully understand what made it slow, let's go back to our previous example and double the number of nodes.

For communication between nodes, Flannel performs several tasks:

• Assigns each cluster a range of IP addresses for each node and a range of IPs for nodes to give to each pod (subnet allocation).

• Creates route tables to keep track of which node has which IP address.

• With this information, if data needs to be sent to a specific pod, Flannel will know the exact IP address of the node to which the pod belongs to (traffic routing).

• It also labels traveling data (packets) so that if it needs to pass through a cluster to reach its destination, it is not read by accident (packet encapsulation).

The aim of Flannel is to speed up communication, but with all these processes on top of thousands of nodes, it actually slows down the network.

As shown in the image below, with 150,000 requests per second (RPS), Flannel’s VXLAN tends to have the worst performance.

Of course, the team at OpenAI needed to do something about this.

Sidenote: VXLAN

VXLAN (Virtual eXtensible Local Area Network) is used to create a virtual network that allows machines to communicate.

Long before the internet, if you had two physical computers, you would have to connect a cable between them so they could communicate.

A VXLAN simulates this using the power of the internet or a larger network.

If two computers are connected to the internet and have VXLAN software installed, they each get a unique IP address, and packets are encapsulated so that it’s not viewed by another machine.

What Was Used Instead?

If you keep up with OpenAI news, or just AI news in general, you'll know that Microsoft has put a lot of money behind them.

You may also know that Microsoft owns Azure, which is a huge cloud provider.

So it only makes sense that OpenAI hosts all of its infrastructure on Azure. I mean, it would be weird if they hosted it on Google Cloud or AWS, right?

Since OpenAI was using the Azure Kubernetes Service (AKS) for their infrastructure, using Azure's CNI was the obvious choice, as both services work better together.

Azure's CNI isn't just a fork of Flannel with Microsoft branding; it's different in many ways:

• Both nodes and pods are given IP addresses from the same range (subnet allocation). For example, if a node has IP 10.0.0.1, the pod gets 10.0.0.2, allowing direct pod-to-pod communication without going through a node.

• It doesn't use a virtual network (VXLAN); it uses direct routing. This means no virtual routes need to be created, and there's no encapsulation of packets.

• It is not a general-purpose CNI; it is specifically designed for Azure Kubernetes Service, meaning it has native integration and better performance with an AKS cluster.

All of these changes made Azure CNI much faster than Flannel, but it's not the same as Flannel.

The team had to use a different solution to keep track of packets and rely on a different routing engine.

However, the benefits outweigh the drawbacks.

OpenAI even claimed that pod-to-pod communication between clusters was as fast as direct communication between pods in the same node.

And That’s a Wrap

It's amazing to think that this whole article was just one paragraph of the original post.

There's a lot of good information in there, and I recommend you check it out.

But if you enjoyed this simplified format, go ahead and subscribe to get more like this.

PS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (LinkedIn, Twitter, YouTube, Instagram). It only takes 10 seconds. Making this one took 20 hours.