Introduction

In the previous part of this series, we explored how a CDN(Content delivery network) help deliver videos from servers to users with low latency and minimal buffering .

We also looked at the step-by-step workflow of how a CDN works behind the scenes to serve millions of viewers efficiently.

Now , let's move to the next piece of this puzzle.

NOTE: If you haven't read the previous blog in this series , I recommend checking out Part 1 before continuing.

A Brief History of Load Balancers

In the early days of the internet, websites ran on a single server and traffic was manageable. But as internet adoption exploded through the late 1990s and 2000s, that single server model started cracking under pressure.

The first solution was hardware load balancers -physical devices that sat in front of servers and distributed incoming traffic. Companies like F5 Networks pioneered this space with dedicated appliances that could handle high volumes of requests. These devices worked well, but they were expensive, difficult to scale, and required manual configuration.

As cloud computing matured, the industry shifted toward software-defined load balancing -load balancers that run as software rather than dedicated hardware. This made them dramatically cheaper, faster to deploy, and infinitely more flexible.

Today, cloud providers like AWS, Google Cloud, and Azure offer fully managed load balancing services that can scale automatically, handle millions of requests per second, and require almost no manual intervention. For platforms like JioHotstar, this evolution was critical it's what makes serving 50+ million concurrent viewers during an IPL final even possible.

How servers handle massive traffic using Load Balancers?

Concept

A load balancer is a system that distributes and balances the incoming traffic among the servers to provide high availability, efficient utilization of servers and high performance.

Let's understand how a Load Balancer works using a simple analogy.

Analogy

Imagine a busy restaurant .Customers keep walking in through the door. And the restaurant has three waiters(waiter A,B,C).If every single customer ran straight to waiter A, that poor waiter would collapse while the other two stand around doing nothing.

So the restaurant tries to place a person/host at the entrance. Here the job of host is simple but effective He will be monitoring each waiters current load, sends the next customer to whoever is free and he never sends customers to an already drowning waiter.

Now coming back to the topic .Here

Customers represent internet traffic.

Waiters represent servers.

The host standing at the entrance acts as the Load Balancer.

The result : No single waiter will sit idle, every customer gets served and restaurant handles 3x crowd at once.

So that's essentially how a Load Balancer works. making sure no single server gets overwhelmed while others sit idle. when one server crashes, the host stops people there and redistribute to the rest. the customers don't even notice.

Now that we understand the basic concept of a Load Balancer, let's see why it becomes so important for large scale streaming platforms like Jio Hotstar.

During an IPL match, millions of users may join the stream within a matter of seconds. Handling such massive traffic reliably requires much more than a powerful server—it requires intelligent traffic distribution.

Benefits of Load Balancing in Video Streaming

Load balancing plays a crucial role in helping streaming platforms handle millions of users efficiently. Some of its major benefits include:

1. Enhanced Performance and Better Streaming Quality

A Load Balancer distributes incoming traffic across multiple servers instead of sending everything to a single server.

As a result:

• Servers process requests faster.

• Users experience lower latency.

• Buffering is reduced.

• Video playback becomes smoother.

For example, during an IPL match, millions of viewers may join the stream simultaneously. Load balancing ensures that this traffic is spread across multiple servers, preventing performance degradation.

2.High Availability and Reliability

Streaming platforms cannot afford downtime, especially during live events.

Load Balancers continuously monitor the health of backend servers. If a server becomes slow, unresponsive, or completely fails, the Load Balancer automatically redirects traffic to healthy servers.

This ensures that:

• The streaming service remains available.

• Users experience minimal disruption.

• Single server failures do not bring down the entire platform.

3. Seamless Scalability

Traffic on streaming platforms is highly unpredictable.

Load Balancers make it easy to add new servers whenever traffic increases. Once a new server is added, the Load Balancer automatically starts sending traffic to it.

This allows platforms to:

• Handle sudden traffic spikes.

• Maintain consistent performance.

• Scale infrastructure without affecting users.

4. Efficient Resource Utilization

Without Load Balancing, some servers may become overloaded while others remain underutilized.

A Load Balancer ensures that traffic is distributed evenly across available servers, making better use of computing resources and improving overall system efficiency.

5. Better User Experience

The ultimate goal of Load Balancing is to provide a seamless experience for viewers.

By reducing server overload, improving availability, and minimizing latency, users can enjoy uninterrupted streaming even during high-traffic events.

For platforms like JioHotstar, this means millions of viewers can watch live matches simultaneously without experiencing significant buffering or service interruptions.

Can a Load Balancer Become a Single Point of Failure?

💡If the Load Balancer itself fails, won't the entire system fail? In production systems, organizations deploy multiple Load Balancers across different Availability Zones. Cloud providers like AWS automatically ensure redundancy, preventing the Load Balancer from becoming a single point of failure.

Step by Step Workflow of Load Balancing In Video Streaming.

Step 1: User Opens JioHotstar

A user opens the Jio Hotstarapp and clicks on an IPL live stream.

Step 2: Request Reaches the Load Balancer

Instead of directly contacting a backend server, the request first reaches the Load Balancer.

The Load Balancer acts as the entry point for all incoming traffic.

Step 3: Load Balancer Checks Server Health

The Load Balancer continuously pings all servers with health checks. It maintains a live registry: which servers are healthy, which have failed, and how loaded each one is.

Step 4: Best Server is Selected

The Load Balancer doesn't randomly pick a server. It uses a routing algorithm to make an intelligent decision every single time a request arrives. The three most commonly used algorithms are:

Round Robin Requests are distributed to each server one by one in a fixed rotation. If there are three servers A, B, and C ,the first request goes to A, the second to B, the third to C, and then the cycle repeats. This works well when all servers have similar capacity and the requests are roughly equal in complexity.

Least Connections Instead of rotating blindly, this algorithm sends each new request to whichever server currently has the fewest active connections. This is smarter than Round Robin because it adapts to real workload — if Server A is handling 500 active streams and Server B only has 100, new traffic automatically goes to Server B. This is particularly useful for video streaming where some sessions last much longer than others.

Least Response Time This algorithm routes traffic to whichever server is responding the fastest at that moment. It combines both active connection count and actual server response speed to make its decision. For a live streaming platform like JioHotstar, where even a 100ms delay can cause buffering during a critical IPL over, this algorithm ensures users are always connected to the most performant server available.

Step 5: Request is Forwarded

The Load Balancer forwards the request to the selected backend server.

Step 6: Server Processes the Request

The server:

→ Authenticates the user

→ Verifies subscription status

→ Retrieves stream information

→ Generates the response

Step 7: Response Returns to User

The processed response is sent back through the Load Balancer to the user.

Step 8: Traffic is Distributed Continuously

As millions of users join the IPL stream, requests are distributed among multiple servers.

No single server becomes overloaded.

Step 9: Server Failure Handling

Suppose Server 2 crashes.

The Load Balancer detects the failure through health checks.

New traffic is automatically redirected to healthy servers.

Users usually don't notice the failure.

Step 10: Smooth Streaming Experience

Because traffic is distributed efficiently:

→ Servers remain healthy

→ Response times stay low

→ The platform remains available

→ Millions of users can watch IPL simultaneously

Complete Work Flow

AWS Connection

Elastic Load Balancer (ELB)

After understanding how Load Balancers work, a natural question is:

How do companies implement load balancing in the cloud?

AWS provides a managed service called Elastic Load Balancer (ELB) that automatically distributes incoming traffic across multiple targets such as EC2 instances, containers, and applications.

Instead of manually building and managing a Load Balancer, engineers can use AWS ELB to handle traffic distribution, health checks, and failover automatically.

Key Features of AWS ELB

1) Traffic Distribution

ELB automatically distributes incoming requests across multiple servers, preventing any single server from becoming overloaded.

2) Health Checks

ELB continuously monitors the health of backend servers. If a server becomes unhealthy, ELB automatically stops sending traffic to it and redirects requests to healthy servers.

3) High Availability

Traffic can be distributed across multiple Availability Zones, helping applications remain available even if one server or zone experiences issues.

4) Seamless Scaling

As traffic increases, new servers can be added behind the Load Balancer, allowing applications to handle growing demand efficiently.

Types of Load Balancers in AWS

To understand AWS load balancers properly, you first need to understand what Layer 4 and Layer 7 mean. These refer to layers in the OSI model — a framework that describes how data travels across a network.

Think of it this way — Layer 4 is like a postal service that reads only the address on the envelope to decide where to deliver it. Layer 7 is like a smart courier who actually opens the envelope, reads the contents, and then decides the best way to deliver it based on what's inside.

Application Load Balancer (ALB) — Layer 7 ALB operates at the application layer, meaning it can actually read the content of incoming HTTP/HTTPS requests — including URLs, headers, and cookies — before deciding where to route them. This makes it extremely powerful for web applications and APIs where different types of requests need to go to different backend services. For example, requests to /video/stream can be routed to one set of servers while requests to /user/login go to a completely different set.

Network Load Balancer (NLB) — Layer 4 NLB operates at the transport layer, meaning it routes traffic based purely on IP addresses and TCP/UDP ports without reading the actual content of the request. Because it doesn't inspect request content, it is significantly faster and can handle millions of requests per second with ultra-low latency. This makes it ideal for scenarios where raw performance matters more than intelligent routing ,such as handling the massive connection spike when millions of users simultaneously open a live IPL stream.

Large-scale streaming platforms often combine multiple layers of load balancing to achieve both high performance and intelligent request routing. NLB at the outer edge to absorb the massive volume of incoming connections, and ALB deeper in the architecture to intelligently route requests to the right backend services.

Key Takeaways

• A Load Balancer distributes incoming traffic across multiple servers to prevent any single server from becoming overloaded.

• Load balancing improves performance, availability, scalability, and reliability.

• During live events such as IPL matches, millions of user requests can be handled efficiently through load balancing.

• Modern Load Balancers continuously perform health checks and automatically redirect traffic away from failed servers.

• AWS provides Elastic Load Balancer (ELB), a managed service that helps organizations distribute traffic and maintain high availability.

• Load Balancers are one of the fundamental building blocks behind large scale streaming platforms such as JioHotstar.

In the next part we will explore How Auto Scaling Works?

References

1. AWS Elastic Load Balancing Documentation

2. Geeks for Geeks - Load Balancing in Cloud Computing

3. ByteByteGo Blog.

4. Load Balancer