The Domino Effect: How a Tiny Autoscaler Blind Spot Took Down GitHub for 7 Hours

On August 17, 2026, GitHub suffered a massive 7-hour outage ( IST Aug 17 18:58 - 02:45 Aug 18). If you were trying to review pull requests, trigger GitHub Actions, or write code with GitHub Copilot that day, you likely hit a wall. Web and API error rates surged to 20%, and archive downloads plummeted with 50% failure rates.

In a candid post-mortem The August 17 outage, and the work ahead, GitHub’s CTO Vlad Fedorov explained that exponential platform growth had exposed critical architectural bottlenecks.

Note: Most of the technical details, component interactions, and cascading timeline details discussed in this article are inferred and synthesized from the official GitHub Status incident report and the GitHub CTO's post-mortem analysis.

To understand how one tiny misconfiguration brought down a global platform, we first need to look at how modern cloud platforms are built under the hood—from the edge load balancers down to the internal service mesh.

Part 1: The Players- How Modern Microservices-Based Cloud Architecture Works

Before diving into the crash, let's explore the core architectural building blocks that power large-scale applications like GitHub. Be aware, this is an oversimplified version of the architecture meant to explain the outage.

1. HAProxy: The Front Door, Load Balancer & SSL Termination

When millions of developers connect to github.com, their requests don't hit the application code directly. They hit an edge gateway powered by a reverse proxy and load balancer like HAProxy.

HAProxy performs several critical jobs:

• Reverse Proxying & Routing: It accepts incoming internet traffic and directs it to the appropriate backend services deep inside the data center.

• Load Balancing: Think of HAProxy like a traffic director at a busy airport terminal. It distributes incoming requests evenly across hundreds of healthy servers so no single machine gets overloaded.

• SSL/TLS Termination: Encrypting and decrypting HTTPS traffic takes heavy computational power. HAProxy handles the SSL decryption at the edge ("terminating" the SSL connection), passing lighter, unencrypted or internally re-encrypted requests into the internal network.

  
  

2. Istio Service Mesh & Sidecars: The Internal City

Once traffic passes the front door, it enters a massive cluster of microservices. Managing thousands of services talking to each other is chaotic. That’s why platforms use a Service Mesh like Istio.

Istio offloads networking duties from developers using the Sidecar Container pattern:

• What is a Sidecar? In Kubernetes, your application runs in a "Pod." Istio injects a lightweight helper container (an Envoy proxy) directly into that same Pod, riding alongside your main application like a motorcycle sidecar.

• Intelligent Routing: The sidecar intercepts all inbound and outbound calls, deciding which microservice instance to send requests to.

• mTLS Encryption: It automatically encrypts service-to-service communication behind the firewall with mutual TLS certificates.

• Telemetry & Observability: The sidecar silently monitors latency, request counts, and error rates, feeding metrics back to engineers without requiring custom logging code in the app.

3. Authentication Tokens: The Digital Keys

When you interact with automated tools like GitHub Copilot in your IDE, the editor doesn't ask you to type your password every three seconds. Instead, it uses Tokens—short-lived, cryptographically signed digital keys.

Your IDE contacts a dedicated Token Service to exchange credentials for a temporary access token. If that token expires or fails, the client must reach out and request a new one before continuing.

4. Autoscaling: Managing Demand

Traffic changes dynamically throughout the day. Cloud platforms rely on Autoscalers (like Kubernetes Horizontal Pod Autoscalers) to monitor resource usage and automatically spin up more Pods during surges.

Autoscaling can be:

• Reactive: Waiting for metrics like CPU or memory to spike before adding instances.

• Proactive: Scaling resources up in advance based on predictable daily schedules. (I covered how to build this in my article on Kubernetes time-based scaling of deployments with a Python client).

Part 2: The Domino Effect (How the Pieces Collapsed)

Now that we understand the architecture, let's trace how a failure cascaded through every single one of these layers on August 17.

Step 1: The Autoscaler Blind Spot

A massive surge in traffic hit GitHub's Central US data center. Naturally, the autoscaler was expected to add more capacity. However, the autoscaling policy was misconfigured to watch only the main host application container. The main app container was idling at normal CPU levels, so the autoscaler assumed everything was healthy and did not scale up. But sitting right next to it, the Istio sidecar proxy hit its hard concurrency limit. Because the autoscaler was blind to the sidecar's metrics, no new Pods were created.

Step 2: The Silent Sidecar Failure

Suffocating under the traffic spike, the Istio sidecars began failing to keep up. While the application code was running fine, its network lifeline was buckling.

Step 3: Network Saturation Takes Down HAProxy — Then Retries Make It Worse

The sidecar-level failure spread pod-to-pod, and the resulting pressure saturated the network path into the edge HAProxy load balancers. Four critical HAProxy nodes in the Central US data center exhausted their internal connection flow limits — not primarily because of a retry flood, but because of that underlying network saturation.

Optimistic retry logic from upstream clients then compounded the problem, adding extra load onto gateways that were already failing and making recovery harder. Because these specific HAProxy nodes served the gateway authentication path, logins and token validation broke down widely across GitHub's services running in that region.

With Central US degraded, engineers failed traffic over to the Northern Virginia data center, which began serving requests successfully while the network issue in Central US was debugged.

Step 4: A Second Failure — The Copilot Retry Storm in Northern Virginia

Once traffic had shifted to Northern Virginia, a separate problem emerged there: delayed replies from a single internal endpoint exposed a latent retry bug in VS Code, which amplified traffic roughly 10x instead of backing off. Traffic to the Copilot Token Service exploded — spiking from a standard 7,000–9,000 requests per second (RPS) to an overwhelming 70,000–100,000 RPS. This self-inflicted surge is why Copilot lagged far behind the rest of GitHub's recovery: most services were back by 16:36 UTC and Actions by around 18:03 UTC, but the Copilot Token Service didn't fully stabilize until 21:02 UTC — nearly five hours later, because of this second, independent failure layered on top of the failover.

Part 3: Regional Failover & The Road to Recovery

To break the loop and restore services, GitHub engineers executed a multi-step recovery plan:

1. Regional Failover: Failing traffic was shifted away from the saturated Central US data center to the Northern Virginia data center.

2. Emergency Throttling: Engineers temporarily reduced gateway retry logic via an emergency code pull request.

3. Blocking Inbound Floods: Edge load balancers were instructed to return 403 Forbidden errors to inbound Copilot token requests, breaking the VS Code retry loop and giving backend servers breathing room.

4. Gradual Ramp-Up: Once internal queues cleared, traffic was slowly re-enabled site-by-site until all systems stabilized.

   
   

The Big Takeaway

The August 17 incident is a textbook example of a modern distributed systems failure. It demonstrates that as architectures evolve:

1. Autoscaling must be holistic: If you run sidecars, your autoscalers must monitor the proxy's concurrency and connection limits, not just the application container's CPU.

2. Client code and client request throttling matter as much as backend scale strategy: A single missing backoff mechanism in a desktop client can turn a brief backend hiccup into a 100,000 RPS avalanche.

   
   

#GitHubOutage #Kubernetes #Istio #ServiceMesh #DevOps #SRE #HAProxy #LoadBalancer #CloudArchitecture #SystemDesign