Skip to content

Mobile Premier League · Performance · Reliability

Performance Engineering at IPL Scale

How do you know a system survives a peak you can't rehearse?

Context

IPL runs once a year, and it produces the single largest traffic peak the Fantasy and Payments platforms see. I owned performance readiness for that peak and for the other peak events through the year — the work of finding out, before the event, whether the systems would hold.

The Problem

A peak like this cannot be rehearsed. There is no dress run at real scale, and the traffic shape on the day — bursts around toss time, match start, and key in-game moments — does not resemble a steady load. Whatever conclusions exist about capacity have to come from tests built beforehand, against a workload that has to be modelled rather than replayed from a quieter day.

The platforms were also not static. Over the years I covered a cloud provider migration, a managed Kubernetes migration, and a message broker migration, each of which reset what "known good" capacity meant and had to be revalidated on its own terms.

How It Worked Before

Each migration or GC change shipped with functional sign-off. Whether it held up under peak load was a separate, unanswered question until someone went and tested it.

A service passing functional tests says nothing about its behaviour at ten times the concurrency, or with a database on a different cloud, or with one pod gone.

The Engineering Question

What does this system actually do at peak load, under real infrastructure change, with something failing partway through — and how do you find that out before the peak happens rather than during it?

Architecture

Load generatorWorkload modelled on observed peak traffic shape, not a flat request rateLoad generatorTarget servicesFantasy and payments paths, exercised as a user journey rather than per endpointTarget servicesData layerDatabases, caches and queues — usually where the real ceiling isData layerObservabilityLatency, throughput, errors and saturation captured across every layerObservabilityFault injectionPod kill and latency injection, to test behaviour under partial failureFault injectionCapacity findingsWhat breaks first, at what load, and what it takes to move that ceilingCapacity findings
Figure 01 — Load and observability path

A distributed load generator drives traffic at the target services. Those services depend on databases, caches and message queues. Metrics from every layer flow to an observability stack, which is where capacity and bottleneck findings come from.

Fault injection targets the same services as the load path rather than sitting off to the side, because the finding that matters is not "the service is fast" or "the service survives a pod kill" in isolation — it is what happens when both are true at once.

Technical Decisions

Why model the workload instead of replaying traffic?

A replay reproduces last year's peak. A model of the traffic shape — bursts around toss, match start, and key in-game moments — lets you ask what happens at twice that peak, which is the question that actually matters before an event you can't rehearse.

Why measure p99 and not averages?

An average hides the users who are actually having a bad time. At peak, the tail is the product — the users who see a five-second payment are the ones who complain, and an average latency number can look fine while that is happening.

Why chaos testing when load testing already passed?

Passing at peak load with every node healthy says nothing about peak load with one node gone, which is the realistic case during a long live event. Chaos testing on GKE with Chaos Mesh — pod kill and HTTP latency and error injection — tests the failure mode that load testing alone can't reach.

Outcomes

Fantasy traffic validated before a peak event

Unverified~1M RPM

Overall Fantasy traffic was validated up to roughly 1M requests per minute, and the payments path up to roughly 60 transactions per second, ahead of peak events rather than being discovered during one. The same validation approach carried through a cloud provider migration, including database latency investigations on the new infrastructure; a Kubernetes migration, benchmarked for latency, CPU and memory under load; a message broker migration, benchmarked for broker and cluster capacity; and a comparison of G1GC against ZGC on Java 8 and 17, looking at pause times, throughput, and CPU and memory usage.

What I'd Change Today

The workload model was built from observed traffic shape, which means it is only as good as what has already been observed. It represents last year's peak scaled up, not a genuinely new traffic pattern — if a future event drives load in a shape nobody has seen yet, the model would not anticipate it, only fail to match it after the fact.

I would also push chaos testing further into the regular cycle rather than around peak events specifically. Fault injection tied to a small number of high-stakes windows tells you the system survives failure near a peak; it tells you much less about ordinary weeks, which is where most failures actually happen.