Someone on your team drops a message: POST /api/checkout took 20,000 requests in a single hour. It reads like a performance bug, and it might be. But that number alone tells you almost nothing. Is this endpoint usually busy at that time of day? Was it one client hammering the API or thousands of users checking out normally? Did the system slow down, or did it handle the load without breaking a sweat? Before a single line of code gets touched, QA has a job to do: turn a raw number into a real diagnosis. Here is what that process actually looks like.

The Report Lands

Someone on your team drops a message: POST /api/checkout took 20,000 requests in a single hour. The timestamp is specific, the number is specific, and it lands in your queue looking like a bug.

But a request count by itself is not a diagnosis. Is this endpoint usually busy at that time of day? Was it one client retrying aggressively, or thousands of users checking out at once? Did response times hold steady, or did the system start buckling under the load? None of that is in the original report, and none of it can be assumed.

This is where QA's job actually starts. Not fixing anything yet, and not filing a ticket on faith either. The first task is turning a raw number into evidence, and the first question is the simplest one: is this actually abnormal?

Step 1: Confirm It's Actually Abnormal

Twenty thousand requests in an hour means nothing on its own. It only becomes meaningful next to a baseline. Before anything else, pull the traffic history for POST /api/checkout specifically, at the same hour, on the same day of the week, across the past several weeks. If that endpoint routinely sees 18,000 to 22,000 requests during that window, the "spike" is just Tuesday. If it normally sees 4,000, you have something worth investigating.

This is also where scope matters. A system-wide traffic dashboard can hide a single endpoint's spike entirely, and it can just as easily make a real spike look unremarkable if the rest of the API absorbed it without issue. The baseline has to be endpoint-specific, or it is not really a baseline.

One honest complication: QA does not always have the access this step implies. Not everyone has a login to Datadog or New Relic, visibility into deploy logs, or permission to query the API gateway directly. That is normal, and it is worth naming rather than working around.

What you can usually get on your own: access logs, existing dashboards, anything already surfaced in a shared monitoring tool. What you will likely need to request: deploy history around the timestamp, gateway-level metrics, or anything living in a system you do not have credentials for. Asking for that access early, as part of triage, is faster than trying to reconstruct the picture from partial data or waiting until the investigation stalls to admit you need it.

By the end of this step, you should be able to say one of two things with confidence: this volume is normal for this endpoint at this time, or it is not. Everything that follows depends on knowing which one is true.

Step 2: Characterize the Spike

Once you know the volume is genuinely abnormal, the next job is not to explain it yet. It is to describe it precisely enough that an explanation becomes possible. Four questions do most of the work here.

Was it a burst or a sustained climb? Twenty thousand requests spread evenly across the hour behaves very differently from twenty thousand requests crammed into a five-minute window. A burst points toward something mechanical, like a retry loop or a batch job firing all at once. A sustained rise over the full hour points more toward organic load or a slow-building client-side issue.

Did it come from one source or many? Check whether the traffic traces back to a single user, IP address, or API key, or whether it is spread across a large number of distinct clients. A single source hammering the endpoint suggests a misbehaving integration or a bug in one client's retry logic. Broad, distributed traffic suggests either real user demand or something closer to bot activity.

What else happened at that timestamp? Cross-reference the spike window against anything else on the calendar: a recent deploy, a scheduled cron job, a marketing email that went out, a partner's integration kicking off a sync. Spikes rarely happen in a vacuum, and the coincidence is often the answer.

Did the endpoint hold up, or did it degrade? Pull latency and error rates for that same window. An endpoint that handled 20,000 requests with flat response times and no error increase is a very different story from one where p95 latency tripled and 500s started climbing. The first suggests capacity that happens to be adequate. The second suggests a real problem, whether that is code, infrastructure, or both.

None of these four questions are separate investigations. Each answer is a piece of evidence, and none of them means much in isolation. A burst alone is not conclusive. A single source alone is not conclusive. What matters is the combination, because that combination is what tells you what kind of problem you are actually looking at, which is exactly the decision the next step is built around.

Step 3: Classify Before You Escalate

The evidence from Step 2 exists to answer one question: what kind of problem is this? The combination of pattern, source, timing, and system behavior almost always points toward one of four buckets, and each bucket has a different owner.

A sustained rise, spread across many distinct clients, with latency and error rates holding steady usually means the traffic is legitimate and the endpoint handled it fine. This is not a bug. It might be worth a note for capacity planning, but there is nothing here for a developer to fix today.

A sustained rise, spread across many clients, but paired with rising latency or errors points toward a capacity problem. The load is real, but the endpoint or the infrastructure behind it was not built to absorb it cleanly. This routes to backend or SRE, not to a bug fix in the traditional sense. It is a scaling conversation.

A sharp burst, traced back to a single user, IP, or API key, especially alongside rising error rates, usually means a client is misbehaving. This is often a retry storm: a consumer's integration failed once, then retried aggressively without backoff, multiplying one failure into thousands of requests. This routes to whoever owns that integration, internal or external, and often pairs with a fix on your side too, like adding idempotency or tightening rate limits.

A sharp burst from a single source with no correlating deploy, no scheduled job, and traffic that does not look like normal usage is worth treating as possible abuse or bot activity. This routes to security or whoever owns rate limiting, not to a feature team.

The pattern from Step 2 is what tells you which bucket you are in. Sustained plus distributed plus clean latency lands you in the first bucket. Bursty plus single source plus rising errors lands you in the third. The classification is not a separate judgment call sitting on top of the evidence. It is a direct read of it.

This step is worth taking seriously because getting it wrong is expensive. Handing a capacity problem to a developer as if it were a code bug wastes their time chasing something that is not there. Handing a retry storm to SRE as if it were a scaling issue means the actual misbehaving client keeps retrying while the real fix sits unassigned. Misrouting a spike like this does not just slow things down. It can burn a sprint on the wrong fix while the real cause stays untouched.

Step 4: Reproduce and Report

Once you know roughly what kind of problem you are dealing with, it helps to see if you can make it happen again on demand. A spike you can reproduce in staging is a spike a developer can actually debug. A spike that only exists in a slice of production logs is much harder to work with.

This does not need to be elaborate. If Step 3 pointed toward a retry storm or a capacity issue, a basic load test against POST /api/checkout that mimics the pattern you found (same burst shape, same rough request volume, same single source if that was the signature) is usually enough to confirm the theory. This is a triage step, not a tooling exercise. If your team already has Locust or Artillery scripts from other load testing work, reuse them here rather than building something new. The goal is confirmation, not a fresh benchmark.

What matters more at this stage is the report itself, because a raw request count was never actionable and a confirmed reproduction still needs to be communicated clearly. A useful report on a spike like this should include:

• Baseline versus actual. What this endpoint normally sees during that window, and what it saw this time.
• Pattern. Burst or sustained, and roughly what that shape looked like.
• Source. Single client, single API key, or distributed across many users.
• Correlating events. Any deploy, cron job, or external trigger that lined up with the timestamp.
• Impact. Whether latency or error rates moved during the spike, and by how much.
• Suspected classification. Which of the four buckets from Step 3 this looks like, and why.
• Reproduction status. Whether you were able to recreate the pattern in staging, and what happened when you did.

A report built this way tells the receiving team exactly what they are looking at before they open a single log file. It also does the routing work for them. A developer reading "single API key, burst pattern, rising 500s, reproduced in staging with the same retry shape" already knows this is a client-side retry bug before they have written a line of code. That is the difference between a report someone has to investigate from scratch and one they can act on immediately.

Step 5: Verify and Prevent Recurrence

A fix is not done just because the spike stops reproducing. Once a change goes in, re-run the same pattern you used in Step 4, the same burst shape, the same source, the same rough volume, against the fixed version of the endpoint. If the fix addressed a retry storm, confirm the retries are actually being handled correctly now, not just that the immediate symptom disappeared. It is also worth checking anything the fix touched beyond the endpoint itself, like a shared database connection pool or a cache layer, since a fix scoped too narrowly can quietly shift the same pressure somewhere else.

The other half of closing this out is making sure the next spike does not need a human to notice it. A one-off investigation is useful, but it does not scale as a strategy. Turn what you learned into something automatic: an alert on POST /api/checkout that fires when volume or error rate departs from the baseline you established in Step 1, or an automated check that runs the same reproduction test as part of regular regression coverage. If your team already has patterns for handling traffic spikes at scale, like the retry and backoff strategies covered in our post on the thundering herd effect, this is a good place to link out rather than rebuild that thinking from scratch.

The report that started this whole process showed up as a Slack message and a request count. The goal is that the next one shows up as a dashboard alert instead, hours before anyone has to ask what happened.

Conclusion

Twenty thousand requests on one endpoint is not a verdict. It is a starting point. The work between that first message and a resolved ticket is what actually separates a fixed problem from a misrouted one: checking the baseline before assuming anything is wrong, reading the pattern before guessing at the cause, and routing the fix to whoever actually owns it. None of that requires special tooling. It requires treating a request count as evidence to be interpreted, not a bug to be assigned.

The next spike on POST /api/checkout will not announce itself any more clearly than this one did. The difference is whether your team has to work it out from scratch again, or whether the alert, the baseline, and the reproduction test are already waiting for it.