What 5xx means compared with 4xx for webhook retries#
A 4xx says the request was wrong. A 5xx says the request may have been fine and the server failed anyway. To a webhook sender that is the clearest signal there is to try again later, so 5xx answers are retried by nearly every provider that retries at all.
- Stripe keeps retrying a failed live-mode delivery for up to three days, with growing gaps between attempts.
- Slack retries a failed event up to three times.
- GitHub does not retry by itself. A failed delivery stays failed until you redeliver it.
Two things follow. A handler that fails halfway and is then retried must be safe to run twice, so key your work on the event's id. And for a sender that never retries, the copy of the request you captured is the only one you will get.
| Code | Meaning | Who produced it |
|---|---|---|
500 | Internal Server Error: the handler crashed. | Your application |
502 | Bad Gateway: the proxy got no usable answer. | A proxy, tunnel or load balancer |
503 | Service Unavailable: overloaded or down for now. | Your application or its platform |
504 | Gateway Timeout: the app took too long to answer. | A proxy, tunnel or load balancer |
520 | Unknown error from the origin (Cloudflare). | Cloudflare, in front of your server |
500 Internal Server Error: finding the crash in your handler#
A 500 is an exception nobody caught. The response rarely says more than "internal server problem", on purpose, so the answer is in your server's log and not in the sender's dashboard. In webhook handlers the usual causes are:
- Reading a field the event doesn't have, such as a new event type or an empty body.
- A signing secret or API key missing from the environment.
- A database or downstream API that was down for a moment.
- Parsing the body as JSON when the provider sent form data.
The quickest way to find it is to stop waiting for the provider. Capture the request once, then send that exact request to your local server as often as you need, with the stack trace in front of you:
# Catch the event on an always-on endpoint
npx otterkit webhook --daemon --subdomain stripe-dev
# Send the latest captured request to your handler
npx otterkit replay stripe-dev --target 127.0.0.1:3000replay exits with 0 whenever your server answers, even with a 500. Add --json for a machine-readable response, so a script or an agent can loop on it until the handler returns 200.
503 and 504: timeouts and overloaded agents#
503 Service Unavailable means the server can't take the request right now: it is restarting, deploying or out of capacity. It can include a Retry-After header. 504 Gateway Timeout comes from something in front of your app, a proxy that waited for an answer and stopped waiting.
Agent handlers run into both. A model call or a chain of tool calls inside the request can take longer than any proxy or sender will wait, and a burst of events can tie up every worker at once.
Queue the work instead#
Don't do the slow part inside the request. Accept the event, answer 200 or 202, and process it from a queue. With an OtterKit webhook endpoint the accepting is done for you: the endpoint answers the sender itself within milliseconds, whether or not your machine is on, and your agent collects the event when it is ready.
# Block until the next matching request lands, then handle it
npx otterkit await stripe-dev --method POST --path /webhook520 and other proxy errors#
Code 520 is not part of HTTP. Cloudflare returns it when your origin server sent back something it could not understand: an empty reply, a dropped connection or malformed headers. Its neighbors are just as specific:
| Code | What Cloudflare saw |
|---|---|
520 | The origin returned an empty, unknown or unexpected response. |
521 | The origin refused the connection. The web server is down or blocking Cloudflare. |
522 | The connection to the origin timed out before it was established. |
524 | The connection worked, but the origin took too long to answer. |
These are written by the proxy, so your application log shows nothing for them. Look at the origin instead: was the process running, and did it close the connection or crash while it was answering?
A tunnel is a proxy too, and it fails the same way. An OtterKit tunnel answers 502 when it is disconnected or nothing is listening on the local port, and 504 when your local server takes longer than 30 seconds. If a sender reports either one for a tunnel URL, check that the dev server is up and answering in time.
Live endpoints that return each 5xx code#
To see what your sender, your retry code or your agent does with a server error, give it a URL that fails on purpose. The status can be anything from 100 to 599:
npx otterkit login
npx otterkit webhook --respond 500
npx otterkit webhook --respond 503 --respond-header 'Retry-After: 60'
npx otterkit webhook --respond 504
npx otterkit webhook --respond 520
# Fail one path and leave the rest alone
npx otterkit webhook --respond-match path=/flaky --respond 500Real outages are rarely total. A response script can fail some requests and pass the rest, which is closer to what retry logic has to survive:
// Fail about 30% of requests to exercise the sender's retries
export default {
respond(req) {
if (Math.random() < 0.3) return { status: 500, body: '{"error":"chaos"}' };
return null; // everything else gets the default response
},
};npx otterkit webhook --respond-script @respond.jsEach endpoint costs 1 credit ($0.01) per hour it is live, and new accounts get 10 free credits. Custom responses are covered in the webhooks guide.
Capturing and replaying the failed request locally#
Every request to an OtterKit endpoint or tunnel is recorded with its headers, body and the status it was answered with. Find the failures, then replay one into your handler:
# Everything that was answered with a 5xx
npx otterkit inspect stripe-dev --status 5xx
# Replay request #3, changing one field to test the fix
npx otterkit replay stripe-dev --index 3 --target 127.0.0.1:3000 --set data.amount=999