Delivery and retries

How many times we try, what marks an endpoint unhealthy, and how to make your handler safe.

What counts as success

Return a 2xx within 10 seconds. Anything else — a 4xx, a 5xx, a timeout, a connection error — is a failure and gets retried.

The 10-second timeout is the reason to acknowledge first and work afterwards: read the body, verify the signature, return 200, then do the slow part on your own time. A handler that calls three other APIs before responding will eventually time out and be retried, which produces duplicates rather than reliability.

Retries

We attempt each delivery up to four times: the first immediately, then after 1s, 5s and 30s.

After the fourth failure that delivery is given up on. It stays visible in the delivery log so you can see what happened.

Unhealthy endpoints

Failures against the same endpoint accumulate. After 10 consecutive failures we:

  1. mark the endpoint unhealthy,
  2. disable it, so we stop sending, and
  3. email the workspace owner.

A single success resets the counter. Test deliveries never count toward it, so you can debug freely.

Re-enable an endpoint from Settings → Developer once the receiving end is fixed.

Be idempotent

Delivery is at-least-once. A retry after a timeout that actually succeeded on your side, or a resend, means the same event can arrive twice.

Key your handler on event_id:

async function handle(event) {
  if (await seen(event.event_id)) return;   // already processed
  await process(event);
  await remember(event.event_id);
}

event_id is stable for one logical event across retries and across endpoints. id changes per delivery attempt, so it is the wrong key for deduplication — use it for tracing a single attempt in your logs.

Ordering

Events are not ordered. message.sent for a second message can arrive before the first, and conversation.ended can beat a message.sent that preceded it.

Where order matters, use the timestamps inside data rather than arrival order.

URL rules

We re-validate the destination on every delivery, not only when you save it:

  • https is required in production.
  • Localhost, private ranges, link-local and cloud metadata addresses are rejected.

An endpoint that starts resolving to a private address — for instance because a DNS record changed — stops receiving deliveries.

To develop locally, use a tunnel that gives you a public https URL.