Retries and timeouts

odnoga handles vendor-side retries and fallback. You still need to handle network / 429 / 5xx on your side — but only those.

Rules

• Never retry 4xx other than 408 and 429. They are deterministic; retrying just wastes money.
• Always retry network errors, 408, 429, 5xx. Exponential backoff, jitter, cap at 3 attempts.
• Honor Retry-After when present (capacity and rate-limit answers carry it). x-ratelimit-reset is only sent when your workspace has a budget window; it is when that budget resets, not a retry hint.
• Set a hard request timeout (e.g. 60s). Don't rely on the platform.

Helper

import { callodnoga, OdnogaError } from './odnoga.ts';

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

export async function callWithRetry<T>(
  path: string, body: unknown, opts: { endUser: string; timeoutMs?: number } = { endUser: 'anonymous' },
) {
  const timeoutMs = opts.timeoutMs ?? 60_000;
  let lastErr: unknown;

  for (let attempt = 0; attempt < 3; attempt++) {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), timeoutMs);
    try {
      return await callodnoga<T>(path, body, { endUser: opts.endUser, signal: ctrl.signal });
    } catch (e) {
      lastErr = e;
      const status = e instanceof OdnogaError ? e.status : 0;
      const retryable = status === 0 || RETRYABLE.has(status);
      if (!retryable) throw e;
      // A 503 capacity_unavailable says how long to wait; honour it.
      const retryAfter = e instanceof OdnogaError ? e.retryAfter : null;
      const backoff = retryAfter && retryAfter > 0
        ? retryAfter * 1000
        : Math.min(8000, 250 * 2 ** attempt) + Math.random() * 250;
      await new Promise((r) => setTimeout(r, backoff));
    } finally {
      clearTimeout(t);
    }
  }
  throw lastErr;
}

Idempotency

For a non-streamed request, send an idempotency key (any unique string per logical call) so a retry after a dropped connection never runs — or bills — twice. odnoga reads it from any of these headers: Idempotency-Key, x-odnoga-idempotency-key or x-airouter-idempotency-key.

Streamed requests are not deduplicated. A stream cannot be replayed, so the key is ignored when "stream": true; a retried stream is a new call and is billed as one. If a duplicate would hurt, retry a stream only when you received no frames at all.

How long a single call may run

odnoga gives every call to the model 300 seconds — the same upstream deadline for streamed and non-streamed calls. If the model has not answered by then, odnoga abandons that attempt and moves to the next model in your fallback chain; when no model answers, you get 503 vendor_unavailable. You can ask for a shorter deadline per request with x-airouter-timeout-ms (milliseconds; values are clamped to 1 000–300 000), so a fast use case fails over quickly instead of waiting.

Separately, the network in front of odnoga closes any response that sends no bytes for about 150 seconds. That is what decides which limit you actually hit:

Streaming does not buy more than 300 seconds; it removes the ~150 s idle cut-off so you can use all of them.

This bites hardest on reasoning models at high effort, which think for a long time before emitting anything. A prompt that returns in 40 s at medium effort can exceed the ~150 s non-streamed limit at xhigh — same model, same prompt, and no amount of budget changes it.

What to do about it

Stream it — see Streaming. The fix when a caller is genuinely waiting for the answer. Set stream: true and accumulate the frames yourself if you want one string at the end. You get the same answer and the same price, with the full 300-second deadline instead of the ~150 s idle limit.

If you only stream to escape the limit and want the answer in one piece, add the header x-odnoga-stream: final: odnoga keeps the connection alive with : keepalive comments and sends a single frame holding the complete chat.completion (content, tool calls, usage, _odnoga cost), then [DONE]. No frame stitching on your side — see Streaming → Final-only mode.

One trap when you reassemble frames: AbortSignal.timeout on fetch covers getting a response, and a streamed call gets one almost immediately. After that the signal has done its job and the body can stall indefinitely. Give the read loop its own deadline, or a stalled stream hangs until your platform kills it — which looks exactly like a slow model and bills like a success.

const res = await fetch(`${BASE}/chat/completions`, {
  method: 'POST',
  headers: { authorization: `Bearer ${KEY}`, 'content-type': 'application/json' },
  body: JSON.stringify({ model, messages, stream: true }),
});
let out = '';
for await (const frame of sseFrames(res.body!)) {
  out += frame.choices?.[0]?.delta?.content ?? '';
}

Or don't wait at all. If the work is not interactive — a nightly report, a bulk classification, a long research pass — use the Batch API. It has no duration ceiling, and it is billed at 50% of the standard rate on the vendors that deliver it. For anything on a schedule this is strictly better than making a caller sit still for three minutes.

Lower the effort. reasoning.effort is a correctness knob, not only a cost one. If high answers well enough, it also answers in time.

If your own code is serverless too

Your function has two budgets, and they are not the same one. Confusing them is the single most expensive mistake on this page.

The 150 seconds is not a limit on the work. It is a limit on making the caller wait for it. The platform's own wording is that during the wall-clock period a worker "can serve multiple requests or process background tasks" — so if nobody needs the answer in the response, nothing has to finish in 150 s.

So the rule is respond first, work after:

Deno.serve((req) => {
  // Hand back the acknowledgement immediately; the idle timeout is now met.
  const work = generateAndStore(req);          // may take minutes
  EdgeRuntime.waitUntil(work);                 // keeps the isolate alive
  return new Response(null, { status: 202 });
});

waitUntil is not optional here, and this is the part that bites. Without it the isolate counts as idle the moment the response returns and may be retired mid-run — so returning early without it loses the work by a different route than the one you were avoiding. With it, you have the full wall clock.

Two consequences worth planning for:

• Nobody sees the verdict. A cron dispatch is fire-and-forget, so the outcome has to land somewhere durable — a row, a log line, an alert. Write it when the work finishes, not when it starts, or a crash looks like a success.
• Keep the synchronous path for dry runs. When the answer is the point — a manual invocation, a preview, a test — respond synchronously and accept the shorter budget. That is what makes a dry run useful.

Interactive requests are the other case: there a human is waiting, so you cannot defer, and streaming is the answer instead.