For its first 62 hours, our AI feature never called the AI

cookd has exactly one place where a language model is called: an Edge Function that takes your Claude usage statistics and returns a card with a headline and a body, written by a character who diagnoses developers for burning through their limit. It is the thing people share.

The function was first committed on 10 June. It was fixed 62 hours later, on 12 June, and the fix commit opens:

Qwen bridge 500s on every completion upstream; all roasts were the canned fallback.

For those 62 hours, the model provider behind it returned 500 on every completion. Both configured models failed on every request. The function caught each failure, logged it, and returned one of twenty hand-written cards. It returned 200 every time, fast, with well-formed JSON in the right shape. By every measure a health check would use, the feature was up.

It happened in the product's first three days, while the app around it was still being built, so the damage was small. The shape is worth writing up anyway, because it is what a good fallback does by default, and because nothing we have built since would catch it faster.

Two facts that decide everything

The function cannot visibly fail. Tapping "roast me" is the product's core interaction. "The model is down" is not an acceptable screen.

Its output is untrusted. Whatever comes back is text from a probabilistic system that will be shown to the user, written to the database, and possibly published under their name.

Everything below follows from holding both at once.

A fallback in the product's voice

When the model is unreachable, the function returns one of twenty canned roasts, each about the roast engine failing:

export const CANNED_ROASTS = [
  { headline: "broke the f*cking roast machine",
    body: "You managed to get rate-limited by the thing that roasts you. …" },
  { headline: "even the fallback roast is embarrassed",
    body: "The roast engine crashed looking at your numbers. That's never happened before. …" },
  …
];

A generic error breaks the fiction. A fallback that is about the failure, in the same voice, reads as a joke about having broken the machine, which is the product's whole premise. Twenty of them, so a repeat visitor does not see the same card twice in a row.

That is exactly why it hid the outage.

Why a good fallback hides a dead provider

Each decision here was defensible on its own:

• The output was plausible. The canned cards are in voice. There had never been a working version to compare against.
• The errors were logged. Both failures called console.error. The information existed. Logging is not monitoring, and nobody opens a log they have no reason to open.
• The function returned 200. Latency was low, the JSON was well-formed, nothing threw. The degradation lived entirely inside a successful response.
• The fallback was good. An empty card or an apology would have been noticed in an hour.

Call it the silencer effect. Graceful degradation turns a loud failure into a quiet one. That is right for the user. It is right for the operator only if something else now makes the noise. Otherwise the quality of the fallback sets the length of the outage: the better it is, the longer you stay broken.

"Provider-agnostic" was a claim, and the swap tested it

The first integration went through a hosted bridge exposing an OpenAI-compatible endpoint, chosen so the provider could change without code. The header comment said so:

// LLM is provider-agnostic via an OpenAI-compatible chat endpoint (Qwen now, Anthropic later
// by changing QWEN_* env to the new provider's base/model/key).

When the swap came, environment variables were not enough:

-    const res = await fetch(`${Deno.env.get("QWEN_BASE_URL")}/chat/completions`, {
+    const res = await fetch("https://api.anthropic.com/v1/messages", {
-        Authorization: `Bearer ${Deno.env.get("QWEN_API_KEY")}`,
+        "x-api-key": Deno.env.get("ANTHROPIC_API_KEY") ?? "",
+        "anthropic-version": "2023-06-01",
-      body: JSON.stringify({ model, messages, temperature: 0.7, max_tokens: 220 }),
+      body: JSON.stringify({ model, system: SYSTEM_PROMPT, messages, … }),
-    const content = data?.choices?.[0]?.message?.content;
+    const content = data?.content?.[0]?.text;

The endpoint, the auth header, the request body and the response path all changed. The system prompt became a top-level parameter instead of a role: "system" message. It was nine insertions and ten deletions in one file, which is small. It is also not what the comment promised. The thin design bought us a single file we knew to edit, and nothing more. The only way to know how shallow a compatibility layer is, is to do the swap once.

The commit's last line matters most: "live roast verified end-to-end." That had never been done for the original integration, because a broken integration behind a good fallback never looks unverified.

The call

const TIMEOUT_MS = 25_000;

async function callLLM(messages: unknown[], model: string, system: string): Promise<string> {
  const ctrl = new AbortController();
  const timer = setTimeout(() => ctrl.abort(), TIMEOUT_MS);
  try {
    const res = await fetch("https://api.anthropic.com/v1/messages", {
      method: "POST", signal: ctrl.signal, headers: { … },
      body: JSON.stringify({ model, system, messages, temperature: 0.9, max_tokens: 300 }),
    });
    if (!res.ok) throw new Error("LLM " + res.status + ": " + await res.text());
    const data = await res.json();
    const content = data?.content?.[0]?.text;
    if (typeof content !== "string" || !content.trim()) throw new Error("empty LLM content");
    return content;
  } finally {
    clearTimeout(timer);
  }
}

Three things here are not optional. An explicit timeout, because a hung request in a serverless function makes the user wait and bills you for it. A non-OK response thrown with its body attached, because every failure this function has had was diagnosed from that string. And an empty successful response thrown too, because a 200 with no text is a failure that does not look like one.

max_tokens: 300 is a product limit: the headline targets 60 characters and the body 280, one tweet. Temperature is 0.9 because the output is supposed to be funny and varied. (It was 0.7 when the provider was swapped and was raised on 20 June, with the prompt rewrite.) Extraction paths elsewhere in the codebase run much colder. The two uses want opposite things and should not share a default.

A parser that never throws

The system prompt asks, in capitals, for minified JSON with no code fence. Models mostly comply. Mostly is not a contract:

export function parseRoast(raw: string, handle: string): { headline: string; body: string } {
  const cleaned = raw.replace(/^```(?:json)?/i, "").replace(/```$/, "").trim();
  try {
    const o = JSON.parse(cleaned);
    if (typeof o.headline === "string" && typeof o.body === "string") {
      return { headline: o.headline.slice(0, 80), body: o.body.slice(0, 320) };
    }
  } catch { /* fall through */ }
  return { headline: "@" + handle + " broke the format too", body: cleaned.slice(0, 320) };
}

It strips a fence the model was told not to use. It checks the shape after parsing, because valid JSON with the wrong keys is a different failure from invalid JSON. And it never throws. If the text is not JSON, it keeps the model's words as the body and makes the malfunction part of the character.

The caps (80 and 320) are looser than the prompt's targets (60 and 280). The prompt sets the aim; the parser enforces the ceiling. A model that overshoots by ten characters should not break a card.

Keeping unparsed text is a real choice. It never wastes a generation, and it also lets less-structured output reach the user. What makes it defensible is the next layer.

Screening output before it exists anywhere

The moderation work added a prohibited-term guard as database triggers on the content tables. The design record said the roast function needed no check of its own, because the trigger fires when a user publishes a roast.

True for publishing, false for everything before it. The implementation commit explains:

index.ts service-role-inserts into roast_log and the client renders the card, both before any trigger can fire. So a slur-bearing roast was displayed and stored with no posts row involved.

A user who never publishes is never covered. So the check runs right after parsing, before any write:

try {
  const { data: termRows, error: termErr } = await admin.from("banned_terms").select("term");
  if (termErr) throw termErr;
  const terms = (termRows ?? []).map((r: { term: string }) => r.term);
  if (containsBannedTerm(card.headline + " " + card.body, terms)) {
    card = fallbackRoast(handle);
  }
} catch (e) {
  // Fail OPEN, deliberately. The guard is a floor, not the boundary: the posts
  // trigger still blocks publication no matter what happens here. Failing
  // closed would take the app's core feature down whenever this read hiccups.
  console.error("banned_terms lookup failed, skipping output screen", e);
}

It fails open, and the reason sits next to the catch. That is only defensible because an unskippable guard sits behind it. Someone who does not know that will read the fallthrough as sloppiness and tighten it.

One divergence, stated plainly. The decision record rejected giving this function a service-role database client, so the one function handling untrusted model output would not gain privileges. The shipped code reads the term table with the service-role client that was already there for the rate limit and the log insert. No new privileged surface was added, which is the intent of the rule. It is not what the rule literally said.

The TypeScript screen mirrors the SQL one, and carries the same framing in its comments: it is a floor, not a filter.

A rate limit read from the user row

export function effectiveLimit(userLimit: number | null | undefined): number {
  return typeof userLimit === "number" && userLimit > 0 ? userLimit : ROAST_DAILY_LIMIT;
}

The daily limit is a column on the user row with a code default, pinned by a BEFORE UPDATE trigger that reverts any client change. That is this codebase's pattern for a server-owned entitlement the UI needs to read. It exists because the user table's RLS is row-scoped, not column-scoped, so every new protected column is client-writable until a trigger pins it. The 429 message states the account's real limit rather than a hardcoded number.

Enforcement is a count over the log table for the last 24 hours: deliberately simple, and sized for the product's volume.

The prompt is code

const PROMPT_HAIKU_V1 = `…`;   // compact, rule-forward
const PROMPT_SONNET_V1 = `…`;  // long, character-forward

// ACTIVE PROMPT — change this line to swap prompt versions
const SYSTEM_PROMPT = PROMPT_HAIKU_V1;

Two full prompts in source, one line to switch. It diffs, it ships atomically with the parser that depends on it, and a change gets a code review. For one prompt with two variants, a prompt store would add flexibility nobody needs and lose the property that prompt and parser change together. The few-shot examples are checked in too, each showing a different structure, because four examples of one shape teach the shape, not the range.

What it gives up

• Fallback cards look real. That is the point for users, and it means the path that produced a card has to be measured separately from whether a card came back. A good fallback never shows up in an error rate.
• Quality is judged by reading. For "is this funny", human judgement is the real bar. A regression signal does not appear on its own; it has to be built from fixed inputs and a rubric.
• A simple rate limiter. A count over a log table, chosen over a dedicated limiter for simplicity.
• The term screen is a floor. One layer, with the publication trigger behind it.
• No streaming. The user waits for the whole card, up to 25 s in the worst case. For 280 characters that is acceptable against the complexity streaming would add to parse-then-screen.

What we would do differently

• Record which path ran. One boolean on the log row (from_model) turns a hidden outage into a one-line query, and answers the question the feature will eventually raise: do the canned cards perform worse?
• Alert on the ratio, not the error. Errors were logged and that changed nothing. The signal is the fraction of requests that took the degraded path. During these 62 hours it was 100%.
• Verify a real generation before trusting any path you have never seen work. One end-to-end check on 10 June would have made this a ten-minute bug.
• Give prompt changes a regression signal. Twenty golden stat blocks and a rubric turn "this one reads funnier" into a comparison two people can agree on.

Health is which branch ran

For anything with a degraded path, "did it respond" is the wrong health check. The function was fast, correct in shape, and error-free for 62 hours while its core value was absent.

Write the failure into the product's voice. Then put a number somewhere that says how often the joke is the only thing running.