Skip to main content

Spend Guards — ad-account overspend alerts + kill switch

Status: built and live (approved 2026-09-23; client update applied same day — see §10). This is the design record; the operator guide is Spend Guards. Where this page and the code differ, the code and the guide win. Requester: Brian O'Connell (ODMG). Owner: Henry Fuentes (SBD). Risk tier: 3 — the kill switch pauses live client ad spend on Meta / Google.

1. Problem​

ODMG wants two defensive layers on every ad account, so a platform glitch, a fat-fingered budget, or a broken rule cannot burn money overnight:

  1. Velocity alert (Slack, alert only). If ≥ X% (default 20%) of an ad account's daily budget is spent inside any ~1-hour window, post to Slack. The daily budget is yesterday's actual spend for that ad account (in the account's timezone), computed automatically. No action buttons — the team opens the ad account and judges.
  2. Hard-cap kill switch. Each ad account has a daily spend cap (set with headroom, e.g. $20–25k on a $15k/day account). When today's spend reaches the cap, pause every active campaign in the account and post to Slack. Damage control first; the team decides what to bring back. The cap is independent of the daily budget.

Explicit non-goals (per Brian, 2026-09-22 Slack): no human-approval button in Slack, no "keep profitable campaigns running" exceptions, no automatic un-pausing. The data model must not preclude adding exception rules later.

2. What exists (verified 2026-09-23)​

Building blockV2 todayGap
Slack credsslack.bot_token service credential (vault/env, platform-service-credentials.ts, shown in /admin/settings)Not used for alerts; only the weekly report posts, via webhook
Ad-account registryclient_ad_accounts (Supabase) + per-account settings columnsNo guard settings
Spend datamarketing_paid_acquisition (ClickHouse, daily grain), serial conveyor syncToo slow/coarse for intraday — needs its own poller
Platform writesnone; Meta OAuth scope is ads_read,business_managementNeed Meta ads_management + Google campaign mutate
V1 referenceFBApiManager::updateStatus() pauses Meta entities with per-account tokens; Slack via bot tokenV1 has no intraday polling and no Google pause

3. Design​

3.1 Spend pulse (shared by both rules)​

  • New Inngest cron every 15 min: spendGuardTick fans out one spendGuardCheckAccount event per enabled guard. Separate from the marketing conveyor so it is never starved by it. Concurrency limit 5.
  • Per account, one cheap platform call:
    • Meta: GET act_<id>/insights?date_preset=today&fields=spend,account_currency plus act_<id>?fields=timezone_name (cached on the settings row).
    • Google: GAQL SELECT metrics.cost_micros FROM customer WHERE segments.date DURING TODAY (+ customer.time_zone). cost_micros / 1e6 = dollars (a unit conversion from micros, not the forbidden cents conversion — money stays dollar decimals).
  • Store a snapshot row (ad_account, captured_at, account_date, spend_today). account_date is the calendar day in the ad account's timezone.
  • Last-hour spend = latest spend_today − spend_today of the snapshot nearest to 60 min earlier on the same account_date (if none, the first snapshot of the day, i.e. midnight = 0).
  • Platform reporting lags ~15–30 min, so end-to-end detection is ~30–45 min. Documented to the client.

3.2 Velocity alert​

  • Daily budget = yesterday's actual spend (account timezone): Meta insights?date_preset=yesterday, Google segments.date DURING YESTERDAY. Fetched once per account_date and cached on the settings row (yesterday_spend, yesterday_date); a failed fetch is not cached, so the next tick retries. An optional manual daily_budget_override replaces it.
  • If yesterday's spend is unavailable or $0 and there is no override, the velocity rule is skipped for that account and the UI shows "No baseline" — it never alerts without a baseline.
  • Fires when last_hour_spend ≥ alert_pct/100 × daily_budget.
  • Slack message: account name + platform, last-hour spend, today's spend vs daily budget and vs cap, projected end-of-day pace, link to Ads Manager / Google Ads. No buttons.
  • Cooldown: at most one velocity alert per account per 2 h.

3.3 Hard-cap kill switch​

  • Fires when spend_today ≥ hard_cap and the account is armed.
  • Mode per account: alert_only (no cap action) · dry_run (default for a new cap: posts "WOULD pause N campaigns: …", pauses nothing) · live.
  • live flow: list campaigns currently ACTIVE (Meta effective_status=ACTIVE, Google campaign.status = ENABLED) → pause each (Meta Graph batch status=PAUSED; Google campaigns:mutate status PAUSED) → record the exact list paused + per-campaign success/failure → Slack post.
  • Idempotent: at most one trip per account per account_date, enforced by a unique DB constraint on the trip event, not only by code.
  • Never auto-resume. After a trip the guard is tripped until an admin clicks Re-arm or the next account_date begins.
  • Loud failure: any pause error (e.g. token lacks ads_management) posts "⚠️ AUTO-PAUSE FAILED — act manually" with the error and the campaigns still running.

3.4 Slack delivery​

  • Native Slack Web API chat.postMessage using the slack.bot_token service credential (vault first, env fallback — existing resolver). No new token store.
  • Channel resolution: guard's channel → Spend Guard default channel (new slack.spend_guard_default_channel service-credential field) → skip + log.
  • The bot must be invited to the channel; the UI's "Send test" reveals this.

4. Data model (Supabase migration)​

  • spend_guard_settings — ad_account_id (FK client_ad_accounts.id, unique), enabled, mode (alert_only|dry_run|live, default dry_run), daily_budget_override numeric(dollars) null, yesterday_spend numeric(dollars) null + yesterday_date (auto baseline cache), alert_pct numeric default 20, hard_cap numeric null (> 0, no relation to the budget), slack_channel_id, slack_channel_name, timezone, currency, state (armed|tripped), tripped_on date, tripped_at, denormalized last-pulse fields (last_checked_at, last_account_date, last_spend_today, last_hour_spend, last_error), updated_by, updated_at.
  • spend_guard_snapshots — ad_account_id, captured_at, account_date, spend_today, currency. Index (ad_account_id, captured_at desc). Prune

    7 days in the tick.

  • spend_guard_events — id, ad_account_id, account_date, kind (velocity_alert|cap_trip|pause_dry_run|pause_executed|pause_failed|rearm|settings_changed), payload jsonb (spend figures, campaign ids paused/failed, error), actor (system or user email), created_at. Partial unique index on (ad_account_id, account_date) WHERE kind = 'cap_trip'.
  • Service credential catalog: add slack.spend_guard_default_channel.

5. UI​

5.1 /admin/settings → Slack card (extend existing)​

  • Existing bot-token field (vault). New: Test connection (auth.test → shows workspace + bot name, or the error).
  • New Spend Guard default channel picker: lists channels via conversations.list (public + private the bot is in) with a manual channel-ID fallback; Send test message button.

5.2 /admin/spend-guards (new page; nav under Admin, super_admin only)​

  • Header summary: guarded accounts, tripped today, alerts today, last tick time.
  • Table, one row per ad account (all bound accounts; unguarded rows show "Off"): Client · Account (platform icon) · Mode badge · Daily budget · Alert % · Hard cap · Today's spend (with % of cap bar) · Last-hour spend · State (Armed / Tripped at 03:12) · Channel · Last checked.
  • Filters: client, platform, mode, state. Search by account name.
  • Row click → edit drawer:
    • Enable toggle; Mode segmented control (Alert only / Dry run / Live) — switching to Live requires a confirm dialog naming the account and cap.
    • Daily budget: read-only "Yesterday's spend: $X (auto)" (or "No baseline"), with an optional manual override field. Alert % (default 20). Hard cap (suggested 1.5× the trailing 7-day average spend). Validation: override > 0 when set, 1 ≤ alert % ≤ 100, cap > 0 and required for Dry run / Live. The cap is independent of the budget.
    • Slack channel picker (defaults to the Spend Guard default channel) + Send test.
    • Check write access button (Meta /me/permissions for ads_management; Google a validate-only mutate) → Ready / Missing.
    • Read-only: timezone, currency.
  • Tripped row actions: View paused campaigns (list + copy IDs) and Re-arm (confirm).
  • Tab Activity: spend_guard_events feed, filterable by account/kind.

5.3 Clients → ad-account connector (Daily Spend Cap)​

  • On /admin/clients/[slug] each Meta/Google ad account's connector row (ConnectorCard.tsx, next to Validated / Pause / Campaigns) gets a Daily Spend Cap row: inline editable $ cap, mode (Off / Alert only / Dry run / Live — Live asks for a typed confirmation), and Slack channel, plus the velocity budget label and a link to the Spend Guards drawer.
  • It writes the same spend_guard_settings row through the same super_admin-guarded PUT /api/admin/spend-guards/:id — one source of truth. /admin/spend-guards remains the fleet overview + activity view.
  • Collapsed rows show a Guarded / Tripped badge. (/admin/ad-accounts is a redirect to /admin/clients, so the badge lives here.) Only rendered for super_admin; the API 403s for every other role.

6. Security & invariants​

  1. Only super_admin can read/write guard settings, re-arm, or send tests (/api/admin/spend-guards/* each call their own require* guard).
  2. A guard only ever acts on its own ad_account_id using that account's own resolved credentials — no cross-tenant credential use.
  3. dry_run and alert_only must never call a platform write client (tested).
  4. At most one trip per account per account-day (DB-enforced, tested).
  5. The system never un-pauses anything.
  6. Money is dollar decimals end to end; Google micros ÷ 1e6 only at the edge.
  7. No tokens in logs, events, or Slack messages.

7. Tests (confidence-per-minute)​

  • Tier A: dry-run/alert-only never writes; trip idempotency under concurrent ticks; re-arm/next-day reset; API routes reject non-super-admin; pause failure produces a pause_failed event and loud Slack post.
  • Tier B: last-hour window math (midnight rollover, timezone, sparse snapshots); velocity cooldown; Google micros conversion.
  • Tier C: typecheck, build, page smoke.

8. Rollout​

  1. Ship pulse + velocity alerts + UI + Slack settings (no campaign writes).
  2. Ship kill switch with every cap defaulting to dry run; independent review before merge (Tier 3).
  3. Verify Meta tokens carry ads_management (Check write access); re-connect accounts that do not.
  4. ODMG reviews a few days of dry-run posts, then flips accounts to Live per account.

9. Open inputs from ODMG​

  • Daily budget + hard cap per ad account (UI pre-fills suggestions).
  • Slack channel(s) for alerts; invite the Launchpad bot to them.

10. Client update (ODMG, 2026-09-23)​

  1. Daily budget for the velocity % = yesterday's actual spend (account timezone), automatic. Optional nullable manual override on the settings row. No baseline (unavailable or $0) ⇒ velocity skipped, UI "No baseline".
  2. The daily spend cap is editable from the Clients section (ad-account connector row), writing the same settings row via the same API as /admin/spend-guards.
  3. All caps start in dry_run (unchanged). The cap rule is independent of the daily budget.

Implementation notes (build, 2026-09-23):

  • One LIVE cap trip (cap_trip) per account per account-day is DB-enforced, so a same-day re-arm will not trip again until the next account day. Dry-run trips are a separate kind (cap_trip_dry, own one-per-day index) and never set the guard to tripped, so switching Dry run → Live the same day trips on the next check.
  • After the trip claim, nothing is allowed to abort the run: a failure marking the guard tripped or resolving the Slack channel still pauses (Live) and still attempts the post; undeliverable cap messages go to console.error in full.
  • Live requires a reachable Slack channel (guard channel or a valid default). A Live cap at/below today's spend needs an explicit confirmation (confirm_cap_below_spend). One guard per underlying platform account (instagram→meta, act_/dashes normalized) across bindings and clients (409).
  • slack.bot_token and slack.spend_guard_default_channel are super_admin-only in the service-credentials API; the default channel must be a channel ID.
  • Slack text escapes &, <, > in account, client and campaign names and in platform error strings.
  • alert_only never evaluates the cap.
  • Last-hour baseline: the snapshot nearest to 60 min earlier is accepted only if it is 30–90 min old. With no such snapshot, midnight ($0) is used only within the first 90 min of the account day; otherwise (sparse snapshots, a guard enabled mid-day) the velocity rule does not evaluate that tick, so it never reports all of today as "the last hour".
  • Arming Live (switching into it, or re-enabling a disabled Live guard) requires a typed account-name confirmation in the UI; the API refuses to arm Live without confirm_live.

11. Campaign scope (ODMG, 2026-09-24)​

Some ad accounts are shared with the end client, who runs their own campaigns. ODMG names theirs with a prefix: Google campaigns start with do-; Meta campaigns start with DO- or DO -.

  • Per-guard campaign_scope: all (default, original behaviour) or managed. Column added by the append-only migration 20260924120000_spend_guard_campaign_scope.sql. The 20260923120000 migration is applied and is never edited.
  • One pure matcher, isManagedCampaign(platform, name) in src/lib/spend-guards/scope.ts, is the single source of truth. It is exact case, per the client's naming rule. Google: /^\s*do-/ (lowercase only). Meta and Instagram bindings: /^\s*DO\s?-/ (uppercase only). Leading whitespace, including NBSP, is tolerated. Client names such as Donut-, DOG- or dox never match.
  • Near-misses fail open but are surfaced. looksLikeManagedButUnmatched flags names that look like the prefix but fail the exact rule: another case of do/DO, an en or em dash, or DO -. Such campaigns are never counted and never paused. Their names are listed, escaped, on a ⚠️ Not counted — check naming: … line in the velocity, dry-run and cap Slack posts, so the naming can be fixed.
  • With managed:
    • Spend is read at campaign level and only matching campaigns are summed. This covers today's pulse and the yesterday baseline, so the last-hour window is in scope too. Meta uses insights level=campaign with campaign_name; Google uses GAQL FROM campaign with campaign.name. Money stays in dollars, and micros are converted only at the fetch edge.
    • Only matching ACTIVE/ENABLED campaigns are listed for a dry run or a pause. The platform client and the engine both filter, and the pause call drops any out-of-scope id it is given.
    • Every Slack message adds • Scope: ODMG campaigns only (DO-/do- prefix). With all, messages are byte-for-byte unchanged.
  • Changing the scope clears the cached yesterday baseline, last_spend_today, last_hour_spend and that account's snapshots. The next check recomputes all of them under the new scope, and neither the UI nor validation uses an old-scope figure. For about an hour after the change, the velocity rule has no last-hour baseline.
  • The API and validate.ts accept campaign_scope. It is still super_admin-only. If a PUT omits the field, the stored scope is kept, so an older client cannot widen a guard to all by accident. Widening a Live guard from managed to all requires confirm_scope_widen: true. Without it the API returns a 400 with code: 'confirm_scope_widen', even when the cap is unchanged. Both UIs show a confirm dialog before sending the flag.
  • Paging never truncates silently. If Meta still returns paging.next after 20 pages (spend or campaign list), the read throws IncompleteDataError. The guard records it in last_error. A pulse failure posts a loud "SPEND GUARD BLIND" message. A listing failure in Live becomes a loud AUTO-PAUSE FAILED, with nothing partially paused. A truncated yesterday baseline gives no baseline, and the error is kept in last_error. Google searchStream returns the full result set and has no pages to truncate.
  • UI: the Spend Guards drawer and the Clients "Daily spend cap" row both get a Campaigns control ("All campaigns" / "Only ours — do-/DO- prefix"). The Spend Guards table has a Campaigns column.