Skip to main content

Incident Response

Symptom​

You are on call and something is visibly wrong: the portal is down, ingestion has gone silent, a report is showing corrupted numbers, or a deploy just broke something in production.

Diagnosis​

1. Decide the severity​

SevDefinitionResponse
SEV-1Customer-visible outage. Portal down, ingestion silent >1h, data loss risk.Page immediately.
SEV-2Degraded but usable. Single-platform ingestion failing, slow queries.Same-business-hour response.
SEV-3Non-urgent. Cosmetic bug, isolated report anomaly.Next-business-day.

2. Correlate with recent deploys​

If a deploy happened in the last 2 hours, that is your primary hypothesis. Roll back before further investigation.

Fix​

First 10 minutes (SEV-1)​

  1. Post in #nls-odmg-sbd-tech: "SEV-1 declared. Investigating. [link to failing surface]"
  2. Snapshot state. Screenshot the Vercel deployments page, Inngest dashboard, ClickHouse system tables. Attach to the thread.
  3. Correlate with deploys. If a deploy happened in the last 2 hours, that's your primary hypothesis. Roll back before further investigation.
  4. Do not fix in prod. If a rollback isn't possible, put the affected surface behind a maintenance flag rather than hotfix-deploying.

Rollback​

  • Application code: promote the previous production deployment from the Vercel Deployments tab.
  • Migrations do not roll back with the deployment. Both ledgers run at build time (scripts/migrate-on-deploy.ts before next build), so promoting an older deployment does not un-apply a migration the newer build ran.
  • Supabase: supabase db reset --linked is destructive — never use in prod. Roll forward with a corrective migration.
  • ClickHouse: schema lives in clickhouse/migrations/, tracked by the schema_migrations ledger. There are no down migrations — do not go looking for one. Never replay migrations to "reset" state: a replay once destroyed the production fact table (see Disaster recovery). Roll forward with a new migration file; the ledger keys on filename, so editing an applied file does nothing.
  • Attribution rule change: git revert the rule commit. mv_ad_daily_rollup is a plain VIEW computed at read time, so there is nothing to re-materialize — run npm run mv:rebuild only to verify parity.
Verify the rollback landed

A green Vercel deploy is not evidence. Confirm /api/health returns 200, probe the affected behavior live on launch.lynkr.cc, and — if background jobs were involved — check cron_liveness in /admin/sync-health.

Prevent​

  • Any SEV-1 or repeated SEV-2 gets a postmortem within 72h. Blameless, filed as docs/postmortems/YYYY-MM-DD-<slug>.md, and must include:
    • Timeline (UTC)
    • Impact (users, dollars, data)
    • Root cause (5-whys, not "the developer")
    • Corrective actions with owners and dates
Unverified

There is no docs/postmortems/ directory in the repo at HEAD and no template file — create both on the first postmortem.

See also​