Skip to main content

Sync Failures

Symptom​

One of:

  • A scheduled paid-acquisition sync fails (Inngest run marked failed).
  • A manual re-run of the same sync succeeds.
  • Ad-platform data simply stops arriving, with no failed run anywhere.

Diagnosis​

Work down this list. The last two causes produce no error at all, which is why they take longest to find.

1. Is anything registered?​

A function missing from the functions array in src/inngest/functions.ts never runs and never errors. Check that first if the function is new or was recently refactored.

Then check the app actually synced after the last deploy. modified: true on a manual PUT is not drift on SDK v3.54+ — check latestSync.status ("duplicate" is healthy) and the cron_liveness block in /api/admin/sync-health.

If registration is failing for everything at once: the plan concurrency cap is 5. One function declaring more gets a 400 at sync time and blocks registration of every function in the app.

2. Is it actually a failure, or the conveyor cadence?​

Scheduled paid fetching is a conveyor. paidAccountConveyor dispatches exactly one account per */10 tick, fleet-wide, and syncMarketingAccount runs strictly serially (global concurrency limit 1). On a large fleet an individual account waits its turn — that is the design, not a failure.

Two related non-failures:

  • Env-configured bindings with no row id never ride the conveyor. They only sync via manual full-client syncs.
  • Meta device/country segment cuts refresh only on hour-divisible-by-6 cycles (heavyCutsDue). Missing segment cuts inside a 6-hour window are expected.

3. Is the binding credentialed and active?​

A binding with no usable credentials is a logged no-op — by design it never fabricates data and never falls back to another tenant's account. So "no data" can be the correct, safe outcome.

Check client_ad_accounts.status (a paused/archived binding is skipped), then Connectors on /marketing/settings. If every connector looks uncredentialed at once, suspect CREDENTIAL_ENCRYPTION_KEY being unset — both vaults fail closed without it.

Config-missing errors are raised as NonRetriableError deliberately, so they fail fast rather than burning retries.

4. Is the platform value normalized?​

platform=instagram must canonicalize to facebook (canonicalPaidDataPlatform). Without that it matches no stage and the whole sync no-ops silently.

5. Did a step blow its budget?​

If a run fails mid-way with no vendor error, check step size and duration:

  • Durable-step output size. A step returning row arrays or raw vendor payloads can exceed the limit. Step outputs are persisted forever and re-read on every retry.
  • maxDuration = 800 on the serve route buys headroom, not boundedness. A step that outgrows it must be split. (At the old 300s default this endpoint threw 626 runtime timeouts in one week.)

6. Did the schedule stop firing entirely?​

If a cron function simply stopped with no failed runs, changing the cron expression forces Inngest Cloud to recreate the stored schedule. That is the documented fix — it resolved sync-hyros-orders on 2026-08-17.

Fix​

For a step-output failure, keep the rows inside the step and return metadata:

// ❌ Wrong — returns an unbounded array that is persisted forever
const rows = await step.run('fetch-day', async () => {
  return await fetchDay({ creds, date })
})

// ✅ Right — writes inside, returns a compact summary
const result = await step.run('fetch-day', async () => {
  const rows = await fetchDay({ creds, date })
  await writeToClickHouse(rows)
  return { count: rows.length, date }
})

For everything else, fix the specific cause found above, then trigger a manual sync (/admin/sync-health per-account buttons, or npm run sync:marketing). A manual sync runs everything including paid, so it is the fastest way to prove credentials independently of the conveyor.

Re-running is safe: writes are idempotent. ReplacingMergeTree dedupes by the table's ORDER BY grain, and synced_at derives from event.ts rather than wall-clock time, so a replay does not split one run's rows across versions.

Prevent​

  • Every fetcher's step returns a { count, date, … } shape by convention.
  • Never use new Date() for synced_at in a durable function body — a wall-clock stamp mints a new version per retry attempt.
  • Do not raise the deliberate low concurrency caps. The global limit: 1 on syncMarketingAccount is a measured operator directive from live Meta throttle testing.
  • After any deploy that changes an Inngest function, verify against /admin/sync-health — a green Vercel deploy is not evidence the job runs.

See also​

  • The Inngest folder guide (src/inngest/AGENTS.md) — authoritative current sync model.
  • Inngest durable steps — the permanent reference.
  • Deployment — sync-after-deploy and verification.