Skip to main content

Ingestion

How data gets into Launchpad V2.

Shape​

Every ingestion is a durable Inngest function under src/inngest/. The authoritative description of the current model — including the exact function names, cron expressions, and concurrency configs — is the Inngest folder guide (src/inngest/AGENTS.md). This page is the conceptual summary; that file wins on any conflict.

Rules, learned the hard way:

  1. Return compact step outputs. Step output has a size limit and is persisted forever, re-read on every retry. Return counts and ids; keep row arrays inside the step body.
  2. Concurrency is per-function and deliberate. There is no blanket "key by client_id" rule — several functions key on binding_id, and syncMarketingAccount is globally serial (limit: 1) by operator directive after live Meta throttle measurement. Read each function's config and its comment before changing it.
  3. Every write is idempotent. ReplacingMergeTree dedupes by the table's ORDER BY grain — not by event_id, which is an audit stamp only.
  4. synced_at derives from event.ts, never new Date(). A wall-clock stamp mints a new version per retry attempt and splits one run's rows across versions.
  5. A function missing from the functions array in functions.ts never runs, and nothing errors.

The scheduled model is a conveyor​

Scheduled paid (ad-platform) fetching is not a nightly fleet sweep:

  • paidAccountConveyor (cron */10 * * * *) dispatches exactly one least-recently-synced active paid binding per tick, fleet-wide. Freshness comes from max(synced_at) in the warehouse — there is no cursor state.
  • syncMarketingMetrics (cron env.MARKETING_SYNC_CRON) fans out per-client events marked include_paid:false; its unscoped runs handle commerce, attribution, and post-stages only.
  • Manual syncs run everything, including paid. That is why a manual sync is the fastest way to prove a new binding's credentials.

Each tick dispatches exactly one account, fleet-wide — this is why freshness is measured from max(synced_at), not a per-tick guarantee for any single client. syncMarketingAccount runs at global concurrency 1, a measured operator directive from live Meta throttle testing. The write to ClickHouse is idempotent: retrying the same tick produces the same event_id and the same ORDER BY grain, so a retry after a partial failure never double-counts.

See Inngest durable steps for the full picture.

Per-client credentials​

Credentials live encrypted in the connector vault, referenced by client_ad_accounts.credential_ref — a pointer, not an inline value. There is no client_ad_accounts.credentials column. Encryption is the AES-256-GCM envelope in src/lib/crypto.ts, keyed by CREDENTIAL_ENCRYPTION_KEY; the vault fails closed when that key is unset.

Bindings are re-resolved inside each run and never carried on the event — credential refs must not become durable payload, and a paused binding must drop out immediately.

A binding with no usable credentials is a logged no-op. It never fabricates data and never leaks the default account under another tenant.

Backfill​

Backfills run through the same durable path as scheduled fetches. Never bypass Inngest.

CommandPurpose
npm run backfill:adsAd history window-walk (src/inngest/functions/backfill-ad-history.ts) — one walk per platform fleet-wide
npm run backfill:utmMarketing UTM backfill
npm run backfill:clickup-projectsClient → ClickUp project association
npm run reconcile:clickup-identityClickUp identity normalization
npm run repair:clickup-sidecarsRepair sidecar history after a partial snapshot wrote blanks
npm run sync:marketingTrigger a marketing sync

Two backfills also run as nightly crons in their own right: backfill-creative-metadata (04:00 UTC) and backfill-launchpad-metadata (04:30 UTC).

Failure modes​

See Sync failures for the runbook.

Adding a new platform​

  1. Write the connector under src/lib/marketing/platforms/ — read that folder's guide (AGENTS.md) first.
  2. Wire it into src/lib/marketing/platform-fetch.ts.
  3. Ensure the platform value canonicalizes correctly (canonicalPaidDataPlatform). instagram normalizes to facebook; an unnormalized value matches no stage and the whole sync silently no-ops.
  4. If a new background job is genuinely needed: define it in its own file under src/inngest/functions/, type its event in src/inngest/client.ts, add it to the functions array in functions.ts, and pick a cron minute that does not collide. Keep concurrency at 5 or below — a higher value 400s at sync time and blocks registration of every function in the app.
  5. Add the platform to the connectors reference.
  6. Add tests: npm run test:isolation must stay green, and add coverage for the new connector's normalization.