Sync Contentful entries to Algolia every 15 minutes

Every 15 minutes, sync recently published Contentful entries into an Algolia search index so our public site search stays in step with the CMS. This is a deterministic, code-style workflow triggered by a cron schedule running every 15 minutes.

Configuration (all editable in one place at the top of the workflow):

1) Contentful space id, environment id (default "master"), and Delivery API access token. 2) Algolia application id, admin API key, and target index name. 3) A list of content types to index — default to article, product, and landingPage — and for each content type, three arrays: searchableFields (mapped into Algolia's searchable attributes), filterableFields (mapped into attributesForFaceting), and displayFields (kept on the record but not searchable). 4) A path in the workflow's key/value store where the last successful run's high-water mark is persisted.

On each run:

Step 1 — Read the last successful high-water mark (sys.updatedAt) from the workflow store. If none exists yet (first run), default to 24 hours ago so we backfill a reasonable window without hammering the API.

Step 2 — For each configured content type, call Contentful "CDA - Get Published Entries" with query params content_type=<type>, sys.updatedAt[gt]=<lastRunIso>, order=sys.updatedAt, and limit=200. Page through results using skip/limit until skip + items.length >= total. Collect every returned entry.

Step 3 — Build the Algolia batch. For each Contentful entry, produce a flat record: objectID = the Contentful sys.id; contentType = sys.contentType.sys.id; updatedAt = sys.updatedAt; locale = sys.locale (or the space's default locale). Then copy fields listed in the content type's searchableFields, filterableFields, and displayFields into the record, resolving localized values against the default locale. Reference fields should be flattened to their linked entry's id and (where available) title/slug — do not recurse. Wrap each record in an Algolia batch action of type "addObject" (add or replace by objectID).

Step 4 — Handle deletions and unpublishes. The Content Delivery API only returns published entries, so anything previously in the index but no longer present needs to be removed. Two approaches are acceptable — pick whichever fits the space best: (a) maintain a persisted set of known objectIDs per content type in the workflow store, diff against the current published set for the content types we just fetched (using a full paged fetch without the updatedAt filter on the same 15-minute cadence would be expensive, so instead diff only the content types where we saw new activity); or (b) subscribe to sync deletions via the Contentful sync API when available. For MVP, use approach (a): for each content type touched this run, list currently-published objectIDs, compare with the stored set, and emit an Algolia "deleteObject" action for every objectID missing from the current set. Persist the updated known-id set at the end of the run.

Step 5 — Push the combined add/replace + delete actions to Algolia in a single call to "Batch Write (single index)" against the target index. If the batch is larger than 1,000 operations (Algolia's per-request cap in practice), chunk it into requests of 1,000 and issue them sequentially, keeping every taskID.

Step 6 — Record the returned taskID(s) in the workflow logs and add a log line noting that Algolia writes are asynchronous and reads will settle within seconds. Do not poll the task endpoint — the 15-minute cadence provides plenty of settling time before the next run.

Step 7 — On success, persist the new high-water mark: the maximum sys.updatedAt across all entries fetched this run. If nothing was fetched, leave the previous high-water mark in place so the next run re-attempts the same window.

Error handling: on a Contentful 429, back off using the X-Contentful-RateLimit-Reset header before retrying. On an Algolia 5xx or network failure, retry the batch against the numbered fallback hosts (<app>-1.algolianet.com, -2, -3). On any 4xx from Algolia other than 429, fail the run and leave the high-water mark unchanged so the next run picks the same window back up. Log the count of adds, replaces, and deletes emitted per run.

One-time setup at first run: configure the Algolia index's searchableAttributes and attributesForFaceting from the union of the configured searchableFields and filterableFields across all content types. This lets the same index serve multiple content types with sensible defaults; leave it alone on subsequent runs unless the configuration changes.