An integration can return a successful HTTP response and still leave the business with incomplete data. The common reason is pagination: the source system returned the first page of records, while the integration treated that page as the whole result.
This matters whenever a workflow copies contacts, orders, tickets, invoices, inventory items, or other records between systems. A small test dataset may fit on one page. Real data eventually may not. Pagination therefore belongs in the integration design, not as a last-minute repair.
What pagination changes
Many APIs limit how many records appear in one response. The response may include a continuation URL, a cursor, a page number, or a header containing a link to the next page. Some APIs use offset and limit parameters; others deliberately avoid exposing a numeric page and return an opaque token.
These mechanisms are not interchangeable. A client should follow the source API’s documented continuation method instead of assuming that adding page=2 will work. GitHub’s REST documentation, for example, describes using the HTTP Link header to find additional pages, while other APIs place a nextLink or cursor in the response body.
Define what a complete sync means
Before writing a loop, define the business result. Does a run need to copy every record available at the start of the run? Only records changed since the last successful checkpoint? Or all records in a particular date range?
Also decide what happens when a record changes while the sync is running. A simple full export may be acceptable for a small, mostly stable dataset. A busy system may need a timestamp filter, a source-provided snapshot, or a follow-up reconciliation pass. Without this decision, the integration can appear complete while its boundary is ambiguous.
Follow the continuation signal explicitly
A reliable pagination loop has a small number of visible steps:
- Request the first page with the required filters and a bounded page size.
- Process the records in that response.
- Persist enough progress to resume safely.
- Read the documented continuation signal.
- Request the next page until the source indicates that no page remains.
Do not manufacture the next request by copying only part of the original query. A continuation URL or cursor may carry filters, sort order, or server state that the client must preserve. If the API returns a next-page URL, treat it as the source of truth and log a sanitized representation of the page transition.
Make each page safe to retry
Networks fail, rate limits appear, and a worker can stop after receiving a page but before finishing its work. The page-processing operation should therefore be repeatable. Store a source record identifier and a synchronization run identifier at the destination, and use an upsert or an equivalent duplicate check when appropriate.
Checkpoint timing deserves care. If the integration advances its cursor before the records are committed, a crash can skip them. If it commits records but never records progress, a retry may process the page again. The destination operation and checkpoint should have a clear consistency strategy, such as a transaction where the systems support it, or an explicitly designed replay-safe sequence.
Protect ordering and filtering assumptions
Offset pagination can become fragile when records are inserted or removed between requests. A later page may shift, causing a record to appear twice or not at all. Cursor-based pagination can reduce that class of problem, but it still depends on the source’s documented ordering and cursor behavior.
Use a stable sort when the API supports one, avoid changing filters between pages, and record the filter and sort configuration with the run. If the source offers incremental synchronization or change tokens, evaluate that option instead of repeatedly walking a large mutable collection.
Handle partial failure as a normal state
A sync that processes six pages successfully and fails on page seven is not the same as a sync that processed all seven pages. Give the run a visible state such as in progress, partially completed, completed, or failed. Include the last confirmed continuation point and the count of records accepted, rejected, and retried.
Retries should be bounded and should respect the source’s rate-limit guidance. A retry queue or operator alert is more useful than an endless loop. If one malformed record blocks a page, decide whether the page should be quarantined for review or whether the record can be rejected while the rest continues. That decision should be part of the workflow’s operating rules.
Verify completion instead of assuming it
At the end of a run, check more than the absence of an error. Confirm that the source returned no continuation signal, that the destination accepted the expected page transitions, and that the run’s record counts are plausible. Where practical, compare a source-side count or watermark with the destination result, then run a reconciliation query for records changed during the sync window.
Monitoring should make missing data visible. Useful signals include pages processed, records received, records written, duplicates avoided, validation failures, retry count, duration, and the age of the last completed run. These measures are operational clues, not guarantees; a successful count can still reflect a flawed filter, so periodic spot checks remain valuable.
A practical review checklist
- What pagination mechanism does the source document?
- What exact event makes a sync complete?
- Can a page be safely replayed after a timeout?
- Where is progress stored, and can an operator resume it?
- How are rate limits, malformed records, and partial failures surfaced?
- What reconciliation or spot check detects a quiet omission?
Pagination is a small implementation detail only when the data set is guaranteed to stay small and the guarantee is real. For most business integrations, it is part of the contract between systems. Designing the continuation path, checkpoint, retry behavior, and completion check together gives a small team a workflow it can explain and maintain.
Next step: Schedule a short consultation to identify the next useful improvement.