Handle paginated APIs without missing the final page

Stop pagination according to the API's documented continuation rule, not an assumption that a short or empty page must be the last one. Some APIs can return an empty page with a valid continuation token, while others use links, offsets or explicit completion flags.

The extraction is complete only when the termination contract is satisfied and the collected records pass the relevant reconciliation checks.

Inspect a controlled cursor contract

The original simulated API uses next_cursor=None as its only end marker. Pages also declare a stable snapshot ID and a total unique-item count.

Its default fixture has two populated pages and an empty third page. E2 appears identically on both populated pages. The result should contain three unique events totaling 3,500 paise after the final page confirms completion.

These semantics belong to the teaching API. Do not copy them into a real client without matching that provider's documented rules.

Follow the cursor through the empty final page

from api_extract import collect_pages,fixture_pages

pages = fixture_pages()
cursors = []
def fetch(cursor):
    cursors.append(cursor)
    return pages[cursor]

result = collect_pages(fetch)
assert cursors==[None,'page2','page3']
assert result['pages']==3 and result['unique_items']==3
assert result['raw_items']==4 and result['identical_replays']==1
assert sum(row['amount_paise'] for row in result['items'])==3500

# A separate valid fixture has an empty first page with a continuation cursor.
other = fixture_pages()
other[None]['items']=[]
other['page2']['items']=result['items']
continued = collect_pages(other.__getitem__)
assert continued['pages']==3 and continued['unique_items']==3
print({'default_request_cursors':cursors,'empty_intermediate_page_handled':True})

The second fixture demonstrates why if not items: break would be wrong under this contract. An empty page can still point to more data.

Detect loops and unexpectedly long extracts

Record cursors already requested. If the server returns the same continuation cursor again, fail instead of looping forever. Also impose a maximum page count appropriate to the expected extract and operational budget.

A page limit is a failure boundary, not a successful truncation rule. If the limit is reached while a continuation remains, do not label the partial records as complete. Investigate the expected volume or pagination behavior and rerun under a reviewed limit.

The lab rejects both cursor cycles and page-limit exhaustion. Its tests demonstrate those failure paths without making network calls.

Preserve snapshot consistency across pages

If records change while an offset-based extraction runs, items can move between pages and be missed or repeated. A stable snapshot or provider-supported cursor contract can reduce that risk, but its guarantees vary by API.

This teaching collector requires the same snapshot ID and declared total on every page. A changed snapshot fails the extract. Repeated IDs collapse only when payloads match exactly; conflicting payloads fail rather than being resolved by arrival order.

For a real API without snapshot support, document the remaining consistency risk and consider provider-supported updated-at windows, overlap with deduplication, change feeds or another suitable extraction strategy. No generic pagination loop can guarantee completeness when the source contract does not support it.

Reconcile the completed result

The fixture's expected unique total is three. Raw item count is four because one item repeats. Compare the declared total with the correct identity-level count, not with the sum of page lengths.

A server-declared count is still source evidence, not independent truth. Where practical, compare additional control totals or source partitions. Preserve page count, raw count, unique count and replay count in the extraction record so discrepancies are diagnosable.

Exercise: change every page's declared total to four while leaving the records unchanged. Verify that extraction fails at completion. Then make page two point to itself and verify that the cycle is detected before the maximum-page limit is exhausted.

NeuraPath's Data Analytics with Generative AI course connects API handling with data completeness. A reliable paginated extract follows the source contract and verifies the resulting population before analysis begins.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Validate an API response before loading it into a dataframe.
• Continue with Respect API rate limits in a reporting pipeline.