Read the API's first page, then follow the documented next link, cursor, or page number. Stop when the API says there is no next page. Also set a page limit and detect repeated cursors. If either check stops the run early, label the collected records as partial.
How to tell whether an API has another page
A service that returns ten records may have ten records total, or it may default to ten per page. Look for its documented continuation field. A page can contain fewer than ten records or even none and still give you a next cursor. Check that field before deciding you are finished.
If the API gives a next link, preserve its documented query information. A server may place time filters or ordering in that link. Keep follow-up requests within the intended endpoint and credentials scope. A URL supplied in a response is an input that still needs validation against the service's expected origin, especially if the client adds a secret header.
Decide what complete means
A report can aim to collect a stable snapshot, or it can be an observation over a changing period. These are different outcomes. Consider a page numbered API that remains active during a read. If a new entry is inserted ahead of the first page, existing entries can shift and appear again later. If an entry is removed, another can shift to a page the collector already visited.
Use a consistent ordering or snapshot feature if the API documents one. If it provides exact record IDs, keep duplicates identifiable. Record the start and end of the run and how the collection ended. Deduplication addresses repeated copies, but does not reconstruct a record that moved out of the collector's view.
Bound and reconcile the request loop
Keep count of successful pages, seen continuation values, and records. Use a reasonable maximum page count and elapsed-time budget. If a next cursor repeats, stop and mark the collection incomplete. Keep even a single-page request bounded with a timeout. If the API asks you to wait, honor its documented backoff and rate rules.
Write incomplete output separately or retain the previous complete result. For instance, a collector that retrieves two pages and hits a deadline on page three can report “Two pages collected, incomplete.” It should not publish the partial count as the total.
Test a cursor contract that fits on a page
Here is an offline demonstration. The two fixture responses use an opaque string as the continuation value. An actual API supplies its own extraction rules. This function is concerned with whether the read ended as promised.
def collect(fetch, max_pages):
cursor = None
seen = set()
collected = []
for _ in range(max_pages):
page = fetch(cursor)
collected.extend(page["items"])
cursor = page["next"]
if cursor is None:
return collected, True
if cursor in seen:
return collected, False
seen.add(cursor)
return collected, False
fixture = {
None: {"items": ["oak"], "next": "opaque-1"},
"opaque-1": {"items": ["birch"], "next": None},
}
items, complete = collect(fixture.__getitem__, max_pages=5)
print(items, complete)
Expect two items and True. Change the second next value to "opaque-1" and expect a result marked incomplete. Set the page limit to one and you should collect only the first page, also incomplete. The example tests the continuation rule offline. For a real endpoint, configure a bounded GET request and validate the response fields before handing them to the collector.
Test the collection with a tiny set of fixtures
Make samples for a first page with more data, an ordinary final page, an entirely empty collection, and a repeated next token. Add an error on the second request. Write expected page counts and completion status by hand before testing the code. These checks reveal whether the code stops for the right reason without repeatedly contacting the service.
Review the most important question with a sample output in hand. Could someone distinguish a collection that truly had no records from a collection that never finished? If they can, the result is useful even when the next request fails.