Python API script missing records? How to make it fetch every page safely

Most small business API scripts do not fail loudly. They fail quietly. A script that pulls orders, contacts or bookings from an API can run without an error and still miss most of the data. I build, fix and improve technical systems for businesses, and this guide walks through a repair I made to a typical script, tested against a fake API with made-up orders.

1. The script that looked fine

The original is the kind of script that works on the day it is written. It builds a URL, calls it and reads the items from the response:

def fetch_orders(status):
    url = BASE + "/orders?status=" + status
    with urllib.request.urlopen(url) as r:
        return json.load(r)["items"]

I tested it against a small local API that behaves the way real ones do. Results are split across pages, the server sometimes answers with a rate limit or a temporary 503, and some responses are slow. There were 13 matching orders. The script returned 5. There was no error and no warning, because it only ever read the first page.

2. Five problems hiding in four lines

A review found five faults. There is no timeout, so a server that stops responding can hang the script forever. Only page 1 is read, so most records are missed. A 429 rate limit or a 503 crashes the whole run. Nothing is retried, so one network blip loses the export. The query string is built by joining text, so a value containing an ampersand or a space breaks the request.

None of these show up when you test with a handful of records on a quiet afternoon. They show up when the script runs unattended overnight and someone relies on the totals the next morning.

3. Follow every page, with a limit

The repaired client asks for page 1, yields its items, then follows the next page number until the API says there is no more. It also stops after a set number of pages, so a faulty API that always claims there is another page cannot loop forever.

for _ in range(max_pages):
    data = self.get_json(path, {**params, "page": page})
    yield from data.get("items", [])
    page = data.get("next_page")
    if not page:
        return

Your API may use a cursor or a link header instead of page numbers. The idea is the same: keep going until the API says you have everything, and never assume the first response is complete.

4. Retry the right errors, and only those

Every request now has a timeout. When the server returns 429, 500, 502, 503 or 504, or the connection drops, the client waits and tries again. If the server sends a Retry-After header, the client honours it, capped at 30 seconds. Otherwise it backs off for half a second, then 1, 2 and 4 seconds.

Other client errors such as 401 or 404 are not retried. A bad token or a wrong address will be just as wrong on the fifth attempt, so the client stops at once with a clear message. A response that is not valid JSON, such as an HTML error page, is also reported plainly instead of causing a confusing crash further down.

The query string is built with urllib.parse.urlencode, so awkward values are encoded properly. Everything uses the Python standard library, so there is nothing new to install on the machine that runs it.

5. Prove it with tests, not promises

There are 14 automated tests. They start a fake API on the local machine and script its behaviour on purpose: rate limits, server errors, slow responses, endless pages and HTML instead of JSON. One test runs the original script against the same API to show the bug, 5 of 13 orders, while the repaired client returns all 13. Waits are recorded rather than slept, so the whole suite finishes in about a second.

All of the data in the tests is synthetic. No client data was used at any point.

6. What it deliberately leaves out

The client only reads data. Writes such as creating an order are left out on purpose, because retrying a write can create duplicates unless the API supports idempotency keys. That needs a decision for each API, not a blanket retry.

Checking your own script

If you have a script that pulls data from an API, try three quick checks. Count the records it returns and compare them with the total shown in the app. Look for a timeout on every request. Find out what happens when the API answers with a 429. If any of those answers is unclear, the script may be missing data without telling you.

That is the kind of thing I check and fix. See my Python repair service for how a scoped repair works, or book a free systems check to talk it through.