Tools and workflows

Python Requests GET: query parameters and timeouts

Send a GET request with params and timeout. Check the HTTP status and returned data before treating an empty result as a successful read.

Use requests.get(url, params=filters, timeout=(3.05, 10)) to read a resource without building the query string by hand. The timeout tuple supplies separate connect and read limits. Call raise_for_status() to surface HTTP error statuses, then check the data against the endpoint's contract before using it.

Pass query parameters with params=

This function takes the endpoint URL as an argument. Check that your service supports the department query field or substitute the documented field. If import requests fails first, install Requests in the interpreter you want to use.

import requests

def fetch_response(url, department):
    response = requests.get(
        url,
        params={"department": department},
        timeout=(3.05, 10),
    )
    response.raise_for_status()
    return response

For an endpoint that documents a department parameter, the client handles its encoding. Building a query string by joining values yourself can change the request meaning when a value contains a space or ampersand. The server decides which parameter names and values it understands.

Compare the same query across two runs

If a call returns no records, verify the query separately from the parser. Record the endpoint path and the non-sensitive filter values, then use a known test value the API documentation says should return something. A typo in the filter name can cause a server to ignore it. The reply can be entirely valid JSON and still not answer the intended question.

A filter that needs repeated parameters can be represented as a list of pairs if the service documents that shape. For example, an API might accept [("tag", "reading"), ("tag", "reference")]. The repeated name is kept distinct from an invented value such as "reading,reference". Encoding a syntactically valid URL does not establish that two representations have the same application meaning.

Distinguish connection time from read time

The timeout pair passes separate connect and read timeouts. A read timeout limits waiting for data on a socket. It does not necessarily bound the total elapsed time of the whole download. If a service gradually sends bytes over a longer interval, a job may still need a separate overall deadline and a documented size limit.

A connection error means there may be no HTTP response to parse. An HTTP error status means a server or intermediary did return a response. In both cases, keep operational failures distinct from a valid response containing zero records. A report that says “No items found” when a service could not be reached is misleading.

Read the contract before reading fields

A JSON endpoint might return a list, a dictionary, or an object containing a list of records. A service that expects sign-in may return an HTML login page. Check the status and content type before decoding, then verify the fields you use. The Requests response guide explains what status, headers, and body each tell you.

Capture a small sanitized sample that represents both an ordinary result and a valid empty response. Test your parsing logic with those samples without contacting the service. Separately verify one real request against the endpoint to check the documented contract.

Distinguish three outcomes on the same input

Imagine a collection used to prepare a reading list. A 200 response with an empty list can mean nothing matched the documented query. A 403 response may mean access is denied. A connect timeout means you could not establish a connection in the time allowed. None should lead to a report saying “There are no readings” without qualification. Report the query criteria on a genuine empty result and record the failure category for the other cases.

A body can also contain a list of errors while appearing syntactically valid. Read a real documented response and confirm the parent fields and item shape before writing the import. Where possible, compare a fixture with the public provider's examples while keeping any identifying values out of the checked-in data.

Verify only the fields you use

Start by checking whether the top-level value is what your caller expects. If your job uses a count of records, distinguish a valid empty list from a missing list. If a service's reply includes optional metadata, decide how it affects your result. Keep validation focused on the documented fields the application uses.

Keep a retrieval polite and accountable

Use the endpoint only for its stated purpose, within its access rules and limits. Identify which requests the client makes, keep them bounded, and do not hammer a failing service. Cache or reuse data where the publisher's guidance permits it.

In a log, record the action, status, and retrieval time if the project needs provenance. Avoid retaining full URLs with secret query values or entire responses with personal information. The goal is to explain what you retrieved and how a failure affected the result without copying every private detail into a report.

Sources and further reading

Related reading

Browse Tools and workflows · Explore project tutorials

Search PythonProjects.org