Tools and workflows

Python Requests Session: cookies and connection reuse

Use requests.Session() for related calls to one service. Know which cookies and headers persist, when connections can be reused, and how to close the session.

A requests.Session() can retain cookies and shared headers across a group of requests, and it can reuse connections when the service and response handling allow it. Use a session for related calls under one application job or account. Wrap it in a with block so the client closes after the job.

Which requests should share a session?

A one-off GET can use requests.get() directly. For a series of authorized reads from the same service, a session can hold shared headers and reduce repeated connection setup. If the server sets cookies, those cookies can carry into later matching requests. That is useful only within the identity and service scope you intended.

Keep separate clients for unrelated services or user accounts. A session reused indiscriminately across operations is harder to inspect, and default headers may contain values that belong to only one target. Review where any secret header is applied.

Keep request-specific values close to the call

Set a shared header such as Accept when it applies to every call. Put query parameters and request bodies on the calls that need them. Keep the GET request timeout explicit, or use a project helper that supplies the same bound consistently.

import requests

def fetch_json(urls):
    with requests.Session() as session:
        session.headers["Accept"] = "application/json"
        for url in urls:
            response = session.get(url, timeout=(3.05, 10))
            response.raise_for_status()
            yield response.json()

Call this iterator only with a bounded list of intended URLs. Full consumption closes the session. If the caller stops early, explicitly close the generator or choose an ordinary eager loop within a with block so ownership is obvious.

Understand how connection reuse can fail

Connection pooling can reuse connections when the response body has been consumed, subject to server behavior and client settings. If you explicitly stream a body, it needs its own resource handling. Close a streamed response that you will not fully consume. Otherwise an open response can keep its connection occupied.

A strange delay can come from connection handling, the upstream service, or the volume of data. Measure a small job, observe results, and avoid assuming that enabling a session accelerates every request. If a job hits limits, stay within the publisher's usage rules.

Identify which state is changing across calls

Some responses set cookies, so the same endpoint can behave differently on a fresh session than it did after an earlier request. If a request seems to succeed only in the middle of a long workflow, try it from a new client configured exactly as the service expects. Confirm which initial call is required for context. A reader of the code should be able to tell what earlier state the result relies on.

Use separate sessions when the identity changes. Think about a reporting process that services two accounts on the same provider. A credential or cookie stored on a shared client can make the second account's operation depend on the first. Scope clients and sensitive settings at the account boundary, and discard them when you finish that work.

Plan one cleanup check

A successful run is easiest to inspect when it reads each response completely and closes its session. Add a fixture case where the second request returns an error, then check that the client leaves the intended failure state without keeping a reference to the session. If the workflow streams results, test the code path that stops reading early. That case is often more revealing than the ordinary full read.

Keep the service contract visible

Sessions do not validate response bodies or handle your application errors. Decide which results count as success and what a job returns when one request fails after others have succeeded. If you need all-or-nothing output, assemble results in a staging location before replacing a verified previous result.

Test the independently meaningful parts. Check response parsing with local samples. Verify session behavior with a controlled service you own. Record when the job closes connections and what it exposes after a partial failure, especially before moving from a one-off script to a scheduled run.

Sources and further reading

Related reading

Browse Tools and workflows · Explore project tutorials

Search PythonProjects.org