Data and reporting

How to List Files in a Directory With Python

Use pathlib to list immediate files, match a pattern recursively, filter by type, and sort results for a repeatable batch.

With pathlib, use Path(folder).iterdir() to list immediate children, .glob("*.csv") to match names in that folder, or .rglob("*.csv") to search subfolders. Keep only paths that pass .is_file(), and sort the results if your import needs to run in a repeatable order.

List only the inputs you plan to process

from pathlib import Path

folder = Path("incoming")
if not folder.is_dir():
    raise NotADirectoryError(folder)

files = sorted(path for path in folder.glob("*.csv") if path.is_file())
for path in files:
    print(path.name)

This assumes an existing directory called incoming under the current working directory. A matching name such as old.csv could belong to a directory, so the type check matters. By design, no matching files can be a legitimate result. For a daily job that expects at least one file, treat that case separately from a successful empty report and tell the operator which folder and pattern were used.

Use an explicit rule if uppercase suffixes should be treated like lowercase suffixes across platforms. For a nonrecursive selection, iterate over folder.iterdir() and compare path.suffix.casefold() == ".csv". Case sensitivity can vary with platform and file system. Decide whether differently capitalized names can coexist and whether two such exports represent separate inputs.

Decide when to scan subdirectories

Recursive search brings convenience and risk of scope creep. The folder may contain backups, hidden test data, or a report directory from the previous run. If the task expects a date hierarchy, define exactly where the program starts and which subdirectories belong to that input. Do not move the root up one level to make a pattern match more results unless that is the intended contract.

Check how you handle links. A path that passes is_file() can refer to the target of a symbolic link. Another directory might include a link to a file outside the approved input folder. If you process a shared, untrusted directory, enforce the directory and link policy at open time or use a trusted staging area. Ordinary directory checks alone do not make a file access sandbox.

Sort by the value the filename represents

Directory listing order is not a report order. If export files contain a machine-readable date in their names, parse and sort by that date when order represents time. Lexical sorting works for certain fixed-width date formats. It will place report-10.csv before report-2.csv if you sort digit strings. Sort parsed numbers if the names represent sequence numbers.

Two different subfolders can contain weekly.csv. Keep the relative path instead of reducing both to the basename. When archiving processed data, a stable input identifier can incorporate source, logical period and source version. Hashing entire files can tell you whether byte content changed, but identical bytes from two different producers are not always duplicate observations.

Account for changes while processing

A list of paths is a view of what was present during discovery. A producer may be adding files, removing them or updating them while the job runs. Decide how files are marked ready. A clean pattern is for the producer to write outside the known input name and rename the completed file into place, under a naming and file system convention agreed by both sides.

A file can disappear after you list its path. Decide whether one missing file should stop the report or produce a clearly marked partial result. Record the skipped path and why opening failed. For a recurring job, keep the report in a separate output folder so tomorrow’s scan cannot count today’s output as a new input. See how to write and replace report files safely.

Sources and further reading

Related reading

Browse Data and reporting · Explore project tutorials

Search PythonProjects.org