Data and reporting

Python File Encoding and Unicode Errors

Diagnose UnicodeDecodeError and UnicodeEncodeError, identify the source encoding, and handle UTF-8 byte order marks without dropping data.

If reading raises UnicodeDecodeError, check which encoding the file’s producer used and pass it to open(). If printing raises UnicodeEncodeError, check the output stream instead. In both cases keep the original bytes. Removing undecodable characters can change a name or make two identifiers look alike.

Distinguish text from bytes

The text café is four Unicode characters. Its UTF-8 representation has five bytes, because the final character uses two. Python strings hold text and bytes objects hold raw bytes. A decoder takes bytes and produces text. An encoder takes text and produces bytes. Choosing a different decoder can alter how the same stored sequence appears.

text = "café"
data = text.encode("utf-8")
assert len(text) == 4 and len(data) == 5
assert data.decode("utf-8") == text
print(data.hex())  # 636166c3a9

Opening a file in text mode puts a decoder between your program and the file. Opening in binary mode gives you bytes. A file checksum belongs to the latter. If an import step normalizes line endings or rewrites accented characters, a hash of the new output documents the new bytes only.

Check whether the error comes from reading or printing

A UnicodeDecodeError while opening or iterating over a file points to a byte sequence the selected decoder cannot handle. A UnicodeEncodeError during printing can mean the receiving terminal or redirected output cannot encode a character that already exists in a valid Python string. Read where the error was raised before changing the input codec.

If names look corrupted both in the terminal and in a UTF-8 file, inspect a small sample and compare it against the producer’s export settings. If a UTF-8 output file looks right in a compatible editor but redirected console text fails, investigate the output stream. Rewriting valid source data to match a terminal display can lose information.

When debugging terminal output, inspect sys.stdout.encoding in the same context where printing fails. An interactive terminal and a redirected file can use different encodings, so a test in one may not explain the other. If the desired result is a UTF-8 report, write to an explicitly encoded file with Path.write_text(text, encoding="utf-8"). That makes the file format independent of the terminal. If another program reads stdout, agree on the stream encoding with that consumer.

Check the encoding used by the export

Ask which application or export settings produced the file. Some older Windows exports use Windows-1252. That codec differs from ISO-8859-1 in part of its byte range, so visually similar results on common ASCII text do not establish a match. A known name containing a typographic apostrophe or accented letter can help validate the interpretation. If a pipeline processes many producers, make encoding a per-source option with a clear default.

If a source may export a UTF-8 byte order mark, utf-8-sig consumes a leading BOM when decoding. That can fix a CSV header appearing as \ufeffname after opening with plain UTF-8. It does not detect different encodings. Use it when the producer’s format permits that signature. UTF-16 needs its own compatible decoder.

Handle damaged data deliberately

Avoid using errors="ignore" as a blanket fix. Dropping bytes can turn two different identifiers into the same string. errors="replace" can help preview damaged text by making substitutions visible, but any resulting report should acknowledge that information is missing. Keep strict decoding for authoritative imports, or reject affected records into an explicit review path that preserves the originals.

If you have mixed encodings in a single file, a single codec might not produce a valid and trustworthy reading. Ask for a clean export where possible. When repair is necessary, record what segment was repaired and how it was verified. Keep the repair reproducible so the next import can flag the same condition instead of silently applying an accidental workaround.

Separate encoding from text normalization

After the bytes decode, two strings can look the same on screen while containing different Unicode character sequences. For example, a letter with an accent may be represented as one character or as a base letter followed by a combining mark. That is a text normalization question. Switching file encodings does not make equivalent text have identical Unicode sequences.

If search or matching needs a common representation, keep the source spelling and create a comparison key using an explicit normalization rule. A field used as an exact identifier may need a different policy from a search term. Check for collisions before using normalized strings as unique keys. Treat the comparison form as part of how the application searches, and the original text as the record it displays or preserves.

Specify an encoding and test a round trip

For new output, choose an explicit encoding and state it alongside the file format and newline convention. Test an example that includes ASCII, accented characters and non-Latin characters you expect in real data. Confirm that a round trip through the receiving application preserves the intended string. A one-line fixture made entirely of ASCII will miss many problems because ASCII-compatible encodings agree on its bytes.

Store the original file as received if the exact source matters. Produce cleaned text separately with its own file name and conversion record. That gives you an answer to two different questions: what the producer supplied, and what the report actually used.

Sources and further reading

Related reading

Browse Data and reporting · Explore project tutorials

Search PythonProjects.org