Design an encoding-failure report that a shell script can use

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

codecs.lookup validates an encoding name, while decoding can separately raise UnicodeDecodeError for invalid byte sequences.

Example

The helper returns a stable kind and code for three paths: a supported UTF-8 payload, invalid UTF-8 bytes, and an unknown codec name. That gives a shell caller a useful branch without parsing traceback text.

import codecs

def check(data, encoding):
    try:
        codecs.lookup(encoding)
        data.decode(encoding)
    except LookupError:
        return ('unknown-encoding', 3)
    except UnicodeDecodeError:
        return ('decode-error', 2)
    return ('ok', 0)
assert check(b'ok', 'utf-8') == ('ok', 0)
assert check(b'\xff', 'utf-8') == ('decode-error', 2)
assert check(b'x', 'no-such-codec') == ('unknown-encoding', 3)
print('ok=0 decode-error=2 unknown-encoding=3')

Expected stdout:

ok=0 decode-error=2 unknown-encoding=3

Reading the result

The example keeps diagnostics short. A command that exposes byte offsets or input paths should avoid printing source contents, because invalid input can contain data that does not belong in logs.

Keep the exit mapping documented beside the command help. A batch runner should not have to inspect a JSON message, an exception class, or localized prose to choose its next action.

The valid path is asserted too, so the command does not only demonstrate failures. Stable successful output is just as important when a shell loop treats zero as permission to continue.

Sources

- Python codecs module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.