Report unexpected success in an expectedFailure unittest

By Batu · English technical notes

Also published in our primary archive.

Report unexpected success in an expectedFailure unittest

Remove an expectedFailure marker once the test starts passing. This controlled suite has two decorated methods: one still raises an assertion failure and is recorded in expectedFailures; the other now passes and is recorded in unexpectedSuccesses. The exact result is one item in each bucket, and wasSuccessful() is False because the passing test still carries the stale marker.

The Python expectedFailure documentation defines the decorator as marking a test as an anticipated failure. It also documents that an unexpectedly passing decorated test is considered a failure by the suite. Inspecting the result categories makes the diagnosis clearer than treating every non-successful run as an ordinary assertion failure.

The example runs the suite with an in-memory StringIO stream so the runner's variable progress text is not part of stdout. Instead, it asserts and prints stable counts from the returned TestResult. In a real suite, delete the decorator after confirming the repaired behavior is the intended contract; leaving it in place means a future regression can be classified incorrectly. This experiment only demonstrates unittest's classification of these two methods. It does not establish that a production bug is fixed or that all test runners display the categories identically. expectedFailure has been available since Python 3.1.

AI assistance disclosure: this article was drafted with AI assistance and executes an isolated synthetic unittest suite.

import io
import unittest

class MarkerExamples(unittest.TestCase):
    @unittest.expectedFailure
    def test_known_bug(self):
        self.assertEqual("old", "new")

    @unittest.expectedFailure
    def test_fixed_but_stale_marker(self):
        self.assertEqual("fixed", "fixed")

suite = unittest.defaultTestLoader.loadTestsFromTestCase(MarkerExamples)
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

assert len(result.expectedFailures) == 1
assert len(result.unexpectedSuccesses) == 1
assert result.wasSuccessful() is False

print(f"expected failures: {len(result.expectedFailures)}")
print(f"unexpected successes: {len(result.unexpectedSuccesses)}")
print(f"suite successful: {result.wasSuccessful()}")

expected failures: 1
unexpected successes: 1
suite successful: False