Silent test failures

A test that always passes is worse than no test. It looks like coverage and provides none. Agents produce these by accident with surprising regularity. The patterns below are the common ones; learn to spot them on sight.

The patterns

1. The test that exits before asserting

def test_metric():
    if not torch.cuda.is_available():
        return  # silently skips on CPU CI
    result = compute_metric(model, data)
    assert result > 0.5

On CPU CI this test is a no-op. It "passes." Coverage looks fine. Use pytest.skip(...) so the skip is visible:

def test_metric():
    if not torch.cuda.is_available():
        pytest.skip("requires GPU")
    ...

A skipped test is honest. A returned-early test is a lie.

2. The test that asserts on an empty collection

def test_predictions_are_valid():
    preds = model.predict(test_data)
    for p in preds:
        assert 0 <= p <= 1

If preds is empty, the loop runs zero times. The test passes. Add a length assertion:

assert len(preds) == len(test_data)
for p in preds:
    assert 0 <= p <= 1

3. The test that catches and discards

def test_data_loads():
    try:
        df = load_data()
        assert len(df) > 0
    except Exception:
        pass  # "for robustness"

The agent occasionally writes this when prompted to "make the test more resilient." A test that swallows its own failure is not a test. Delete the try/except.

4. The metric that returns 0.0 on failure

def f1_score(preds, labels) -> float:
    try:
        ...
    except Exception:
        return 0.0

Now your eval script reports F1=0.0 when there's a bug, not when the model is bad. A run with a real bug looks like "model failed completely" instead of "this code threw at line 47." Crash loudly:

def f1_score(preds, labels) -> float:
    if len(preds) != len(labels):
        raise ValueError(f"length mismatch: {len(preds)} vs {len(labels)}")
    ...

5. The mock that lies

@mock.patch("model.predict", return_value=[0.9] * 100)
def test_pipeline(mock_predict):
    output = pipeline.run(input_data)
    assert output["score"] > 0.5

The test passes because the mock returns 0.9. The real model.predict could return anything. The test verifies the pipeline plumbing, not the model. That is fine if you label it as such; it is dangerous if it sits next to a test named test_model_quality.

Convention: mocked unit tests live in tests/unit/, real-stack integration tests live in tests/integration/. Never let a mocked test imply quality.

The detection trick: mutation testing

The cheapest signal that your tests are real:

# break the function on purpose, run tests
git diff > /tmp/safe.patch
sed -i 's/return result/return None/' src/my_module/important.py
pytest -q
git apply -R /tmp/safe.patch

If tests still pass after you broke important.py, your tests are not testing it. Real mutation-testing tools (mutmut, cosmic-ray) automate this. Worth running once per quarter on your eval and metric code.

Test count is not coverage

The agent will gladly add 50 tests to bump a number. Most will be:

Tests of getters/setters.
Tests that exercise lines without verifying behavior.
Tests of code the agent wrote in the same PR (so the test passes by construction).

Better signal: does each test fail when the production code is broken? If not, the test is decoration.

CLAUDE.md rules that catch most of this

## Tests (hard rules)

- Tests use pytest.skip(reason="..."), never bare return, when conditionally skipping.
- Tests on collections assert length first, then per-element.
- Try/except in tests must re-raise or assert. Never silently pass.
- Metric/eval functions raise on bad input. Never return a sentinel like 0.0.
- Mocked tests live in tests/unit/. Integration tests in tests/integration/. Never mock the system under test.
- Tests that exist to exercise lines without behavior assertion are not allowed.

These are mechanical rules the agent can follow. They prevent the worst patterns; review catches the rest.

What to do when you find one

Either make it fail correctly when the underlying thing is broken, or delete it.
If you delete, log it in the PR description so the team knows what coverage shifted.
Add the pattern to CLAUDE.md so the next agent run does not recreate it.

A test that does not fail is taking up cognitive space without paying rent. Evict it.