Tests that repair themselves, and the ones that should not

Every automation suite starts clean and ends up distrusted. The sequence is always the same: a build goes red, someone looks, the application was fine and the selector had moved. That happens a few more times. Then a team learns the habit that kills a test suite — re-running until green, and shipping on the second result.

TestNova repairs its own tests when the application changes underneath them. That removes most of this decay. It also introduces a real risk, and the interesting engineering is in telling the two apart.

What a self-repairing test actually does

When a step cannot find what it is looking for, the naive response is to fail. The useful response is to ask a narrower question: has this thing moved, or has it gone? A button whose id changed but which sits in the same place, carries the same label and does the same thing on click, has moved. A button that is simply not on the page any more has gone.

Repair is legitimate for the first case and dishonest for the second. The whole design problem is drawing that line and refusing to let it drift.

Three rules we enforce

1. Repair identity, never intent

A test may re-find an element by a different attribute. It may not decide that a different element will do. If the step was "click Checkout" and there is no Checkout, the correct outcome is a failure — not a click on the most similar-looking button on the page.

2. Every repair is reported as a change, not silence

A repaired run is green, but it is green with a note: this selector moved, here is what it was and what it is now. Repairs that nobody reads accumulate into a suite that no longer describes the product. Repairs that are reviewed are free maintenance.

3. Assertions are never repaired

Locating an element is a mechanical detail of how the test reaches the thing it cares about. The assertion is the thing it cares about. Expected text, expected total, expected state after an action — these fail loudly, always. Self-repair stops at the assertion boundary, and this is the rule we would never trade away.

Why flaky tests are usually not test problems

In our own work, most instability traces back to one of four causes, and only the last is really about the test:

• Test data that changes under you. Shared environments where another run, or a person, edits the record your test depends on.
• Timing assumptions. Waiting for a fixed number of seconds rather than for the condition that actually matters.
• Markup with nothing stable to grab. Generated class names and no test attributes mean every selector is a guess with a short shelf life.
• Tests that know too much. A test asserting on internal structure breaks when the structure changes, which is not a defect.

Self-repair helps with the third. It papers over the first two, which is why reporting matters so much: without the note, a repaired run quietly converts an environment problem into a false pass.

Real devices change the answer

A suite that passes in a desktop browser and a simulator tells you about a desktop browser and a simulator. Keyboards that cover inputs, permission dialogs, interrupted network, a date picker that behaves differently on one Android version — these appear on real hardware and nowhere else. TestNova runs on real phones and real browsers for that reason, and it is the single change that has caught the most defects that would otherwise have reached users.

How we use it on ourselves

TestNova runs against our own releases before we propose it to anyone. That is partly a quality argument and mostly a design one: running your own tool on your own product is the fastest way to find the places where it is annoying, and annoying is how a test suite starts being avoided.