Justifying selection

In the late 1990s I built an analytics system for the government agency running a workers compensation insurance scheme. It selected employers where premium avoidance looked likely: wages under-declared, a business classified in a cheaper industry than the work it did, or a claims history that didn’t fit the description of its operations. Selection led to an intervention scaled to the risk, from a letter, to a review, to a full audit with an investigator.

What I remember most isn’t the modelling. It’s how much time we spent explaining why, and building that reasoning into the process itself. “The computer says so” was never enough. If we selected an employer, the burden was on us to explain the selection, not on them to prove they were compliant. Every level of intervention needed a reason a person could understand and challenge.

Robodebt, nearly two decades later, reversed that. Debts were raised from averaged income data, and the onus fell on individuals to disprove a number a system had produced about them. The Royal Commission called the scheme “crude and cruel”. It is still baffling, and disappointing, that practices already in use elsewhere in government weren’t followed. It wasn’t a failure of technology.

None of this was new then, and it isn’t now. Many industries have been doing it for decades. Banks have long run model risk management: mathematical assessment of a model’s performance, independent validation, and monitoring for drift. Discrimination law in hiring, lending and insurance has shaped what models may use and how their decisions are explained. Long before AI standards, the people building these systems had to justify them.

What has changed is how many organisations now face the question. With AI, plenty of them are making automated judgements about people for the first time, and frameworks like ISO 42001 make the thinking mandatory, through the impact assessment (clause 6.1.4) and the controls on information for the people your AI affects (Annex A.8).

Once you’ve had to explain a selection to the person who was selected, you design differently. You favour features you can explain, you scale the response to how confident you are, and you put a person between the model and the consequence. That reflex doesn’t come from reading a policy. It comes from having had to answer for the system.

The good news is that there is plenty of precedent for good practice. Nobody needs to invent it, and an organisation new to this can borrow from people and industries that have been doing it for years.

If you’re making automated judgements about people, fractional AI leadership can bring that experience into the design, and a readiness assessment can check whether your governance would hold up.