The same AI model can be useful in one workflow and dangerous in another. Outcomes depend on what the system is allowed to do, what it can access, who checks its output and what happens when it is wrong.
Think in terms of verification cost
AI is most valuable when generating a candidate is much cheaper than doing the work manually and checking the candidate is also cheap. Drafting, summarising, restructuring, exploring options and writing code that will be tested all fit this pattern well.
The value falls sharply when verification costs as much as doing the task yourself: obscure legal positions, sensitive figures spread across systems, claims about named people or decisions where the reviewer would have to repeat the whole analysis to know whether it is correct.
Human oversight has to be real
The NIST AI Risk Management Framework asks organisations to define and document human oversight, roles and responsibilities rather than leaving them implicit. NIST AI Risk Management Framework.
A nominal “human in the loop” is not enough if the reviewer has no time, no evidence, no authority to disagree or hundreds of outputs to approve. Effective oversight needs a named owner, realistic review time and a genuine route to overrule the system.
AI literacy is becoming an organisational responsibility
The EU AI Act defines AI literacy as the skills and understanding needed to make informed use of AI systems and understand their opportunities and risks. Article 4 requires providers and deployers to take proportionate measures to support AI literacy among staff and others operating systems on their behalf. EU AI Act — Article 4.
What to do with this
- Sort tasks by verification cost before choosing where to use AI.
- Automate more where checking is cheap and errors are reversible.
- Keep stronger human control where errors are expensive or hard to detect.
- Name the person who owns the output.
- Design for failure: who notices, how quickly, and how the decision gets reversed.
- Keep records of what was generated and what was reviewed.
Rule to remember: do not evaluate only the model. Evaluate the process around it.
