Avoid unnecessary inference
Every call to a model costs money, adds delay, and introduces a way for the system to be wrong that did not exist before. A surprising amount of what gets sold as AI is a lookup, a rule, a calculation or a template wearing a costume.
The question to ask first
The first question on any workflow is not where can we add AI. It is what here actually requires judgement?
Judgement means reading something unstructured, weighing an ambiguous case, or drafting language a person will review. Everything else — retrieving a price, applying a discount rule, formatting a document, checking whether a field is populated — is a solved problem with a deterministic answer.
| Needs judgement — use a model | Needs an answer — use code |
|---|---|
| Reading a free-text customer enquiry and working out what is being asked | Looking up the price and availability of what they asked for |
| Summarising a month of complaints into themes | Counting how many complaints there were |
| Drafting a reply a person will edit and send | Deciding which template applies |
| Judging whether an unusual case needs escalation | Applying the escalation threshold once the judgement is made |
What each unnecessary call costs
- Money — a per-call charge on work a function would have done for nothing, multiplied by every run, forever
- Latency — a wait where there could have been an immediate answer
- A new failure mode — something that can now be confidently wrong in a way a lookup could not
- Untestability — deterministic code can be tested exhaustively; a model can only be sampled
- A dependency — on a provider, their pricing, their availability, and their next model update
If a rule would do, a rule is the better engineering. It is also the cheaper invoice.
The result is a smaller system
Applied honestly, this usually produces something smaller than the client expected. Less of the system depends on a model, so it costs less to run each month, breaks in fewer places, and can be properly tested.
Inference is a tool with a price. The discipline is spending it where it buys something.