Smart routing and the cost of getting work done
Routing can reduce model spending while increasing the cost of finishing the job. Start with the customer workload, measure retries and human correction, and compare every router against a suitable single model and practical rules.
These are companion notes. The full essay lives on The Turing Pilgrim, my Substack publication.
Why this matters
An industrial maintenance assistant may extract equipment identifiers, retrieve manuals and investigate ambiguous failures within one customer request. These steps have different capability needs and different consequences when they go wrong. A useful routing policy begins by defining successful work, eligible models and the point at which a person must intervene.
The relevant cost includes more than inference. Routing, validation, tool calls, retries and human correction all contribute to the finished result. The full essay uses an explicitly illustrative example in which lower model spending is overtaken by additional correction costs. Self-hosted models introduce another constraint: directing work elsewhere may leave committed infrastructure costs unchanged.
Open models expand the choices, while cloud platforms, specialist providers and open-source infrastructure make routing widely available. A durable advantage requires evidence that particular decisions improve customer outcomes. That learning may belong to an application vendor or enterprise with workflow access and data rights. An independent router must keep demonstrating value as models and customer workloads change.
Key takeaways
- Define the customer outcome before choosing an optimization metric. Track cost per successfully completed task alongside quality and delivery time.
- Separate model selection, provider selection and infrastructure scheduling. Identify which decision creates value for the workload.
- Include validation, retries, tool calls, human correction and committed infrastructure in the comparison.
- Test a router against the best suitable single model and sensible task-specific rules on held-out cases.
- Build feedback around accepted, corrected and rejected outcomes, while recognizing that acceptance alone does not establish correctness.
Who should read it
B2B product leaders and enterprise teams deciding where model routing can improve service economics and who can retain the resulting advantage.
Related field notes
The exception queue is the real interface for industrial AI
Make the remaining human decisions understandable when software handles routine work.
A flexible data center starts with the customer contract
Translate customer commitments into realistic operating boundaries for compute services.
The next AI shift is delegation
Define which decisions an AI system may take on behalf of its users.
Working through this decision?
A short note on the AI bet you are weighing is enough to start. I usually reply within one business day.