When The Assistant Reaches For The Keyboard
A chatbot drafts. A human stays in control. The cost of error is usually time.
An agent commits work. It opens systems, updates CRM, triggers workflows, and acts on behalf of a company. Leadership stops asking whether the model understood the request. It asks:
Can it access the right system without accessing everything? Can it act inside a defined authority boundary? Can we reconstruct the decision? Can we stop it when context changes? Can we recover cleanly when it fails?
Those are trust-stack questions — not model questions.
A Perfect Demo, Then Friday Afternoon
A regional insurance agency deploys an assistant to work inbound leads. In the demo it looks miraculous: form at 10:03, enriched record, producer assigned, follow-up drafted by 10:04. For two weeks, response times fall and meetings rise.
Then a malformed Friday submission is read as a high-value commercial account. The system assigns the top producer, creates duplicates, sends an overconfident coverage message, and fires follow-ups nobody intended. A day of untangling. Nothing catastrophic.
The model was not the failure. The failure was no usable boundary between suggesting work and committing work.
The Moat Moves Up
Model capability is increasingly available from many vendors. The moat belongs to the system that stays dependable when the world is ambiguous:
| Layer | What It Enforces | |---|---| | Delegated identity | Narrow authority — never a shared admin credential | | Policy | Prohibit or require approval regardless of model confidence | | Durable state | Tasks survive retries and handoffs without lost context | | Observability | Tool calls, data access, decisions, and outcomes are inspectable | | Evaluation | Measure real operational work — not abstract benchmarks | | Recovery | Reverse, escalate, or contain before an exception becomes an incident |
Banks do not trust a confident speaker. They establish identity, permissions, limits, and dispute processes. Agents need the same operating discipline.
Appropriate Autonomy
Do not design for “always human” or “fully autonomous.” Design a spectrum.
Enrich a lead, classify a document, draft a response, create a task, send a low-risk follow-up inside policy. Escalate pricing, underwriting, payments, and regulated communications.
The goal is not maximum autonomy. It is appropriate autonomy: enough authority to create value, and no more than the organization can safely supervise.
The Metric That Replaces “Best Model”
Ask: How much verified work can this system complete per dollar, under our policies, with an acceptable failure rate?
Include review time, rework, incident risk, data exposure, and exception cost. A cheap model that creates expensive cleanup is not cheap. A frontier model with no traceability is not enterprise-ready.
Trust Is The Product
Poor governance slows adoption. Teams keep pilots small and bury automation under manual review.
Good controls do the opposite. Leaders expand authority because they can see behavior, measure outcomes, set limits, and intervene.
The winners will not simply build models that can reason. They will build systems that can be trusted to act.
Key Takeaways
- When AI commits work, the bottleneck is trust architecture — not model intelligence.
- Most agent pain is authority and recovery, not insufficient reasoning.
- Appropriate autonomy beats maximum autonomy.
- The board metric is verified work per dollar under policy.
Next Steps
- Mark where suggestion ends and commitment begins for every live or planned agent.
- Replace shared credentials with scoped identity before the next write-access expansion.
- Report verified work, exception cost, and one reconstructable incident — or admit you cannot.
