AI Agent Governance: Decide What Agents May Decide

Written by Breakfast Leadership

Your AI agents are already making decisions, and the only open question is whether anyone chose which ones. AI agent governance is the discipline of deciding, in advance and in writing, what an agent may do on its own, what requires a person, and who answers when it goes wrong. Most mid-market organizations skip that discipline because the tools arrive faster than the policy.

Deloitte's 2026 State of AI in the Enterprise survey of 3,235 information technology and business leaders in 24 countries puts numbers on the gap. By 2027, 74% of respondents expect their companies to be using AI agents at least "moderately." Yet only 21% say their organizations have a mature governance model in place for agentic AI.

This is not a technology problem. It is a decision rights problem. In this article you will see why the gap exists, how to write an agent's decision boundary on one page, who should own each agent, how to report adoption and oversight to your board together, and what to monitor once an agent is live. The advice is written for leaders at nonprofits, associations, healthcare organizations and professional services firms, where an unmonitored agent can become a compliance event.

AI Agent Governance Is a Decision Rights Problem

Leaders tend to treat governance as a review that happens after deployment, a legal checkpoint before launch or a policy document filed with HR. An agent does not behave like a tool in that sense. A tool waits for a person to act. An agent acts, which means someone has delegated a decision to it, whether or not anyone recorded the delegation.

Deloitte's article on the survey, published in April 2026, makes the same point in operational terms. Among the capabilities it lists for agentic AI governance are "clear boundaries for agents that define which decisions they can make independently versus which require human approval." That sentence describes decision rights, not software settings. It is the same question a board asks of any executive: what is this person authorized to commit the organization to without asking?

Consider a membership association whose agent answers renewal questions and offers fee adjustments, or a clinic network whose agent reschedules patient appointments and sends reminders. In both cases the agent is making commitments on the organization's behalf. If nobody wrote down which commitments it may make, the boundary is whatever the vendor's default configuration allows.

Start by listing every agent or agent-like automation now running in your organization, including pilots that departments started on their own. For each, ask one question: what decisions does it make today without a human seeing them first? The answers are usually longer than leaders expect.

Write the Decision Boundary on One Page

A decision boundary does not need to be a long policy. It needs to be a single page that a manager can read in two minutes and an auditor can test. It should say what the agent is for, what it may decide alone, what it may recommend but never execute, and what is never delegated to it at all.

The NIST AI Risk Management Framework, in its Govern function, calls for policies and procedures that "define and differentiate roles and responsibilities for human-AI configurations," which is the formal language for the same page.

Write the boundary in plain verbs. A scheduling agent may rebook an appointment within the same week. It may not cancel a procedure. A donor communications agent may draft acknowledgment letters. It may not promise a naming opportunity. A client research agent may compile sources. It may not send a deliverable to a client without a named reviewer. Each line is a decision someone has made on purpose.

Then add the exception rule: what the agent does when a request falls outside its boundary. The right answer is almost always to stop and route the request to a named person, with the context attached. An agent that guesses at the edge of its authority is the failure you are trying to prevent.

Review the page when the agent's scope changes. Scope creep is how a drafting assistant becomes a sending assistant without a single approval meeting.

Name One Accountable Owner for Every Agent

An agent with five stakeholders has no owner. The IT team runs it, the department head uses it, legal worries about it and the vendor updates it, and when something goes wrong each of them reasonably points to someone else. Governance fails in that handoff far more often than in the technology.

NIST's Govern function addresses this directly. It says that accountability structures should be in place so the appropriate teams and individuals are empowered, and it places responsibility for decisions about AI risk with executive leadership. The practical reading for a mid-sized organization is simple: every agent gets one named executive owner who answers for its outcomes, even if others operate it.

Deloitte recommends cross-functional governance structures that bring together IT, legal, compliance and business unit leaders to set policies, monitor performance and manage escalations. For a mid-market organization, that does not mean a standing committee with a monthly agenda. It means a small group that meets when a new agent is proposed, signs the decision boundary, and sits down again when an incident or a scope change occurs.

Make the owner's name visible. Put it on the decision boundary page, in the vendor record and in the escalation path. When an employee wonders whether an agent's answer is right, they should know whom to ask. This is also where manager capacity matters: the owner needs enough time to do the job, which means the work of oversight has to be planned, not assumed.

Report Adoption and Oversight Together

Boards usually ask for one number: how many people are using the tool. That figure is easy to produce and easy to misread. High usage with weak oversight is a risk. Low usage with heavy oversight is waste. A useful scorecard shows both, side by side.

Gallup's data shows why the usage number alone is not enough. In its 2026 workforce panel, 30% of employed U.S. adults said they use AI frequently, defined as a few times a week or more, while 52% use it at all in their role. Gallup also reports that 25% of U.S. workers strongly agree that their organization has communicated a clear plan or strategy for integrating AI technology or tools into current business practices. That figure counts only the top response, and it measures employees' perception of communication, not the quality of any plan. It still points to the likely gap: people are using tools that their organizations have not clearly explained.

Build the board scorecard around three questions. First, activation: which teams use the agent regularly, and has that changed since last quarter? Second, boundary performance: how often did the agent stop and escalate, and how often did a person overrule it? Third, oversight burden: how many hours of human review does the agent require, and is that burden shrinking or growing?

The third question matters most for organizations short on staff. If reviewing an agent's output takes as long as doing the work, adoption will stall and the pilot will quietly die. Track the review hours now, while the agent is new, so the board sees the real cost and the real benefit.

AI Agent Governance Needs Monitoring and an Audit Trail

A boundary on paper is not governance until someone can see whether the agent respects it. Deloitte warns that "If not properly monitored and centrally controlled, AI agents can make unseen mistakes." Its recommended capabilities include real-time monitoring that tracks agent behavior and flags anomalies, and audit trails that capture the full chain of agent actions to help ensure accountability.

In a regulated environment the audit trail is also your evidence. When a regulator, an insurer or a client asks how a decision was made, you need to show what the agent did, what it was allowed to do and who reviewed the result. A healthcare organization that cannot reconstruct an agent's actions has a compliance problem even if the agent happened to be right.

Write the pause criteria before launch, not during an incident. Decide what would make you stop the agent: a defined number of overridden decisions in a week, a single action outside its boundary, a complaint from a client or patient. Name who has the authority to pause it without asking permission. An agent that can only be paused after a committee meeting is not governed.

Deloitte also notes that the organizations succeeding with agentic AI take a measured approach, starting with lower-risk use cases and building governance capability before they scale. That is the sensible order for a mid-market organization with limited oversight capacity. Prove the boundary, the owner and the monitoring on one agent, then repeat the pattern.

Where the Leadership OS Fits

AI agent governance is not a separate project. It is the same operating discipline a leadership team already needs for every important decision: who decides, who is accountable, how work is reviewed and when the system gets examined. The Leadership OS makes those answers explicit across the executive team, and agents are simply the newest participants in that system.

Leaders who have already clarified decision rights for people find agent governance easier, because the questions are familiar. Leaders who have not will find that agents expose the gap quickly. If this is the situation in your organization, the AI Adoption Advisory is designed to help leadership teams set boundaries, owners and scorecards around the tools they have already bought.

Conclusion

AI agent governance comes down to four decisions your leadership team can make this quarter. Decide what each agent may do alone, in writing, on one page. Name one executive owner for each agent. Report adoption, escalations and oversight hours to the board together. Write the pause criteria and the monitoring plan before the agent scales.

None of these decisions requires new technology. They require leaders to delegate on purpose rather than by default. The Deloitte survey suggests most organizations have not yet done so: strong expectations of agent use, and a minority with mature governance. That gap is where the risk sits, and it is closing only for the leaders who choose to close it.

Start this week. List the agents running in your organization, pick the one with the most authority and write its decision boundary. If you want a second set of eyes on the result, book an intake call with Michael at https://calendly.com/breakfastleadership/intake-call-with-michael and bring the page with you.

One agent with a written boundary is good management. Every agent with a written boundary is a system.

The practices in this article are sound, but they are fragile because they depend on someone remembering to apply them as new agents arrive. Without a standing owner and review rhythm, boundaries drift, scope creeps into new decisions, and oversight becomes an ad hoc fix after the first incident.

Leadership Operating System

Unclear agent authority is a structural gap in decision rights and ownership, not a discipline problem in any one leader. See how a Leadership OS review works.

AI Adoption Advisory

Agents only earn their cost when boundaries, owners and oversight hours are measured alongside usage. See how fractional AI adoption advisory works.

Book a call Either track starts the same way.

Michael D. Levitt is the founder of Breakfast Leadership Network and the author of Burnout Proof and Workplace Culture. More writing at the Breakfast Leadership blog.

Additional Resources

Sources

Breakfast Leadership
Next

Building Meaningful Communication for Your Business