On 27 March 1977, two Boeing 747s collided on the runway at Los Rodeos Airport in Tenerife, killing 583 people, making it the deadliest accident in aviation history. The collision involved KLM Flight 4805 and Pan Am Flight 1736 in foggy conditions, with runway congestion and communication confusion playing major roles.
The KLM captain began takeoff without proper takeoff clearance. The first officer had earlier pointed out that they did not yet have ATC clearance, and the flight engineer later raised concern about whether the Pan Am aircraft had cleared the runway. But the captain’s authority, seniority, and confidence appear to have made further challenge difficult. The result: 583 people lost their lives.
Today, businesses adopting AI face an analogous risk: they may place a human inside the process, but unless that human has the clarity, authority, confidence, and accountability to challenge the system, “human oversight” can become little more than a comforting label.
This is becoming one of the most important questions in AI adoption.
Many businesses now say:
“Don’t worry. We have a human in the loop.”
But what does that actually mean?
- Does the human understand the AI output?
- Does the human have enough domain knowledge to detect if it is wrong?
- Does the human have time to review it properly?
- Can the human challenge the output?
- Can the human override it?
- Can the human stop the process?
- And most importantly, will the organization support the human when they say no?
If the answer is unclear, then human-in-the-loop may not be real oversight. It may only be the illusion of control.
Human presence is not the same as human control
There is a difference between a human being present in a process and a human being truly in control of that process.
- A human may be copied on an email.
- A human may click approve.
- A human may glance at a dashboard.
- A human may receive an AI-generated recommendation.
- A human may be asked to “review” something after the decision has practically already been made.
But none of these automatically mean the human is exercising meaningful judgment.
In many AI-enabled workflows, the human may be physically or procedurally present, but psychologically and organizationally weak. They may not understand the system well enough. They may be under time pressure. They may not want to question a senior person. They may trust the tool too much. They may assume someone else has already checked it. Or they may simply not have the authority to stop what is happening.
That is why we need to separate four ideas.
1. Human presence
This means a human is somewhere in the process.
They may be observing, receiving, checking, or approving something. But their role may be passive.
Example: An employee receives an AI-generated report and forwards it to a customer.
2. Human review
This means a human looks at the AI output.
But review can be shallow or deep. A hurried glance is not the same as thoughtful verification.
Example: A manager reviews an AI-generated proposal for five minutes before sending it to a client.
3. Human authority
This means the human can challenge, reject, delay, override, escalate, or stop the output.
This is where real oversight begins.
Example: A finance head blocks an AI-assisted payment workflow because vendor bank details were changed and not independently verified.
4. Human accountability
This means a named human is responsible for the consequence.
If the output is wrong, misleading, unsafe, illegal, biased, insecure, or commercially damaging, the business knows who owns the decision.
Example: A founder approves a customer-facing claim after checking that marketing, compliance, and delivery teams can support it.
The problem is that many businesses confuse the first two with the last two.
They assume that because a human is present, accountability exists.
That is dangerous.
Real human oversight begins only when review, authority, and accountability meet.
The dangerous comfort of “human-in-the-loop”
“Human-in-the-loop” has become a reassuring phrase.
- It sounds responsible.
- It sounds safe.
- It sounds balanced.
- It suggests that AI is not acting alone and that human judgment is still involved.
But in practice, the phrase can hide many weak arrangements.
- AI writes the content, and the human publishes it.
- AI ranks the candidates, and HR accepts the shortlist.
- AI generates the code, and the developer merges it.
- AI drafts the proposal, and sales sends it.
- AI summarizes the contract, and the founder signs it.
- AI recommends the action, and the manager approves it.
In each case, there may be a human somewhere in the loop. But is the human truly in control?
That is the real question.
A human-in-the-loop arrangement fails when the human lacks one of five things:
- Competence
- Context
- Time
- Authority
- Organizational support
Without these, human review can become rubber stamping.
The business may believe it has control, but in reality, it has only inserted a human-shaped checkpoint into an AI-driven workflow.
The better question
The question is not simply:
“Is there a human in the loop?”
The better question is:
“Can the right accountable human say no at the right point, for the right reason?”
This is important because in a real business, there is rarely only one human who matters.
Different people may need to say no for different reasons:
A sales person may say:
“The proposal sounds good.”
But the commercial owner may say:
“The pricing is not approved.”
A marketing person may say:
“The message is persuasive.”
But the compliance person may say:
“We cannot legally make this claim.”
A developer may say:
“The feature works.”
But the technical lead may say:
“The architecture is fragile.”
A product owner may say:
“The workflow is smooth.”
But the data/privacy owner may say:
“Customer data is being exposed.”
An R&D team may say:
“The idea is exciting.”
But the technical reviewer may say:
“The assumptions are not validated.”
A founder may say:
“This improves speed.”
But the risk owner may say:
“It changes the company’s liability.”
This is why human oversight is not a single checkbox. It is a network of accountable humans with clearly defined stop rights.
Human oversight is a network, not a person
In many discussions, human-in-the-loop is spoken about as if one reviewer is sitting between the AI and the final decision.
That is too simplistic for business.
A business is not one function. It is a living system made of sales, marketing, operations, finance, HR, technology, compliance, customer trust, data, and leadership judgment.
The same AI-assisted output may create different types of risk for different functions:
- A customer proposal may involve sales risk, pricing risk, delivery risk, legal risk, and brand risk.
- An AI-generated software feature may involve product risk, architecture risk, security risk, data risk, and customer experience risk.
- An AI-generated marketing campaign may involve brand risk, regulatory risk, cultural risk, factual risk, and reputational risk.
- An AI-assisted R&D conclusion may involve technical risk, IP risk, safety risk, feasibility risk, and commercialization risk.
So the real question is not whether “a human” looked at it.
The real question is whether the right accountable human had the ability to stop it.
This is where many SMEs may be exposed.
They may have people using AI enthusiastically, but without clearly defining who owns the output, who checks the risk, and who has the authority to say no.
Stop rights: the missing part of AI oversight
Every serious AI workflow needs what I call stop rights.
Stop rights are the clearly assigned rights of accountable people to pause, reject, escalate, or override an AI-assisted workflow when risk appears.
This does not mean everyone can block everything.
That would create confusion.
It means the business defines who can stop what, and why.
For example:
- Marketing may have stop rights over brand tone.
- Compliance may have stop rights over regulated claims.
Finance may have stop rights over payments.
Data/privacy may have stop rights over customer data exposure.
The technical lead may have stop rights over production deployment.
HR may have stop rights over employee-impacting decisions.
R&D may have stop rights over unvalidated technical conclusions.
The founder may have stop rights over strategic, reputational, or commercial risk.
This is not bureaucracy.
This is accountability design.
When stop rights are unclear, AI adoption becomes risky because everyone assumes someone else is responsible.
The user assumes the tool is reliable.
The manager assumes the user checked it.
The founder assumes the manager approved it.
The customer assumes the company stands behind it.
And when something goes wrong, the business discovers that no one truly owned the decision.
Why this matters more for SMEs
Large companies may have legal teams, compliance departments, data protection officers, internal audit teams, security teams, and formal approval structures.
SMEs often do not.
In a small business, one person may wear many hats.
The founder may also be the CEO, finance approver, sales head, compliance reviewer, data owner, and final escalation point. That is normal. But the hats should not be invisible:
- When the founder approves an AI-generated proposal, are they approving it as the sales head, the finance owner, the delivery owner, or the legal risk owner?
- When an operations manager accepts an AI-generated process change, are they checking productivity, safety, customer impact, data exposure, or compliance?
- When a developer accepts AI-generated code, are they checking whether it works, or whether it is secure, maintainable, and aligned with the architecture?
In SMEs, role clarity matters even more because the same person may carry multiple responsibilities.
AI makes this more urgent.
When AI accelerates output, unclear accountability becomes dangerous faster.
AI can increase speed, but it can also increase unowned risk
- AI can draft faster.
- AI can summarize faster.
- AI can analyze faster.
- AI can code faster.
- AI can respond faster.
- AI can generate options faster.
But speed is not the same as control.
In fact, speed can make weak accountability more dangerous.
- Earlier, a weak process may have produced one bad proposal in a week. With AI, the same weak process may produce ten bad proposals in a day.
- Earlier, a careless developer may have written poor code slowly. With AI, poor architecture can be multiplied quickly.
- Earlier, a marketing team may have made one questionable claim. With AI, hundreds of posts, ads, landing pages, and email campaigns can be generated at scale.
- Earlier, customer support mistakes may have been isolated. With AI automation, the same mistake can be repeated across many customers.
AI does not automatically create risk. But AI can amplify existing weakness.
If the business has clear thinking, AI can amplify clarity.
If the business has weak accountability, AI can amplify confusion.
That is why “human-in-the-loop” must be designed carefully.
The Human Authority Test
Before allowing AI into any meaningful business workflow, SMEs should ask five questions.
1. Can the human understand the AI output?
If the human cannot understand the output, they cannot meaningfully review it.
This matters in technical, legal, financial, medical, compliance, HR, and data-heavy contexts.
A person cannot approve what they do not understand.
2. Can the human detect when the output is wrong?
AI can sound confident even when it is wrong.
The reviewer must have enough domain knowledge to identify errors, missing assumptions, weak reasoning, false claims, or unsafe recommendations.
A fluent answer is not the same as a reliable answer.
3. Can the human challenge the output?
The human must be allowed to question the AI, the vendor, the process, the manager, or the deadline.
If questioning is culturally discouraged, human oversight becomes weak.
A person who is afraid to challenge is not truly in control.
4. Can the human override or stop the process?
This is the heart of the issue.
If the AI output moves forward regardless of human concern, the human is not in the loop. They are near the loop.
There must be a clear way to pause, reject, escalate, or return to manual handling.
5. Will the organization support the human when they say no?
This is often forgotten.
A company may formally allow review, but informally punish delay.
If employees are made to feel that caution is obstruction, they will stop raising concerns.
The business must make it clear that saying no for the right reason is not resistance. It is responsibility.
Examples across business functions
Consider a few common situations.
Marketing
AI generates a campaign promising strong results.
The marketing team likes the message.
But the compliance or founder review may need to say:
“No. We cannot make this claim without proof.”
Sales
AI drafts a proposal with attractive timelines and confident language.
The sales person wants to send it quickly.
But the delivery owner may need to say:
“No. We cannot commit to this scope or timeline.”
Finance
AI classifies invoices and prepares payment suggestions.
The process looks efficient.
But the finance owner may need to say:
“No. This payment cannot move until vendor bank details are verified.”
HR
AI shortlists candidates.
The list looks neat and ranked.
But the hiring manager may need to say:
“No. The ranking is based on weak signals and may exclude good candidates.”
Customer support
AI responds to customer complaints.
The tone sounds polite.
But the support lead may need to say:
“No. This case needs human escalation because the customer is angry and the issue is sensitive.”
Software development
AI generates code that appears to work.
The feature passes the happy path.
But the technical lead may need to say:
“No. This is not production-ready. It is not secure, maintainable, or properly tested.”
R&D
AI suggests a product idea or technical direction.
The concept sounds impressive.
But the R&D lead may need to say:
“No. This is only a hypothesis. It needs validation, testing, and IP review before commercialization.”
Data and privacy
AI helps automate a workflow using customer data.
The business case looks strong.
But the data/privacy owner may need to say:
“No. This exposes sensitive customer data to an unapproved tool.”
These examples show why human oversight cannot be reduced to one approval click.
Different risks need different accountable humans.
The real danger: human rubber stamping
The most dangerous version of human-in-the-loop is not the absence of a human.
It is the appearance of a human.
A human rubber stamp creates false confidence.
It allows the business to say:
“We reviewed it.”
- But the review may not have been meaningful.
- The person may not have had the competence to judge it.
- They may not have had the time.
- They may not have had the authority.
- They may not have had the confidence.
- They may not have had the organizational permission to slow things down.
This is worse than openly admitting that the workflow is automated, because it gives everyone a false sense of safety.
A weak human-in-the-loop process can become a legal, operational, and reputational trap.
The business believes it has control.
But when something goes wrong, the question will be:
- Who approved this?
- What exactly did they review?
- Did they understand the risk?
- Could they have stopped it?
- Was the concern documented?
- Who was accountable?
If these questions cannot be answered, then the human-in-the-loop claim was not governance. It was comfort language.
AI accountability should not mean fear of AI
This article is not an argument against AI. Quite the opposite.
AI can be extremely useful for SMEs. It can improve speed, quality, consistency, documentation, research, customer response, internal communication, and decision support.
The problem is not AI usage. The problem is unaccountable AI usage.
SMEs should not avoid AI because it has risks. Every useful technology has risks. But they should avoid blind adoption. They should avoid unclear ownership. They should avoid allowing AI outputs to move through the business without knowing who is responsible for the consequences. Responsible AI adoption is not about slowing everything down. It is about knowing where speed is safe and where human judgment must remain strong.
The SME owner’s first responsibility
For a small business owner, the first step is simple.
Do not start with a 100-page AI policy.
Start with visibility.
Ask your team:
- Where are we using AI today?
- Which tools are we using?
- What data are we uploading?
- Which outputs are customer-facing?
- Which outputs affect money, people, legal exposure, or customer trust?
- Who reviews the output?
- Who owns the final decision?
- Who can stop it?
These questions alone will reveal a lot.
Many owners may discover that AI is already present in their business through marketing, sales, HR, finance, operations, customer support, software development, and research.
The issue is not whether AI is being used. The issue is whether AI is being used with accountability.
The central principle
The central principle is this:
Human oversight is not a checkbox.
It is a system of authority, accountability, and stop rights.
A human in the loop is not enough if the human cannot say no.
And even that is not enough unless the right human can say no at the right point, for the right reason.
This is the real standard SMEs should aim for.
Not bureaucracy.
Not fear.
Not blind trust.
But accountable speed.
Conclusion
Aviation did not become safer merely by putting humans in cockpits.
It became safer by improving communication, authority, escalation, training, and accountability.
Businesses adopting AI need a similar mindset.
AI does not remove the need for human judgment.
It increases the importance of designing where that judgment must appear.
The question is no longer:
“Is there a human in the loop?”
The better question is:
“Can the right accountable human say no at the right point, for the right reason?”
Until that is true, human-in-the-loop is not governance.
It is only a comforting phrase.
In Part 2, we will look at the next challenge: if every AI output goes through multiple approvals, productivity disappears. So how do we design AI oversight without slowing the business down?
That is where AI accountability becomes both art and science.
About the Author
Bhagath Singh Karunakaran is an entrepreneur, systems thinker, and deep-tech practitioner with over two decades of experience across software, IoT, Industry 4.0, and AI-led business transformation. He is the founder of i45G, where he works with SMEs, institutions, and leaders on practical technology adoption, systems thinking, workforce readiness, and AI-enabled business transformation.
Through his writing and consulting, he focuses on helping business owners and decision-makers move beyond hype and adopt technology with clarity, ownership, and measurable value.
