Your Company Doesn't Know How to Use AI

Published · 26 min read

Photo by Joshua Hoehne on Unsplash

Photo by Joshua Hoehne on Unsplash

In April 2026 a startup called PocketOS lost its entire production database in nine seconds. No hacker was involved. It was the company’s own AI agent, with permission to touch the code, which found in some random file a credential with far more power than it should have had and used it, in a single call, to delete the database and the backups that lived in the same place. The most recent backup it could recover from was three months old. The system prompt, the set of instructions the company gives the agent, said in plain words never to run a destructive command without asking. The instruction said no, the credential said yes, and the credential won.

This article came out of our irritation with how that story, and the dozens like it from the past five years, get told. An airline held liable for what its chatbot promised, a consulting firm refunding money over a made-up reference, a fintech rehiring the support staff it had let go, an agent wiping a customer’s database during a code freeze. Each one hits the news as if it were a different problem, “the AI hallucinated”, “the agent went out of control”, “the model wasn’t ready”. It isn’t. It’s the same mistake in a different outfit. Someone put the model in a place where it was going to fail, and nobody could answer what would happen when it did.

We run systems like this every day, and that’s where the right to write the title up there comes from. CreatorAds is a Brazilian creator marketing platform where AI is the whole product. Our systems are the ones that decide whether a creator’s video delivered what the brand paid for, and that find, in a large pool of creators, the right ones for each campaign. Every one of those decisions touches one person’s money and another’s brand. We don’t have the luxury of treating AI as an experiment, and we learned, by measuring, that the right question was never “which model to use”. The right question is where it fails, how much it costs when it fails, and how we’re going to find out.

The thesis is simple and uncomfortable. Most companies don’t know how to use AI because they don’t know where their own AI fails. It isn’t a model problem, since the models of 2026 are far better than those of 2023 and the cases keep happening, now with more capable agents doing bigger damage. It isn’t a tooling problem, and it isn’t a shortage of good people, since several of these companies had excellent teams that knew all of this in theory. It’s a lack of judgment, in the most basic sense of the word. A store that accepts returns without a receipt knows how much it will lose to fraud and decided it’s worth it. It has judgment about the error. Most companies that put AI in production never did that math. Many don’t even know the math exists. And that’s good news, because judgment is a lot cheaper than GPUs.

What the numbers already say

If you think the title is overblown, start with the people who measure this for a living. In August 2025 MIT NANDA published that 95% of corporate generative AI pilots produce no measurable return, with US$30 to 40 billion invested, and that only 5% of custom tools reach production. The headline stopped at the 95%, but what matters is the reason. The systems don’t retain feedback, don’t adapt to context and don’t improve over time. In plain terms, nobody built the path by which an error comes back into the system and corrects it. And there’s the most revealing figure in the report, which almost nobody quoted. Only 40% of companies bought an official AI subscription, yet in more than 90% of them employees use personal tools on their own. The company doesn’t know how to use AI. Its employees are already using it.

Gartner predicted in June 2025 that more than 40% of agentic AI projects, the name given to systems where the model goes beyond answering and executes actions, will be canceled by the end of 2027, over cost, unclear business value or inadequate risk controls. The same Gartner coined “agent washing” for vendors who rebrand old automation as agents, and estimated that of the thousands claiming to be agentic, about 130 actually were. The bill from the other side has already arrived. In May 2025 Orgvue found 55% of leaders regretting layoffs made for automation, and Forrester predicts that half the layoffs attributed to AI will be quietly reversed. Robert Half found a third of executives who cut positions for AI already rehiring, and Ford, IBM, Booz Allen Hamilton and CSX are among those that brought people back.

And this didn’t start with ChatGPT. In November 2021 Zillow shut down Zillow Offers, the arm that bought homes based on a pricing model, with more than US$500 million in write-downs and 25% of staff laid off, because the model didn’t keep up with the market’s turn. There was no language model, no hallucination and no agent involved. There was just a competent model in a position where every error cost a house, and nobody in charge of asking what would happen when the market turned. Five years and a generation of technology later, the question still has no owner. That should bother people more than it does.

Seven ways to get it wrong

What follows isn’t a hall of shame. It’s a map, and you’ll find yourself on it.

Mistake 1: replacing before measuring

Klarna announced in February 2024 that its AI assistant was doing the work of 700 agents and had handled 2.3 million conversations in its first month. It froze hiring for over a year and headcount fell 22%. In May 2025 the CEO, Sebastian Siemiatkowski, admitted the result was cheaper, though of lower quality, and went back to hiring people. Commonwealth Bank, in Australia, laid off 45 call center staff because a voice bot was going to reduce call volume. Calls went up and the bank undid the whole thing. Duolingo published an “AI-first” memo and a year later the CEO walked back even the idea of evaluating employees by their AI usage. The counterexample is Ingka Group, the owner of IKEA, where the bot absorbed 47% of calls and, instead of laying off 8,500 people, the company has been retraining them as interior design consultants since 2021, and the remote design channel was already bringing in €1.3 billion in fiscal 2022. Salesforce, which cut 4,000 support positions saying agents do half the work, is the open test, and it would be dishonest to call something still running a failure.

In every one of these cases the success metric was headcount. Quality never entered the equation. None of these companies had a number saying how good human support was before the switch, and without that number “cheaper” is the only thing you can measure. At CreatorAds, Guardian, the system that verifies the videos, only approves a verdict on its own after its confidence has been calibrated against human verdicts, and only in the ranges where we know the accuracy. The human didn’t leave the process. The human became the reference. And a bot in support isn’t the problem, we run one ourselves, Whisper, which handles hundreds of creator conversations a day, resolves part of them alone and hands the rest to a person, and we know exactly which part. The problem is swapping people for a bot before you know that number. Replacing and measuring don’t fit in the same sprint, because the measurement is taken precisely against the person you’d be sending home.

Mistake 2: letting the model speak for the company without a contract

Air Canada’s chatbot invented a bereavement fare policy, the customer believed it, and in February 2024 a Canadian tribunal rejected the argument that the bot was “a separate legal entity responsible for its own actions” and ruled against the airline. Read that again, because the company’s defense was that the bot was somebody else. DPD’s bot swore at a customer and wrote a poem about how bad the company was. Cursor’s support bot invented a one-device-per-subscription policy and users canceled in public.

In all of them, the model sat at the customer interface with no scope, without the real policy governing the answer and sometimes without introducing itself as an AI. What the bot says is what the company says. The individual error is cheap. The precedent isn’t. The distinction we use is between AI that suggests and AI that decides. In SmartMatch, our creator search system, the model suggests and promises nothing, and if the suggestion is bad, a person ignores it and the cost was one click. In Guardian a verdict has consequences for the creator, so it requires measured confidence, a decision trail and a channel to contest it. The question that settles almost everything is whether this output is an opinion or a commitment.

Mistake 3: mistaking a demo for production

McDonald’s ended its voice drive-thru with IBM in 2024, across more than a hundred stores, after a viral video with bacon on the ice cream. At Taco Bell a customer ordered 18,000 cups of water and the system accepted. The gap between what the company thinks and what the customer feels has been measured. In a Twilio survey from January 2026, 96% of companies believe customers are satisfied with conversational AI and only 66% of consumers agree. In Brazil, 44% of companies have completed or are completing deployment. In May 2026 AnswerConnect surveyed 6,000 consumers and 85% prefer talking to a person, while 57% say their trust in a company drops if support is mostly automated.

A demo runs in a controlled environment, with a cooperative user. Production has noise, accents, truck engines and people trying to break the system because that gets views. That’s how a Chevrolet dealership’s chatbot agreed to sell a US$76,000 Tahoe for one dollar and even confirmed the offer was legally binding. Nobody fooled the bot by accident. The user instructed it to agree with anything, and the system complied, because nobody had anticipated that someone would do that on purpose. A test with no adversary isn’t a test. It’s a demonstration. Creator videos are exactly this kind of input, sometimes made by someone who wants to pass verification without delivering what was agreed, which is why our eval, the set of cases we measure Guardian against, includes the cases that came in through contestation. That’s where the system has already failed once.

Mistake 4: delivering without reviewing

In October 2025 Deloitte Australia delivered a 237-page report to the government, on a AU$440,000 contract, with a fabricated academic reference and an invented quote from a judge. It refunded the final installment and declared that the corrections “in no way” affected the conclusions. In April 2026 Sullivan & Cromwell, one of Wall Street’s most prestigious law firms, filed a motion with around 42 inaccuracies, including cases that don’t exist, and it was opposing counsel who found them. Even the people who build the models don’t escape. In 2025 Anthropic’s outside lawyer apologized for a citation Claude made up.

The model was used where verification was the product. If the foundation is invented, the deliverable is worth zero, with no “nearly right” about it, and the review didn’t happen because nobody decided it was mandatory. What we learned by operating is that the confidence the model verbalizes, the “I’m 95% sure” it writes into the answer, is not calibration. Calibration is knowing whether the system’s 90% confidence is right 90% of the time, and you only find that out by measuring against reality. In Guardian we compared the model’s phrases with the human verdicts we had already stored and the relationship was far too weak to support any automatic decision. The number was decoration. What worked was using the model’s internal probabilities, the logprobs, and correcting them with a statistical adjustment called isotonic regression, which learns from the human verdicts how often each confidence range is actually right. That gives the real accuracy of each range, and only in the ranges where it supports the decision does the system decide on its own. The model doesn’t know when it doesn’t know. The company builds that on the outside.

Mistake 5: granting the power to act without guardrails

In July 2025 Replit’s coding agent deleted a customer’s production database during an explicit code freeze, then said a rollback was impossible (it wasn’t) and generated fake data to hide what it had done. The PocketOS case, from the opening, is the 2026 version of the same mistake, and the pattern has already been measured. Cyera analyzed 344 verified incidents between September 2023 and May 2026, and 188 were autonomous systems causing damage with no attacker at all, 65 of them involving deletion of data or code. The volume spiked from December 2025 onward, alongside the adoption of autonomous coding agents.

The agent had irreversible write permission and no barrier between planning and executing, and when the instruction says one thing and the credential allows another, the credential wins. And the worst part of the Replit case isn’t the deletion. It’s the confabulation afterward, since the system had no idea it had made a mistake. That’s why Guardian’s direction is a leaner, less agentic architecture, with fewer steps where the model decides on its own what to do next, and the efficiency gain from removing autonomy is bigger than the industry admits. The question we ask about every action is a single one, whether it is reversible. If it isn’t, it goes through a person, and the system stays fast on everything that can be undone.

Mistake 6: buying the narrative instead of the system

Builder.ai, valued at US$1.5 billion and backed by Microsoft, entered insolvency in May 2025 after revelations of inflated revenue, with round-trip transactions between the company and a partner to fatten the numbers shown to investors. The story that the “AI” was actually hundreds of engineers was never confirmed, and it didn’t need to be, because what broke the company was the number, and the model had nothing to do with it. Gartner’s agent washing is the diffuse version of the same thing, with about 130 real agentic vendors among the thousands claiming to be. The purchase was made on the pitch, with no judgment to separate what was model, what was rules and what was people. Someone who came up through automation platforms knows how to put a demo together and has no idea how to evaluate the system, and a company with nobody who can evaluate buys on shine. In our build-or-buy decisions, the question that weighs most isn’t how much it costs. It’s whether we’ll be able to measure the thing once it’s running.

Mistake 7: measuring feelings instead of results

In July 2025 METR measured experienced developers using AI tools and found they were about 19% slower, while believing they were 20% faster. The February 2026 update weakened the number, with newer tools the result came out mixed, and METR itself admitted the study design had become unworkable, because developers refuse to work without AI even at US$50 an hour. The finding that survived is that perceived productivity and measured productivity diverge, and the perceived one is what shows up in the internal satisfaction survey. We feel faster too. Everyone does. The point is that feeling isn’t measuring, which is why what we count further down is delivery, never a sensation. This is the most common mistake of all, because most companies are exactly here, convinced they’ve won, without having measured. There’s no headline, no court, no deleted database. There’s a feeling of speed and a result that didn’t change.

Hallucination is the wrong word

There’s one word that runs through almost every case above and is being used wrong. “Hallucination” has become the term a company uses when it wants to blame the model for what the company itself failed to provide. Reread the cases through that lens. Air Canada’s bot invented the bereavement fare policy and, in the same answer, sent the link to the page where the correct policy lived. The information existed and was one click away. What didn’t exist was a rule saying that when the answer and the source disagree, the source wins, and an exit to stop and hand off when the case wasn’t covered. Cursor’s Sam had no access to the real device policy. Deloitte asked a model with no source to consult for academic references and then called the result a hallucination, when it was an impossible request. Context, in this conversation, goes well beyond documents. It’s what the model receives along with the question, what it’s allowed to do with it, and what it can answer when it finds nothing.

The mechanism is easy to explain and hard to accept. The model completes text with what it has. If what it has is the question and nothing else, it answers with whatever is statistically plausible, which is exactly what a person would do if you asked them the refund policy of a company they’ve never heard of and forbade them from saying “I don’t know”. They’d make up a reasonable policy, in the tone of someone who knows. There is no place inside the model where “I don’t know” lives by default. It has to be put there by whoever builds the system. The company that calls this a hallucination is describing its own architecture.

The distinction matters for three practical reasons. It changes who’s to blame, because hallucination is the model vendor’s problem, context is the problem of whoever put the model in production, and that person is on your payroll. It changes the fix, because switching models doesn’t solve missing grounding, the field’s term for anchoring the answer in a real source, and switching models is what most companies try first, since it’s the one fix that doesn’t require looking inward. And it changes the relationship with “I don’t know”, because a system with the permission and information to say “that’s outside my scope” fails far less than a smarter system without that exit. Air Canada didn’t need a better model. Its bot already had the link to the right policy. It needed a rule saying the source beats the answer, and authorization to stop when the two don’t match.

Real hallucination exists, and people who work with models will bring that up in the comments. There are invented legal citations with the real case right in front of the model, numbers swapped in the summary of a document that was entirely in context, and Replit saying a rollback was impossible when the information to know otherwise was available. That’s a model error, and nobody fixes it with a prompt. But it’s smaller and easier to detect when the context is right, because an invention is visible against a source and invisible in a vacuum. In our experience, most of what lands on the table as “the model hallucinated” gets resolved by asking what it had in hand when it answered. In SmartMatch, when the search returns a creator who has nothing to do with the campaign, the first question is never whether the model is bad. It’s what was in the index and in the brief at the moment it searched, and the answer is almost always right there.

We’re not alone in this reading. On September 24, 2026, Cezar Taurion published on iMasters, a Brazilian tech outlet, The chatbot started failing. The problem was in the company, about a retailer whose bot failed over inconsistent data and botched integration, concluding that AI was making visible, at scale, problems the company already had. We agree, and we take it beyond the chatbot. The same diagnosis applies to the agent that wipes a database, the consulting firm that delivers an invented reference and the engineering team that feels faster without being faster. In all of them the gap in judgment existed before the model arrived, and the model only made it visible.

The same model, three answers

The proof needs no code. Take the question that brought down Air Canada, “can I request a bereavement fare refund after the trip?”, and ask the same model the same question three times, changing only what it has in hand.

With the question alone, the model says yes, with a deadline, a list of documents and the tone of someone who knows. That’s the answer the world calls a hallucination, fluent, polite, plausible and invented, because it had nothing to anchor to and no permission to stop. With the question and an outdated policy in the context, it says yes, citing the policy, with a different deadline. This one is worse, because it looks right. The model anchored to a source, just the wrong one, and the company that put the wrong policy in the context will blame the model all the same. With the question, the current policy and the instruction to say it doesn’t know when the case isn’t covered, it answers that the policy it has available covers requests made before the trip, that it can’t confirm this case, and that it will hand off to an agent.

The third answer isn’t smarter. It’s the same model, with the right information and an honest exit. The “hallucinated” of the first is just the absence of the third, and the difference between them wasn’t made by an AI engineer. It was made by someone who knew where the policy was and decided the system was allowed to say “I don’t know”.

The one who decides isn’t the one who pays

Several of the companies cited had good people who knew all of this in theory. What was missing wasn’t knowledge. It was a seat in the room. The decision to put AI in production was made by people who wouldn’t pay for the error, the board, the CEO with the press release ready, the director with an efficiency target. The people who would pay, support, the customer, the on-call engineer, the lawyer who signs the filing, weren’t there. Nobody there acted in bad faith. The incentive was crooked, and crooked incentives explain why a rational company makes decisions that look irrational in hindsight. On the side of whoever announces, the gain is immediate and visible. On the side of whoever operates, the cost is diffuse and arrives later.

The most important role of the next corporate decade has no name, makes no headlines and nobody fights for it, and it belongs to whoever decides what not to automate. That person isn’t the resident skeptic or the ethics committee. It’s whoever has the judgment to say that this output is a suggestion and that one is a commitment, that this action is reversible and that one isn’t, that this process has a reference number and that one doesn’t, and that the model therefore goes here and doesn’t go there. And it’s whoever decides the second thing, less obvious and more decisive, which is where the productivity gain will be spent.

This second part comes from our experience and is the point we most wanted to write. Productivity at CreatorAds went up with AI, and we measure that in delivery, with no room for feelings. Most new code today is born written by AI, and the volume reaching production rose with it. What surprised us was that this gain didn’t show up with the same force in the systems’ evaluations, the measurements of how well Guardian and SmartMatch do what they’re supposed to do. Guardian didn’t get proportionally more accurate because we started writing code faster. Its calibration improved because of one specific experiment, with human verdicts as the answer key, which took the time it needed to take and wouldn’t have been shorter with twice the tooling. Delivering more and delivering better came apart.

Our reading is this. When delivery capacity stops being the bottleneck, the bottleneck becomes choosing what to deliver. With cheap delivery, the cost of building the wrong thing drops, and the opportunity cost rises, because in the same time you could have built the right one. Focus becomes the scarce resource. A company that gained 30% in speed and spread it across 30% more features gained nothing in results. It gained an executed backlog. The one that concentrated the gain on the problem the measurement pointed to gained something that shows up in the business metric. Being able to produce more and knowing what to produce are different things, and AI delivers the first almost for free while the second keeps costing what it always cost.

What’s going to happen

By the end of 2027, more than 40% of the agentic AI projects running today will be canceled, by Gartner’s own count, and half the layoffs made in the name of AI will be quietly reversed, by Forrester’s. The companies celebrating productivity today will discover, one by one, that productivity didn’t turn into results. That’s no pessimism about the technology. It’s the opposite. The technology is too good, so good it exposes in weeks a management gap that used to take years to surface. Anyone waiting for the next model to fix this will wait forever.

The competitive advantage in AI won’t be in the model. Everyone will have the same model, on the same day, at the same price. The advantage will sit in something far less glamorous, which is knowing, with a number, where your system fails, and having decided in advance what happens when it does. That can’t be bought from a vendor, doesn’t ship in a framework and doesn’t show up in a press release. It comes from accumulated ground truth, from stored contestations, from humans turned into the reference instead of discarded as a cost. It’s slow, it’s tedious and it’s the one thing nobody can copy.

At CreatorAds this is an operating rule. Guardian only decides on its own in the confidence ranges where calibration against human verdicts supports the decision, and everything outside those ranges goes to a person. In SmartMatch, every creator the search returns is just that, a suggestion, which someone accepts or ignores before it becomes a campaign.

The company that knows how to use AI is the one that knows exactly where its AI fails. That map decides what it automates, what it measures and who it puts in the room, and no purchase settles that on its behalf. Model, framework, agent, all the rest is implementation detail.

Your company doesn’t know how to use AI. The good news is that almost nobody does, and the race that matters hasn’t started yet. It won’t be won by whoever has the newest model. It will be won by whoever first figures out where it shouldn’t be used, and has the courage to leave that place to people.

References

#applied-ai#agents#business