businesses risk failures from misunderstood AI outputs

Michael Gould has warned that businesses risk major failures if executives rely on artificial intelligence outputs they do not understand. The former Anaplan co-founder said the problem echoes long-standing weaknesses in spreadsheet-driven decision-making.

Gould, who now leads financial modelling software company Kaleidoscope, argued that the biggest risk from AI is not obviously wrong answers but persuasive forecasts and recommendations that decision-makers cannot properly test. Executives, he said, may place too much faith in results they cannot explain, interrogate or trace back to their assumptions.

His comments come amid broader concern about the AI investment surge and the financial returns companies expect from it. Gould argued that too much attention is focused on whether AI spending will pay off for investors, and too little on the risks created when companies use the technology inside their own operations.

“I think we will see major businesses fail because senior executives trusted answers produced by systems they did not properly understand. AI can produce an incredibly convincing forecast, report or recommendation in seconds. The danger comes when the person making the decision cannot explain the assumptions behind it, where the data came from or what happens when those assumptions turn out to be wrong. We are creating the conditions for businesses to make bad decisions faster, at greater scale and with more confidence than ever before,” said Michael Gould, founder, Kaleidoscope.

Gould has spent more than 40 years working on financial planning and modelling systems. He said many businesses are adopting AI while still struggling with fragmented planning processes, conflicting forecasts and models understood by only a small number of staff.

In his view, the issue is less about access to information than about making sense of it. Companies may have abundant financial data, customer information, dashboards and operational reports, yet still fail to build a consistent picture of performance and the assumptions driving decisions.

“Most businesses do not have a data problem, they have an understanding problem.

Finance has one forecast, sales has another and operations is working from different assumptions. Management receives dozens of reports and dashboards and is expected to turn them into a coherent view of what the business should do next. Adding AI to that environment does not fix the problem. It simply gives companies the ability to produce more answers from the same weak foundations,” said Gould.

Spreadsheet parallel

Gould drew a direct comparison between today’s AI boom and the way spreadsheets changed financial planning. Spreadsheets allowed companies to build more detailed models, but that detail often came at a cost. Over time, some organisations became dependent on a handful of employees who understood the formulas and assumptions embedded in them.

That dependence, he said, made errors harder to spot and left senior leaders approving decisions based on models they could not fully question. In his view, AI risks extending the same pattern across a wider range of corporate decisions, and doing so much faster.

“We have already watched this happen with spreadsheets. Businesses built models that became so complicated that only one or two people understood how they worked. Senior management trusted the output because it looked precise.

AI is the same problem with the accelerator pressed down. The technology is more powerful, the outputs are more convincing and the person making the decision can be even further removed from the assumptions producing the answer,” said Gould.

Despite the warning, Gould said he is not arguing that companies should avoid AI. Businesses that use the technology well could gain an advantage, while those that ignore it altogether could fall behind. The difference, he argued, lies in whether organisations combine AI tools with proper understanding and judgement.

“This is not a case for standing still. Businesses that adopt AI well will gain a genuine competitive advantage, and those that ignore it altogether risk being left behind. The potential benefits are real and substantial, but only for organisations that pair the technology with proper understanding and judgement. Used carelessly, the same technology becomes a serious liability rather than an asset,” said Gould.

Planning for uncertainty

Another mistake, Gould said, is treating AI as a tool that can predict the future with precision. He argued that no system can reliably forecast customer behaviour, economic shocks, competitor moves, changing costs or political decisions. Instead, companies should use modelling and AI to test how different scenarios would affect the business.

That shifts the focus from producing a single answer to examining what might happen under different conditions. For executives, Gould said, the more useful exercise is understanding which assumptions matter most and how quickly a plan could fail if those assumptions change.

“Business leaders need to stop asking AI to tell them what is going to happen. The future is not a single number waiting to be calculated.

The useful questions are: What happens if sales fall by 20 per cent? What if our biggest customer leaves? What if costs rise? How long does our cash last? Which assumptions could destroy the plan? AI should help executives explore uncertainty, not give them false confidence that uncertainty has disappeared,” said Gould.

He also said pressure on senior leaders to show they have an AI strategy is pushing some companies to buy tools and build teams before identifying the exact problem they want to solve. In that environment, he argued, spending levels and speed of adoption may matter less than a company’s grasp of its own business drivers.

“Nobody wants to be the CEO who admits they do not have an AI strategy. So businesses are buying tools, hiring teams and adding AI features without first asking whether they understand their own organisation well enough to use the technology properly.

The winners from AI will not necessarily be the companies spending the most money or moving the fastest. They will be the businesses that understand what drives their performance, which assumptions matter and how changes in one part of the organisation affect everything else,” said Gould.

Gould said business leaders should apply the same scepticism to AI-generated recommendations that investors are applying to the wider AI market. “The Bank of England is asking whether investors and businesses have become too confident about the financial returns AI will deliver. Business leaders should ask themselves a similar question about the decisions they are making.

If an AI system recommends investing millions of pounds, closing a division, changing prices or cutting jobs, can the executives approving that decision explain why? If the answer is no, we have a much bigger problem than an AI investment bubble,” said Gould.