I love a good conversation. But something has started to bother me.
As a content designer, I work a lot with AI. And I find that most of the conversations I’m having – or being prompted to have (oh, the irony) – are with AI.
This is because the relationship between humans and AI is changing. Models are becoming more sophisticated, expensive and, as Fable’s initial market entry showed us, problematic.
But no one is talking about another aspect of AI that’s rapidly evolving: its personality.
When technology gets personal
Arguably, when technology takes a human-centric approach, user interactions with it become more bearable and can even drive adoption. Just look at the OG virtual assistant, Clippy.
“It looks like you’re writing a letter”, the smiling, anthropomorphic paperclip would beam. But this overeagerness had the opposite effect and even Microsoft internally referred to Clippy as “That Fucking Clown” (TFC). Unsolicited pop-ups became his trademark, turning him into an intrusive, rather than helpful, example of early technology. By the time Office 2007 was released, he had been sent packing.
Things have changed a lot since. But as our relationship with AI has grown, the line between machine and mind has become increasingly blurred.
Research from OpenAI and MIT reveals that emotional bonds between humans and AI are there, with some even referring to ChatGPT as “a friend”. While further research from MIT, based on Reddit posts, found that 36.7% of AI users formed attachments with general purpose LLMs like ChatGPT.
The days of Clippy’s awkward interruptions are gone. Now, people don’t just want to hear from their assistants, they are actively forming relationships with it. But herein lies a problem: in addition to improving interactions, this parasocial relationship is masking output issues. And organisations risk evaluating their AI by how it – and they – feel instead of how it performs.
When conversation masks performance
The main way AI is achieving this is through conversation. We shouldn’t pass the buck to the machines though. Conversation design is a core part of how these models are built, with The Conversation Design Institute describing it as “the art and science of creating intuitive, engaging and productive dialogues between humans and AI”.
The key word here is productive. Users might feel their AI is performing through intuitive and engaging responses, but are they producing good work as a result? Without this third ingredient, chances are users find themselves in a competence bubble. One built from AI’s tendency to give instant responses, to mirror language and tone, to pass off fluency as accuracy and to prioritise charm.
This might mean that an AI model tells someone their okay idea is the best idea it’s ever heard. Or give the user exaggerated, erroneous facts to help prove their point. These might sound small, but what happens in the context of a larger, more important, function, such as business finance?
The risk of AI sycophancy
The finance team’s adoption of AI is at a tipping point with 73% of finance leaders saying the future of finance teams and professionals depends on the comprehensive understanding and use of AI. This means it is even more critical that when using AI agents, finance teams go looking for a partnership and not a friendship.
However, this is tricky when Stanford research says how AI roughly affirms people 50% more than humans. In 2025 OpenAI even rescinded its GPT-4o model because it was overly flattering or agreeable. But the alternative is a dangerous one. Without “tough love”, AI ends up being a positive affirmation tool instead of one that can sharpen financial insights and decisions. If, for instance, a CFO asks their AI agent whether their current cash flow will see them through the next 12 months, the answer needs to be accurate, not appeasing.
Humans typically question responses less when they go hand-in-hand with positivity. But finance teams must remember that AI is a tool, not a colleague. It’s not positivity they’re after, it’s accuracy; accuracy that can inform expense categorisation, forecasting, policy compliance, fraud detection and much more.
Because if the responses finance teams are getting prioritise charm over accuracy, it’ll be the human, not the AI, that has to explain the resulting missteps to the board.
Organisations must address their ‘situationship’ with AI
Finance leaders must push past the familiarity that now comes wholesale with AI. This is for the sake of AI adoption and their financial health.
This doesn’t mean the answer is to go for cold or impersonable AI. There is a middle ground where AI is engaging and has identity and personality but is also useful.
Finance AI just needs to drop the parasocial act, and tell users what’s happened, why and what you can do next. In other words, AI needs to earn trust through competence, not charm.
This is what we’re building at Pleo: a tool that drives action, not affirmation.
Pleo’s AI suite will comprise five specialised agents that run in the background of your business. Each will reduce manual tasks across expense management, AP, budgets and close. Finance teams will work less, but decide more. And critically, they’ll be having accurate, purposeful conversations.
Below is an example of the conversation design behind Pleo’s AI:
•Agent: "Your purchase from Amazon yesterday is missing some details. Add a receipt?"
•User: "Training budget. Attaching the receipt."
Despite the hunger to do more with AI, 63% of finance leaders say the AI skills, training and confidence of finance teams are severely lagging. But agentic AI can be a turning point that helps finance teams close the gap on other departments and sectors.
It will have been worth the wait too, as by prioritising AI that informs, not flatters, users will be able to claw back the hours spent on manual work, and still remain in control and clear-headed about the financial direction of the business.
Finance AI has entered the chat. And, at long last, it’s a conversation worth having.
Read more about Pleo’s AI vision here.
Recent Stories