Controlling AI-Machinery
Validating the Validation Process
A couple of years ago, I switched teams to support another team that had been working on a project for over two years. The budget had long crossed its original target, and so had the timeline. The goal, on paper, was straightforward: rebuild an internal server management tool. The old one was outdated and probably from around 1998, held together by workarounds with duct tape and everybody strongly agreed it needed to be rebuilt.
Except... nobody agreed on what "rebuild" actually meant.
The Setup
The company had never really developed its own software before. When they said "make the same tool but new!" what they probably meant was: take this old thing and make it better. But "better" turned out to be something completely different from "the same but newer." It wasn't a copy. It was a new product with different features, different behavior, and a different scope. At its core, the job was: manage servers properly. But what "properly" looked like depended on who you asked.
The team was small. Expensive. Technically skilled. And completely unable to align on what they were building.
There was no proper Product Owner (the person responsible for defining what gets built and in what order) for various reasons, but a few managers. And mostly because: "Why should we", the budget and again: they never had development of this sort before. So — no shared understanding of real user needs. Every team member fought for their own corner. When someone said "that's not my job," they were usually right. It just didn't help, because nobody's job was the thing that mattered most: defining what the product should actually do. But there were a lot of customers, stakeholders and opinions.
We spent almost a year on clarification, while the team built in parallel. Which meant they regularly built half of something, only to discover that what they'd built wasn't what was actually needed. Rework. Throwaway. Features that technically worked but solved no actual problem or the core product was not even working yet. So what good was the feature? Not much.
Early on, I asked a provocative question: "You say you can't just buy a tool because you have special requirements. What are those?" Silence. Which made me give them some homework: go look at existing tools on the market, figure out what they do that you also need, what they don't, and what would actually be missing. And the honest result? They could have bought one. But it wasn't their call. But at least: before it, the assumption that they needed something custom built that was never questioned was now, at least, answered. And some references for clarification did help.
Back to the Future
Now imagine it's today. Same project, same requirements, same mess.
One person with an LLM (Large Language Model, the technology behind tools like ChatGPT or Claude) could probably build the entire tool in a month (or less). I've seen what these models can produce when you give them clear requirements: working code, architecture decisions, database schemas, deployment scripts, full documentation. The technical building part has become almost trivially cheap.
Here's the thing that doesn't get talked about enough: "I" could probably do it. Me. A person who isn't technically educated at all. Who can't write production code if my life depended on it. Who wouldn't know a deployment pipeline from a drainpipe. With the right LLM and enough understanding of what the product should do, I could orchestrate the building of that tool.
That should terrify people. Not because AI is replacing developers. But because it exposes what was always the actual bottleneck!
I'll be upfront: I'm still figuring this out myself. I wrote about the three camps of AI in agile recently, and I landed firmly in camp three: collaboration. Humans and machines working together. But saying "collaboration" is easy. Living it, day to day, is where it gets messy. Let's look at that messy part.
Hidden Safety Nets
So that team didn't fail because they couldn't code. They were brilliant at coding, seniors with a lifetime of experience. They failed because they couldn't answer the question that comes before the code work:
What are we actually building, and how do we know it's right?
That question has a name in agile. Several names, actually: Refinement (the session where teams break upcoming work into buildable pieces). Definition of Done (the criteria that say when work is truly finished). Acceptance Criteria (the observable outcomes that confirm the work meets expectations). Increment validation (checking that what was built actually delivers value). Vertical slicing (delivering thin, complete features instead of building layer by layer).
All of these are tools for the same thing: validation. Making sure you're building the right thing before, during, and after you've built it. And with AI, that is happening at high speed. When building is slow, there's a hidden safety net. The long development cycle gives you time to course-correct. Someone eventually notices that the feature doesn't match the requirement, and it gets fixed because there are six more weeks until release anyway. Validation happens almost by accident, spread across months of building.
AI removes that safety net.
When building takes a month instead of two years, the cost of building the wrong thing doesn't go down. It goes up. You can produce a fully functional product that nobody needs in a fraction of the time it used to take to produce a half-functional product that nobody needs.
Speed without validation isn't efficiency. It's expensive waste with the fast-forward button held down.
The Validation Zipper
Those experienced engineers couldn't validate their own product. Not because they didn't care, I presume. Because validation is a different skill from building, and nobody had ever asked them to develop that skill. They could architect systems and debug at three in the morning. But ask them "what should the user have in options when they open this tab?" and the room went uncomfortably quiet. Ask them "how do we know this increment delivers value?" and you'd get five different answers, none of them from the actual user's perspective.
Now add AI to this picture. AI can build. AI can generate. AI can even suggest acceptance criteria and write test cases. But here's the part that makes the validation gap worse, not better:
AI has a sycophancy problem. It wants to agree with you.
Ask it "is this a good approach?" and it will almost always say yes. Ask it "should we build this feature?" and it will find reasons why you should. Show it your acceptance criteria and it will tell you they look amazing. It's not lying. It's optimized for helpfulness, and helpfulness, most of the time, looks like agreement.
That's the opposite of what validation often needs. Validation needs someone to say: "No. Wait. Not like that. Actually, let's rethink this entirely." The whole point of a refinement session, of a good Definition of Done review, is that someone pushes back. Someone says "...this technically works, but it doesn't solve the actual problem!"
AI rarely does that (if not set up properly). And when the tool you're using to build faster is also the tool that confirms every decision you make, you're not just building the wrong thing quickly. You're building the wrong thing very quickly with confidence.
That gap isn't closing. It's becoming a canyon.
Wants and Needs
Over the past months, I've been working with AI on backlog management (the prioritized list of what to build next) and product refinement. Not as a thought experiment. As daily practice. And what I've learned is that AI makes the human skills around building brutally visible.
Can you describe the behavior you want before someone or something builds it? Can you break a complex product into pieces that each deliver something a user actually wants to experience?
These sound like professional questions. They're not. They're the same question you've been failing at since childhood.
"Mom! Dad! I want Fanta!" That's not a need. That's a want dressed up as a demand. The actual need might be: I'm thirsty. Or: I want something sweet. Or: everyone else is having one and I don't want to be left out. But a child doesn't say that. A child says Fanta. And a product team says "we need a dashboard" when what they actually need is: a way to know if we're on track. The solution gets declared before the problem is understood.
Now hand that habit to an AI. Say "build me a dashboard" and it will build you a beautiful dashboard. It won't question your assumption on its own.
Here's the twist, though: AI can actually be remarkably good at uncovering needs. Ask it "what might the real need behind this request be?" and it will draw on patterns across thousands of similar situations. It can surface what a customer might never articulate. Trained on all of human behavior, AI is a powerful thinking partner when it comes to understanding.
But it only does that when you ask. And asking is the human part. Working with AI requires learning to separate the two: what do my users need, and what solution do I want to offer for that need? AI can help with both. But only a human can decide which needs are real, which matter most, and whether the understanding is deep enough to act on.
Bottleneck-Skills
By now we know that agile frameworks always want their focus on customer and user. But this is not a post that says "agile is more important than ever!" and then lists five reasons to keep doing your daily stand-ups. The ceremonies aren't the point.
The point is the thinking underneath: incremental delivery. Breaking complex problems into small, validatable pieces. Defining what "done" looks like before you start. Checking after each increment whether you're still building the right thing.
That thinking is a skill. And it's the skill that becomes the most expensive skill in a world where building is going cheap.
There's an irony here. The person in the room who won't stop asking questions until they truly understand what the product should do, the one who keeps saying "...but what exactly happens when the user clicks this? WHY does the user not..." while everyone else has moved on,- that person used to be the annoying one. Now they're the most valuable person in the room. Because that obsessive need to understand before building is exactly what validation requires. And AI doesn't have it if not set up to do so persistently.
The team I worked with needed almost a year to develop that skill, supported by a framework and a coach and a lot of patience. They got there. Mostly. They did start asking "what does the user need to experience?" instead of only asking "what does the architecture require?"
But imagine a world where every team needs that skill, not because it's good practice, but because without it, they'll burn through AI-generated code at extraordinary speed, building the wrong thing without ever noticing.
That world isn't coming. It's here.
Figuring It Out
I don't have all the answers on this very big and complex subject.
What I know is, the frameworks I work with are built around exactly the things that I feel matter more than ever: incremental value delivery, clear validation criteria, separation of the "what" from the "how." They were designed for complexity long before AI entered the picture. And from what I'm seeing, they're almost eerily well-suited for a world where the building part disappears as a bottleneck.
But "almost" is doing a lot of work in that sentence. I'm still learning what changes when AI handles the execution. Which parts hold. Which parts need rethinking. What new questions emerge when a single person can orchestrate what used to require a team.
And what I do know is this: the teams and individuals who will navigate this well are the ones who learn to validate. Who learn to define what "done" means before they type a prompt. Who understand that the most dangerous thing about a powerful tool isn't what it can't do.
It's how fast it does the wrong thing when you let it run…Unquestioned.
Also figuring out how to validate what AI should build with your team? Book a free consultation call: