Implementing AI: from pilot to production

How do you implement AI? Four steps

Every implementation I do goes through the same four steps. The order matters more than the technology: whoever starts at step three builds something nobody can call finished.

1. Choose one workflow

One process with a bottleneck you can measure, not five at once. If there are several plans, I assess them in the Scan and say per plan: stop, scale or rebuild.

2. Define when it is done

Which questions or tasks the application handles, how many errors are acceptable, how you measure that and who judges it. That agreement is in writing before I build.

3. Build on real data

In your own systems, with the people who will use it. I first measure how the process runs now, so the difference is visible afterwards.

4. Hand over

An owner inside your organisation, a test set that keeps running and a one-page runbook. The full handover list is on my page about forward deployed engineers.

What happens week by week

A Pilot-to-Production sprint takes four weeks at most. This is what it looks like with me.

Before the start

I look at the material before I name a price: how many documents there really are, how many exceptions the process has and which systems touch it. A day of looking prevents a quarter of overrun.

Week 1

The definition of done in writing: which questions, which error rate, which system, which reviewer and which measurement method. This week I also find the person who knows the work and can bring colleagues along.

Day 5

A fixed decision point with three outcomes. Simple: we keep building. Medium: you choose between a smaller version now or a larger project. Complex: the sprint format does not fit, and you hear that now instead of in week four.

Week 2

The baseline: minutes per task, errors per batch and how many people really use the existing software. Alongside it I build the first version on real data.

Week 3

Testing with the people who will use it, on their data and their exceptions. Then I measure the difference with week two.

Week 4

Handover: owner, test set on your machine, runbook, an agreement on who steps in when something goes wrong and a plan for the next model version. Only when those five are in place do I leave.

Checklist: is your pilot ready for production?

Hold your own pilot against these eight questions. Every no is a reason it stays stuck in a demo, even when the technology works.

Problem

Can you say in one sentence which problem this solves, for whom, and how much time or money it saves per week?

Definition of done

Is it in writing which error rate is acceptable and who judges it?

Baseline

Do you know how long the task takes now and how often something goes wrong now?

Real data

Has the pilot been tested on data from day-to-day work, odd cases included, and not only on a tidy selection?

Systems

Does it run in the software people already use, or do they have to open yet another screen?

Owner

Is there one name in your organisation that owns the system after go-live? Not IT and not the agency.

Control

Is it agreed who spots an error, who corrects it and how?

Users

Did people from the middle group take part, not only the enthusiasts who were using it anyway?

Where AI implementations get stuck most often

In the 2024 BCG survey of 1,000 executives, 26% of companies could show measurable value from AI: 22% early value and 4% substantial value. That is a classification of companies, not a success rate for individual applications. In my own projects I saw four causes come back again and again.

The wrong problem

Conversations start with a solution: “we want a chatbot”. After asking further, the bottleneck often sits elsewhere, or in software that already exists but is not used. Building more does not fix that.

The platform ceiling

The chosen AI platform, your own systems, your licences and who has to sign off together decide what is possible. A better prompt does not change that. Before the start I ask for three concrete limits.

Green on the builder’s test set

An application can score well on the builder’s test set and behave differently on day-to-day data. That is why I test on your data and the test set runs on your machine afterwards.

A blocker that is not a blocker

An integration stalls on security, while the same information can be reached another way. For every blocker, first ask which data you minimally need and where it already sits.

Why not every task is an AI task

AI does not help with every task. In an experiment with 758 BCG consultants, people using AI completed 12.2% more tasks and worked 25.1% faster. On a task outside what the model could handle, they were 19% less likely to reach a correct solution (Dell’Acqua et al., Harvard Business School, 2023).

The boundary runs straight through work processes: two tasks that look equally hard can sit on different sides. That is why step one tests each task against that boundary, on a handful of real examples, before a line of code is written.

After go-live: who actually uses it?

An application used by twenty enthusiasts and ignored by two hundred colleagues is not an implementation. In the organisations I worked in, I kept seeing three groups: a small group that starts on day one, fifteen to twenty percent who only move once there is proof, and in between a middle group of sixty to seventy percent who are fine with it but will not start on their own.

That middle group decides whether it works. They need examples from their own work and a colleague who shows them. That is why I involve users from step one, and make sure that by week four at least two people can explain the application, so it does not stall when the one enthusiast is on holiday.