From building AI to deciding on AI

A year ago, first conversations often started with a build request. Can you make a chatbot? Can you make our documents searchable? Can you automate this workflow?

That work still exists, but it is less often the real problem. At a client, existing software proved to be a useful alternative to custom development. At another, the problem lay in how existing software was used: people left the available features untouched, and building more would not have fixed that. Since then I first investigate what is already possible, and only then decide what still needs to be built.

Building is less scarce

ChatGPT, Claude, Copilot, and vertical tools now solve more standard work than they did two years ago. A good prompt inside an existing product sometimes beats a €15,000 bespoke application.

The complexity has moved. Building takes weeks. Deciding which problem to take on, how it fits the existing workflow, who owns it, and how you measure whether it works is at least half the work. The centre of gravity shifts from building to choosing.

The numbers point the same way. In BCG’s 2024 survey of 1,000 executives in 59 countries, 74% of companies had yet to show tangible value from AI; 26% were already getting value from it. That classifies companies, it is not a success rate for individual pilots. It does show that access to tools says little about results.

AI consultancy or build it yourself?

Build it yourself when the problem, the users, and the owner are already clear and you have an engineer of your own who maintains the system after the first release. Get advice when you do not yet know which problem is biggest or which pilot deserves production. There is a third option: someone who makes the choice, builds the first version, and then hands it over to your own team.

A clear sign that you need advice first: plenty runs in a demo, but nothing runs in daily work. A bad sign in an adviser: a report without a path to production. What I do in a one-week Scan is on my AI consultancy page. Whether you are better off hiring someone, I cover under forward deployed engineer.

When should you have AI automation built?

Have it built when the problem, the users, and the owner are clear and no existing product does the job. If you do not yet know which problem is biggest, or who maintains the system after the first release, you need a decision first.

These questions show which situation you are in:

• Where is the most manual time?
• Who uses this every week?
• Which data may the system see, and which data is off limits?
• Who maintains it after the first release?
• What is good enough to measure after four weeks?

A tool without those answers quickly becomes a demo: it works in a screen recording and still disappears from the workflow.

Look at the platform before you start, too. During one implementation the chosen platform turned out not to support everything the design needed, and a better prompt does not fix that. Since then I check four things before kickoff: the AI platform, the company’s own systems, the licences, and who has to sign off.

What does a chatbot implementation involve?

A chatbot implementation starts with a written agreement on what it has to do, and ends with a test on real questions. Write down which questions it answers, at what error rate, in which system, and who judges that. “A working chatbot” is too vague as a brief.

Three things do not show up in a demo. The chatbot has to show its sources and say what it does not know, because for users one mistake weighs more than 99 good answers. It has to be tested on questions from daily work, not only on the builder’s test set. And someone in the organisation has to own it and keep the answers current when the underlying documents change.

Most companies do not have a tool problem

Many teams think they have a tool problem. They have a decision problem. Which problem is large enough? Which use case is small enough? Who inside the organisation can carry it? What should you avoid building? Those questions come before implementation. They decide whether, a year from now, you have a system people use daily or a pilot nobody owns anymore.

So I start narrow: one workflow with one team and one owner, no more than four weeks of building. On day five we decide together whether the case is simple enough to keep building. After four weeks we measure and decide whether to continue.

Three questions for whoever you hire

Whether you hire an adviser, an agency, or a freelancer, these three questions from my memo quickly separate good answers from smooth ones:

• Did you see the material before you named a price? How many documents, how many exceptions, how many systems?
• When is the first moment we can stop without losing face? If that is only after four weeks, it is too late.
• Whose data do you test on, yours or ours? And can we run the test ourselves?

Advice without a production path is too light

My work sits between advice and building. I make a sharp choice and carry it through to a system that runs under real users. Sometimes that means saying no to a request that is framed badly. When the choice holds, I build, and I hand over before the system depends on the builder.

I build for myself and for clients. Signal Match, one of my own products, rewrites and anonymizes candidate CVs. I built Business Boosters for a client who owns the product; it finds B2B prospects for targeted outreach. Both run under real users and were built by the same rules: start narrow, measure, stop when it does not work.