A seedling in a laboratory beaker on a dark table, with blue light glowing inside the beaker

AI Pilot Project: From Idea to a Working System

This is the second part of our series on bringing AI into a business. The first part covered which tasks to start with and how to estimate the payoff before any development begins. Here is the next step: how to run an AI pilot project on a single task and use its results to make an honest call on whether to scale the solution, refine it or shut it down.

A pilot is often confused with a demo: the model is shown on a dozen hand-picked examples, everyone is impressed, and a month later nobody uses the tool. A pilot is an experiment that must answer a question: does the solution work on your data, in your process, with your people? Below is how to set one up so the answer is unambiguous.

Pilot vs. proof of concept vs. product

An AI proof of concept answers a technical question: can the model handle the task at all? It runs on a data export, with no integrations and no users. A pilot goes further: the solution works on a live stream of data, real employees use it, and the results are compared with how the process ran before.

A product is a working system for everyone, with support, monitoring and clear ownership. Skipping the pilot and going straight to a product means spending most of the budget before you know whether there is any benefit.

How to choose a task for an AI pilot

The task should matter enough that someone cares about the result, and be narrow enough to test in a short time. A good candidate meets most of these criteria:

If the task requires AI to take actions across several systems on its own (creating deals, sending emails, changing statuses), narrow it down for the pilot. First the model suggests an action and a person confirms it. Autonomy comes later, once quality has been measured.

The success metric and the "before" baseline

Evaluating an AI project starts before the project does. You need to agree on three things: what you measure, what the number is today and what value counts as success.

The "before" baseline is taken from the current process: how much time it takes, how many errors people make today, what volume goes through each month. Without this measurement, any number after the pilot has nothing to be compared against.

If the success threshold isn't written down before the start, it will be adjusted to fit the result afterward.

Put the thresholds in a document and get the process owner to sign off on them. If the success threshold isn't written down up front, it is easy to bend it to the result after the pilot, and the discussion turns into impressions: some feel things got better, others feel the model makes too many mistakes.

Data and access: what to prepare

Most pilot delays have nothing to do with the model. They happen because the data sits in different places and nobody has access to it. Go through this list in advance:

If the task involves reports and questions about spreadsheets, check separately how carefully the fields are filled in and whether the data matches across systems. The model won't fix inconsistencies; it will reproduce them.

Cloud API or your own model during the pilot

For a pilot, teams usually pick the fastest route to testing quality: an off-the-shelf cloud model via API. It needs no hardware of your own and lets you compare several models on the test set in a short time.

There is one limitation, and it is a serious one: data. If requests include names, phone numbers or addresses of customers or employees, data protection law applies, including GDPR rules on where personal data is processed and how it is transferred. Options for the pilot:

The choice made for the pilot doesn't have to be final. What matters is that the test set and metrics don't depend on the model: then, if you switch models, you can compare quality on the same examples.

How to run the pilot, step by step

The work itself follows a clear sequence. Each step ends with a result you can check, not a report saying "work is in progress."

  1. A prototype on off-the-shelf models. Instructions for the model, search over the knowledge base, a simple interface or integration into a tool people already use. No custom model training at this stage.
  2. A run on the test set. The prototype answers hundreds of real examples, and the results are compared with the reference answers. This shows the realistic quality ceiling and the types of cases where the model fails.
  3. A test on live data. The solution is connected to real data, but in assist mode: the model suggests, an employee confirms or corrects. Customers never see the model's mistakes.
  4. Collecting errors and corrections. Every correction an employee makes is saved. This is the best material for improving the instructions and expanding the test set.
  5. Review with the people in the process. Short interviews with those who used the solution: where it helped, where it got in the way, how long checking took. Numbers show quality; interviews show usability.
  6. Comparison with the baseline. The pilot metrics are compared with the "before" measurement and with the thresholds set at the start.

The review with employees often turns out to be decisive. A model can be quite accurate, but if checking its answer takes as long as doing the work by hand, the process gains nothing.

Scale, refine or shut down

After the pilot, the decision is based on the thresholds set in advance and on user feedback. The three outcomes look like this:

Scaling AI: what changes in production

A successful pilot is not yet a working system. During the pilot, the solution serves one group of users and runs under the team's close watch. In production, it becomes part of a process that deadlines and customers depend on.

PilotProduction
UsersOne group that knows it's an experimentEveryone in the process
IntegrationsMinimal, some data handled manuallyFull: CRM, email, ERP, access rights
Quality controlThe team reviews errors by handOngoing metric monitoring and spot checks
Errors and failuresAcceptable, part of the experimentA fallback scenario and a clear owner
Data and knowledgeA snapshot from the pilotRegular knowledge base updates

Integrations

The solution is built into the tools people already work in: the CRM, the email client, the ERP, Slack or Telegram. A separate portal you have to remember to visit empties out fast. In scope, this is regular web service development: APIs, access rights, logging, load.

Quality monitoring

Products, prices and policies change, the provider releases a new model version, and quality quietly drops. Pilot metrics must keep running in production: regular test set runs, the share of answers corrected by employees, manual spot checks. And there must be a named person who watches these numbers.

Staff training

Users need to understand what the solution can and can't do, when an answer can be trusted and when it needs checking, and where to report errors. Without this, some will blindly accept the model's answers and others will avoid the tool altogether.

If the pilot ran on a cloud model and production data can't leave the company, this is the stage where hosting gets decided: your own model inside the company's infrastructure or a cloud provider that meets your data requirements. The test set built during the pilot lets you compare the options on actual quality.

At Sphere, a digital agency in Paris, we run these projects through the same stages as any digital product: process and data analysis, design, development and testing, launch and ongoing improvement. You can see examples in Sphere's projects.

FAQ

How is an AI pilot different from a proof of concept?

A proof of concept checks whether the model can handle the task at all, on a data export and without users. A pilot tests the solution in a real process: on live data, with employees, and with metrics compared against how the process worked before.

How do you know the pilot succeeded?

By the thresholds set before the start: the quality metric on the test set and on live data, the time saved compared with the baseline, and feedback from the employees who used the solution. If the thresholds weren't written down in advance, success becomes a matter of opinion.

Do we need to train our own model for the pilot?

Usually not. Pilots run on off-the-shelf models with instructions and knowledge base search, which is enough to see what level of quality is realistic. Fine-tuning is worth discussing later, if off-the-shelf models fall short of the threshold on the test set.

What if the pilot doesn't deliver?

Figure out why: not enough data, the task was framed too broadly, or checking the answers takes as long as doing the work by hand. Sometimes it makes sense to narrow the task and try again, sometimes to close it. A closed pilot is a normal outcome: it saves the budget of full-scale development.

Can a development agency run the pilot if we don't have an in-house AI team?

Yes. On the company side, you need a process owner, access to the data and staff time to review results. The development agency handles process analysis, the prototype, integrations and measurement, and, if the pilot succeeds, the move to production.

Let’s discuss your project
Related reading