This is the second part of our series on bringing AI into a business. The first part covered which tasks to start with and how to estimate the payoff before any development begins. Here is the next step: how to run an AI pilot project on a single task and use its results to make an honest call on whether to scale the solution, refine it or shut it down.
A pilot is often confused with a demo: the model is shown on a dozen hand-picked examples, everyone is impressed, and a month later nobody uses the tool. A pilot is an experiment that must answer a question: does the solution work on your data, in your process, with your people? Below is how to set one up so the answer is unambiguous.
Pilot vs. proof of concept vs. product
An AI proof of concept answers a technical question: can the model handle the task at all? It runs on a data export, with no integrations and no users. A pilot goes further: the solution works on a live stream of data, real employees use it, and the results are compared with how the process ran before.
A product is a working system for everyone, with support, monitoring and clear ownership. Skipping the pilot and going straight to a product means spending most of the budget before you know whether there is any benefit.
How to choose a task for an AI pilot
The task should matter enough that someone cares about the result, and be narrow enough to test in a short time. A good candidate meets most of these criteria:
- the operation is frequent and its volume can be counted;
- the task has a clear input and a clear output: email → department, document → key fields, question → answer based on internal policy;
- a model error is easy to spot and fix before it reaches a customer;
- there is historical data or examples with correct answers;
- the process has an owner who needs the result and is willing to give staff time for review.
If the task requires AI to take actions across several systems on its own (creating deals, sending emails, changing statuses), narrow it down for the pilot. First the model suggests an action and a person confirms it. Autonomy comes later, once quality has been measured.
The success metric and the "before" baseline
Evaluating an AI project starts before the project does. You need to agree on three things: what you measure, what the number is today and what value counts as success.
- Quality metric. Classification accuracy, the share of answers that needed no edits, the share of correctly extracted fields.
- Process metric. Time per operation, time to first response to a customer, operations per employee.
- Thresholds. The value at which you scale and the value below which you stop. Between them is the refinement zone.
The "before" baseline is taken from the current process: how much time it takes, how many errors people make today, what volume goes through each month. Without this measurement, any number after the pilot has nothing to be compared against.
If the success threshold isn't written down before the start, it will be adjusted to fit the result afterward.
Put the thresholds in a document and get the process owner to sign off on them. If the success threshold isn't written down up front, it is easy to bend it to the result after the pilot, and the discussion turns into impressions: some feel things got better, others feel the model makes too many mistakes.
Data and access: what to prepare
Most pilot delays have nothing to do with the model. They happen because the data sits in different places and nobody has access to it. Go through this list in advance:
- a test set: a few hundred real examples with reference answers to compare versions of the solution;
- an up-to-date knowledge base, if the model answers based on documents and policies;
- access to the systems the data comes from (email, CRM, ERP, document storage), at least read-only;
- a clear picture of whether the data contains personal information or trade secrets;
- someone on the business side who answers questions about the data and edge cases.
If the task involves reports and questions about spreadsheets, check separately how carefully the fields are filled in and whether the data matches across systems. The model won't fix inconsistencies; it will reproduce them.
Cloud API or your own model during the pilot
For a pilot, teams usually pick the fastest route to testing quality: an off-the-shelf cloud model via API. It needs no hardware of your own and lets you compare several models on the test set in a short time.
There is one limitation, and it is a serious one: data. If requests include names, phone numbers or addresses of customers or employees, data protection law applies, including GDPR rules on where personal data is processed and how it is transferred. Options for the pilot:
- anonymize the data before sending it to the model and put the details back after the response;
- use cloud models that process data in a region and under terms that meet your compliance requirements;
- deploy an open-source model (Llama, Mistral) inside your own infrastructure from the start if the data can't leave it at all.
The choice made for the pilot doesn't have to be final. What matters is that the test set and metrics don't depend on the model: then, if you switch models, you can compare quality on the same examples.
How to run the pilot, step by step
The work itself follows a clear sequence. Each step ends with a result you can check, not a report saying "work is in progress."
- A prototype on off-the-shelf models. Instructions for the model, search over the knowledge base, a simple interface or integration into a tool people already use. No custom model training at this stage.
- A run on the test set. The prototype answers hundreds of real examples, and the results are compared with the reference answers. This shows the realistic quality ceiling and the types of cases where the model fails.
- A test on live data. The solution is connected to real data, but in assist mode: the model suggests, an employee confirms or corrects. Customers never see the model's mistakes.
- Collecting errors and corrections. Every correction an employee makes is saved. This is the best material for improving the instructions and expanding the test set.
- Review with the people in the process. Short interviews with those who used the solution: where it helped, where it got in the way, how long checking took. Numbers show quality; interviews show usability.
- Comparison with the baseline. The pilot metrics are compared with the "before" measurement and with the thresholds set at the start.
The review with employees often turns out to be decisive. A model can be quite accurate, but if checking its answer takes as long as doing the work by hand, the process gains nothing.
Scale, refine or shut down
After the pilot, the decision is based on the thresholds set in advance and on user feedback. The three outcomes look like this:
- Scale. The metric is above the success threshold on both the test set and live data, employees are ready to use the tool, and the projected benefit is confirmed by actual numbers.
- Refine. The metric is in the middle zone, and the errors cluster in clear types of cases: documents missing from the knowledge base, poorly split categories, a missing data source. Refinement is time-boxed and measured again.
- Shut down. Quality is below the lower threshold and refinement can't fix the causes, or the time saved is eaten up by checking. This is a result too: the company learned it during the pilot rather than after full-scale development.
Scaling AI: what changes in production
A successful pilot is not yet a working system. During the pilot, the solution serves one group of users and runs under the team's close watch. In production, it becomes part of a process that deadlines and customers depend on.
| Pilot | Production | |
|---|---|---|
| Users | One group that knows it's an experiment | Everyone in the process |
| Integrations | Minimal, some data handled manually | Full: CRM, email, ERP, access rights |
| Quality control | The team reviews errors by hand | Ongoing metric monitoring and spot checks |
| Errors and failures | Acceptable, part of the experiment | A fallback scenario and a clear owner |
| Data and knowledge | A snapshot from the pilot | Regular knowledge base updates |
Integrations
The solution is built into the tools people already work in: the CRM, the email client, the ERP, Slack or Telegram. A separate portal you have to remember to visit empties out fast. In scope, this is regular web service development: APIs, access rights, logging, load.
Quality monitoring
Products, prices and policies change, the provider releases a new model version, and quality quietly drops. Pilot metrics must keep running in production: regular test set runs, the share of answers corrected by employees, manual spot checks. And there must be a named person who watches these numbers.
Staff training
Users need to understand what the solution can and can't do, when an answer can be trusted and when it needs checking, and where to report errors. Without this, some will blindly accept the model's answers and others will avoid the tool altogether.
If the pilot ran on a cloud model and production data can't leave the company, this is the stage where hosting gets decided: your own model inside the company's infrastructure or a cloud provider that meets your data requirements. The test set built during the pilot lets you compare the options on actual quality.
At Sphere, a digital agency in Paris, we run these projects through the same stages as any digital product: process and data analysis, design, development and testing, launch and ongoing improvement. You can see examples in Sphere's projects.
FAQ
How is an AI pilot different from a proof of concept?
A proof of concept checks whether the model can handle the task at all, on a data export and without users. A pilot tests the solution in a real process: on live data, with employees, and with metrics compared against how the process worked before.
How do you know the pilot succeeded?
By the thresholds set before the start: the quality metric on the test set and on live data, the time saved compared with the baseline, and feedback from the employees who used the solution. If the thresholds weren't written down in advance, success becomes a matter of opinion.
Do we need to train our own model for the pilot?
Usually not. Pilots run on off-the-shelf models with instructions and knowledge base search, which is enough to see what level of quality is realistic. Fine-tuning is worth discussing later, if off-the-shelf models fall short of the threshold on the test set.
What if the pilot doesn't deliver?
Figure out why: not enough data, the task was framed too broadly, or checking the answers takes as long as doing the work by hand. Sometimes it makes sense to narrow the task and try again, sometimes to close it. A closed pilot is a normal outcome: it saves the budget of full-scale development.
Can a development agency run the pilot if we don't have an in-house AI team?
Yes. On the company side, you need a process owner, access to the data and staff time to review results. The development agency handles process analysis, the prototype, integrations and measurement, and, if the pilot succeeds, the move to production.




