Pilot projects that can actually be evaluated

How often is the actual impact of a new system or service measured?

When major changes are to be evaluated, surveys are often used where employees are asked to answer questions such as:

• How well do you think the new way of working works?
• Do you find that the system helps you in your everyday life?
• How satisfied are you with the new solution?

These can be valuable questions. User experience affects both the adoption and the ability to realize the benefits of a new solution.

But the answers do not necessarily show whether the investment has had the effect that justified it in the first place.

If the goal was lower costs, shorter lead times or less manual work, that also needs to be measured.

What does a successful pilot really mean?

An evaluation that is mainly based on experiences is influenced by many factors.

• How were the questions asked? 
• Was the survey anonymous? 
• What is the company culture like? 
• How much training did users receive? 
• Are people generally positive about the change – or do they still miss the old system?

The results can provide important information about how the solution is received, but it is difficult to use as an answer to the fundamental business question:

Did the investment have the effect we wanted to achieve?

Even though it often involves large investments, it is surprisingly common that there is no clear starting point to compare with.

If you don’t know what the process cost, how long it took, and how much manual work it required before implementation, it also becomes difficult to determine what has actually been improved.

If investments were evaluated in the same way

In the article “Why do so many IT savings start with a large IT cost?” I compared large IT investments to investments in the stock market.

Imagine if an investment department invested large amounts in risky securities and then evaluated the results by asking employees how they felt the investments had performed.

I’m pretty sure that wouldn’t be accepted.

One would of course measure the actual economic outcome.

But when large IT investments are to be evaluated, it sometimes seems sufficient to state that the solution has been implemented, that the users have been trained and that a majority have responded fairly positively to a survey.

Then you are surprised that IT costs continue to increase.

Start with a starting point

At Sinos, we want a pilot to be able to be evaluated with concrete results.

Therefore, we start by mapping out what the current process looks like before the pilot. Depending on the area of use, this could include:

• direct external costs per case
• internal working hours
• total lead time
• the number of manual steps
• the quality and completeness of the collected data

This gives us a starting point – a baseline – against which the results from the pilot can be compared.

When a case is then handled through Sinos, we can measure the outcome against the previous process. This way, it is possible to follow how costs, working hours and lead times actually change.

User experience should still be included in the evaluation, but it complements the actual impact measurement rather than replacing it.

A pilot should give an answer

A good pilot should not only show that a technical solution works. It should provide sufficient information to make a decision.

• Does the service create measurable savings? Does the lead time get shorter? 
• Does it reduce manual work? Does the result maintain a sufficiently high quality? 
• Are there parts of the process that need to be changed?

The answer doesn’t always have to be yes.

The important thing is that the answer can be trusted.

This is also why we can say that the operational savings with Sinos can start already with the first case. 

We don’t have to wait for a future transformation program to look for the effect.

We can measure it directly.

Author: Peter Ehrling, Sinos Solutions, 2026-09-26