Everyone Makes a Study. Nobody Opens the Tool.
I once sat in on a review where a client came across a dashboard that showed a 96% course completion rate for an AI release. Everyone nodded. The numbers were raw. Then someone from operations asked, almost as an aside, if anyone on the ground was using this tool. The room fell silent, because no one had guessed that, and the honest answer was no. That gap, between the course being completed and the tools being used, is something that no one is putting on the dashboard. And it’s the only thing the project ever was.
Ending Measures of Existence, Not Change
A gym membership is a useful way to think about this. A gym can tell you to scan your card 11 times in the last month. It won’t tell you if you are strong. Logging is real, easy to calculate, and almost entirely tied to what you actually signed up for. Completion of the course is to scan the card. Passed questions are slightly better. And it doesn’t say that anything has changed where the work is done.
There is a useful framework for this from a training researcher named Donald Kirkpatrick, who has laid out four levels of evaluation. Level 1 is whether people liked the training. The 2nd level is whether they have read the information, the questions I can check. Level 3 (that the behavior has actually changed back to work) is the important one here, and the one that almost no one can measure. There’s a level 4 above it as well, the actual result the business was after, and the reason for the level 3 behavior is what I see as the best indicator of those results: fewer callbacks, less rework, fewer direct errors that the tool was bought to stop. When behavior changes at work, those bottom numbers follow. You don’t need an academic vocabulary to master this. You just need to know that the end points and questions live on the easy levels, and the thing you care about lives on a level where it’s hard to see.
What Adoption Really Looks Like
Changing behavior leaves clues if you know where to look. Is the tool being opened at the intended workflow step, weeks after training ends, with no one to remind people? Do the employees reveal themselves, asking about serious cases, telling you where they are lacking? Those questions only come from people who use something. When the people who trained you start telling you where the tool falls, it might sound like a complaint, but what you’re really hearing are people who have folded something in their day to find its edges. That is the sound of discovery.
Use that lasts past the first two weeks is a very hopeful signal. Almost anything gets a bump right after training. The question is whether the curve flattens out to real value or declines back to zero when innovation and manager attention continue.
Instrument It Without Becoming Surveillance
The wrong way to measure adoption is to start hacking all the keys and track individual employees against each other. When people feel watched, you’ve poisoned the thing you’re trying to measure, and you’ve got yourself a place full of people who will only use the tool if they think someone is watching.
The combined signals are enough, and many of them can be calculated without removing the other person. How often the tool is opened in the relevant workflow step, which is counted by group instead of name. Whether support tickets or callbacks the tool had to terminate actually trended down in the quarter. Short, honest monthly check-ins that ask what’s working and what’s not. A silent word with a few leading lines about whether they see behavior in the wild. All of these are read at the group level, not the individual level. You’re trying to learn if a job has changed, not build a case file on any one person. The easier your hand is, the more you learn.
Set These Expectations Before Starting a Job
This is the part that should happen before the wedding, not after. When a client asks me to build AI training, the finish is usually the number they have in mind as the finish line. So I tell them early on that completion isn’t about delivery, it’s about behavior change, and that we’ll be looking at usage down 60 and 90 days, not just whether everyone hit the deadline. Some habits come right away and others take longer, so I consider those checkpoints to be learning about acquisition trends (or otherwise) rather than final grades.
That discussion changes the entire nature of the project, because a course designed for completion and a course designed for use are not the same course. A few buyers back off at first, because the number they came to expect to report is higher. But most, once they have it, would rather know the truth than carry a 96% completion rate hiding a tool that no one has caught. They wanted a used tool. They had never been told that elimination and consumption were different things to choose between in the beginning. So I tell them, and we measure right from day one.
