Did the Training Work? Measuring AI Adoption, Not Activity

1. Activity Metrics Flatter You and Tell You Nothing

A week after the training, someone asks the obvious question: did it work. The dashboard rushes to answer. Seats activated, logins this month, messages sent, prompts run. The numbers are up and to the right, and they mean almost nothing.

Activity measures whether the tool was touched, not whether the work changed. A partner who opens the app once, types a question, and never comes back counts the same as the analyst who now drafts every memo with it. Both are a login. One is adoption and one is a rounding error, and the dashboard cannot tell them apart.

Worse, activity is the easiest number to manufacture. Mandate that everyone log in, and everyone logs in. You have bought a metric, not a result. The firm feels busy and changes nothing, which is the exact shape of a rollout that is about to fade.

The reason activity gets tracked is that it is easy to collect, not that it tells the truth. The number worth having is harder to get and the only one that matters: is a real job now done the new way, on purpose, by the people whose job it is.

2. The Off-Switch Test

There is one test that cuts through all of it, and you can run it in your head right now. If you turned the tool off tomorrow, who would be upset.

If a team would come to your office annoyed, because the screening pass they now rely on is gone and they are back to reading every CIM cold, that workflow was adopted. The tool became part of how the work gets done, and removing it is felt as a loss. That is the whole signal.

If the honest answer is that nobody would notice for a week, the tool was installed, not adopted. The license is live, the training happened, the dashboard is green, and none of it reached the work. This is the quiet failure mode behind most of the rollouts in the why AI rollouts fail guide: not a disaster, a fade nobody flags because the activity numbers kept everyone comfortable.

The off-switch test is not a metric you can chart. It is the sanity check you run before you trust any chart. Ask it per workflow, answer it honestly, and you will know more than the dashboard does.

3. What Adoption Actually Means

Adoption has a precise definition, and holding to it keeps you honest: the workflow is done the new way, unprompted, by the people who own it.

Each word in that sentence is load-bearing. The workflow, not the tool, because the unit that matters is a real job, not a piece of software. Done the new way, meaning the firm's process actually changed, not that a tool sits beside the old process unused. Unprompted, because a thing people only do when the champion is watching reverts the day the champion looks away. The people who own it, because adoption by an enthusiast who does not do the work proves nothing about the desk.

This is why a tool can have heavy usage and zero adoption. People can be prompted into activity. They cannot be prompted into a habit. The day the reminders stop is the day you find out what was real, and a habit is the only thing that survives it.

Get the definition right and the measurement gets simpler, because you stop counting events and start asking one question of each workflow: would this still happen next month if everyone stopped paying attention to it.

4. The Few Metrics Worth Tracking

You do not need a dashboard with forty tiles. You need four numbers per workflow, and most of them you collect by asking the people who do the work, not by querying a log.

None of these is hard to gather. Time-to-task is a question and a stopwatch. Penetration is a headcount. Dependence is the off-switch test. Quality is a spot-check of the output against what the firm would have produced by hand. The discipline is not measurement, it is refusing to be satisfied by the easy number when these four are the true ones.

5. Per Workflow, Not Per Firm

There is no such thing as a firm-wide adoption percentage worth quoting. Adoption lives at the workflow, and averaging across them hides exactly what you need to see.

A firm can be at ninety percent on deal screening and zero on investor reporting, and the blended number, forty-something, describes neither and misleads everyone. It tells leadership the effort is half-done when the truth is that one workflow is won and another has not started. You cannot manage the average, because the average is not a real thing anyone does.

Measuring per workflow also tells you where to push and where to celebrate. The won workflow is your proof and your recruiting story for the next one: this is what it looks like when it works, who wants it next. The stalled workflow is a specific problem with a specific cause, not a vague sense that adoption is lagging. This is the same logic that runs the whole arc in the AI change management playbook, where you win one workflow completely before widening.

So keep one line per workflow and resist the urge to roll it up. A scorecard that says screening is won, diligence is climbing, and IR has not moved is worth more than any single firm-wide figure, because every line points at an action.

6. The Quality Guardrail

Speed without quality is not adoption, it is a liability you have not noticed yet. Every adoption number needs a guardrail beside it, and the guardrail is whether the work is still as good.

It is possible to make a workflow faster and quietly worse. The screen runs in a tenth of the time and misses the thing a careful analyst would have caught. The memo drafts itself and reads slightly generic, slightly off the house view. If you only track hours saved, you will reward exactly this, and you will find out it was a bad trade at the worst possible moment, in front of an investment committee or an LP.

So pair the speed with a check. Spot-sample the AI-assisted output and compare it to the standard the firm held before. Is the screen catching what it should. Does the memo carry the firm's judgment or just its format. The goal is adoption with quality held, never adoption at the cost of the work, and a firm that prizes individual judgment will not forgive the second kind for long.

Done right, the guardrail is also a trust-builder. When the desk can see that the assisted work holds up to scrutiny, the trust gap closes on its own, and trust is what turns a trial into a habit.

7. The Adoption Curve and the Dip

Adoption does not rise in a straight line, and knowing the shape of the curve keeps you from panicking at the wrong moment.

There is almost always a dip. The first few weeks after training, the new way is slower than the old way, because people are still learning it while the old habit is still faster. The numbers look worse before they look better. This is the moment most rollouts get quietly abandoned, not because the approach was wrong but because someone read the dip as failure and pulled the energy out.

The firms that get through it expect the dip and say so out loud. They tell the desk that the new way will feel slower for a fortnight and faster after that, which turns a discouraging week into an expected stage. Then they hold steady, keep the champion close, and let the curve turn. The off-switch test is the early warning that it is turning: the day someone would be annoyed to lose the tool is the day the habit took, often before the time-savings fully show up in the numbers.

If the dip never recovers after a reasonable run, that is real information, and section nine is about reading it. But do not confuse the normal valley with a dead end. Most of the time, the curve is doing exactly what it is supposed to do.

8. A Simple Quarterly Scorecard

The whole thing fits on one page, reviewed once a quarter. Not a live dashboard anyone games, a quarterly snapshot a partner can read in two minutes.

One row per workflow. Five columns: the workflow, time-to-task now versus baseline, penetration across the people who own it, the dependence answer (would they be upset), and a quality note (held, or a flag). That is the whole instrument. A partner glances down the rows and knows immediately which workflows are won, which are climbing through the dip, and which have not moved and need a decision.

Quarterly is the right cadence on purpose. Measure adoption weekly and you are watching noise and tempting everyone to game the number. Measure it once a year and you find out too late to fix anything. A quarter is long enough for a habit to form or fail to, and short enough to act on. The same scorecard, kept over a few quarters, becomes the honest history of the program, far more useful than any single point in time.

Keep it boring and keep it consistent. The value is not in the formatting, it is in asking the same true questions of the same workflows every quarter, so the trend is visible and the stalls are obvious.

9. What to Do When a Workflow Stalls

A workflow shows up red for two quarters running: low penetration, no dependence, no time saved. The instinct is to buy a different tool. That is almost always the wrong move, because the tool is almost never the reason.

Before you change anything, find out which of three things is actually broken. Is the workflow a bad fit, meaning the job was never one AI was going to help much, in which case the honest answer is to drop it and move the energy to a workflow that pays. Is the trust not there, meaning people tried it, got burned or stayed wary, and went back, in which case you need smaller, more checkable wins to rebuild it. Or is there no owner, meaning nobody on that desk actually drove it, in which case no tool will save it.

Most stalls are the last one. A workflow with no person who wants it and pushes it does not adopt, no matter how good the setup. The fix is a champion for that specific workflow, not a procurement decision. Buying a new tool to fix a no-owner problem just gives you a second tool nobody owns.

This is exactly the running diagnosis a fractional AI Operating Partner does between quarters: read the scorecard, find the real cause behind each stall, and fix the cause rather than reaching for another license. The measurement is only useful if it leads to the right action, and the right action is usually about people, not software.

10. Where to Start

Pick the one workflow your last training touched and run the off-switch test on it this week. If you turned it off tomorrow, would the desk be upset. That single answer tells you more than the usage report has all quarter.

Then build the one-page scorecard for that workflow: time-to-task, penetration, dependence, quality. Four honest numbers beat forty vanity ones. Add a row each time you take a new workflow on, and review the page once a quarter, not once a week.

If you want the measurement run for you, alongside the work itself, that is what an AI Operating Partner does: track adoption per workflow, hold the quality guardrail, and fix the stalls at their real cause. It pairs with the training that starts the habit and the AI Readiness Sprint that picks the right first workflow, so what you measure is a workflow worth winning. Did the training work is a question you should never have to guess at.