AI-assisted engineering is easy to measure badly and the usual numbers appear first: active users, accepted completions, generated lines of code, tokens consumed, agent sessions and pull requests opened. They prove that something happened, but not that it helped.
For platform teams, the useful question is:
Is AI-assisted work producing worthwhile engineering outcomes at a sensible cost, with acceptable risk and review effort?
That is the measurement problem I would spend time on.
Measure engineering workflows, not individual tools
I would not design an engineering dashboard around one AI product.
A team might use GitHub Copilot in the IDE, Copilot Chat, agent mode, custom agent skills, MCP tools, APIM-fronted model APIs, scripts and internal automation. Those interfaces will change. The underlying engineering work is more stable.
Start with the workflow:
- Terraform provider upgrade
- dependency update
- test generation
- pull request review
- architecture review
- cost optimisation assessment
- documentation update
- migration planning
- incident summary
- platform support response
Token usage for a Terraform provider upgrade means something different from token usage for explaining a YAML file. A generated test that is accepted and maintained is different again from one that has to be rewritten before it can merge.
Where the telemetry allows it, I would break measurements down by team, product, repository, workflow, model, task type, environment and tool or gateway path.
Without those dimensions, you end up with a generic usage report that looks good on a large screen and answers very little.
A measurement model to get going
I would start with a small model that helps the platform team spot patterns and make decisions. It does not need to answer every possible question about AI-assisted delivery.
| Area | What to measure | What it can tell you | What to avoid |
|---|---|---|---|
| Consumption | Token and model usage by team, product, repository and workflow | Where activity and cost are coming from | Assuming low usage is good or high usage is bad |
| Workflow success | Completed, failed, abandoned and retried runs | Which workflows are dependable enough to keep using | Treating every completed run as successful |
| Reviewability | Pull request acceptance, review cycles, rework and diff size | Whether AI reduces effort or moves it into review | Celebrating generated pull requests without measuring review burden |
| Guardrails | Scope creep, missing tests, unsafe tool calls and policy failures | Where skills, controls or repository standards need work | Treating every failure as a user problem |
| Context reuse | Skill usage, repository instructions, MCP tools and reference files | Whether reusable context improves consistency | Giving agents more context by default |
| Cost and value | Cost per accepted change or useful workflow | Whether spending maps to an engineering outcome | Pretending the calculation is more precise than it is |
| Developer confidence | Trust, friction and perceived review burden | Whether engineers will keep using the workflow | Relying on telemetry alone |
A smaller model that leads to changes is more useful than a large dashboard nobody trusts.
Token usage needs an outcome
Token and model usage should be visible – the cost is becoming too material to ignore, particularly where access is shared across teams.
For platform teams, I would break consumption down by team, product, repository, workflow, model, task type and environment where possible. The same principle applies to shared AI APIs and developer tooling.
When model access goes through APIM, useful dimensions can include:
- consumer identity
- product or subscription context
- model name
- prompt and completion tokens
- rate-limit events
- failures
- backend route
OpenTelemetry is also developing a more consistent shape for generative AI telemetry, including token metrics and spans for model requests and tool calls. That should make it easier to compare workflows across tools and providers without building a separate observability model for each one.
Still, token totals are only consumption data.
A high-token run may be good value if it produces a safe migration plan or a well-scoped pull request. A cheap interaction is still waste if nobody uses the output.
Token usage becomes useful when you can connect it to the task, the result and what happened next.
Review burden is part of the cost
A generated pull request is not valuable because it exists. Someone still has to understand it, validate it and decide whether it should merge.
If an agent creates a large diff, mixes unrelated changes, hides its assumptions or misses obvious tests, the effort has not disappeared. It has moved from writing code to reviewing code.
I would measure:
- pull request acceptance
- number of review cycles
- reviewer comments
- rework after review
- diff size relative to the task
- whether the change stayed within scope
- whether tests were included
- whether validation evidence was provided
These measures need judgement. A large refactoring will naturally produce a large diff, and a review comment does not automatically mean the original work was poor.
Repeated heavy rework is still a useful signal. It may mean the skill is too broad, the repository instructions are weak, important context is missing or the task is not suitable for automation yet.
The point is to improve the workflow, not score the engineer who used it.
Define success before measuring agents
“Agent success rate” sounds useful until you try to define it.
A Terraform provider upgrade, documentation update, incident summary and architecture review do not share the same definition of success. Completing a run only tells you that the agent stopped working.
For a provider upgrade, I might define success as:
- identifying the current and target versions
- reviewing the changelog or upgrade guide
- calling out breaking changes
- updating the code without unrelated refactoring
- running the relevant tests or validation commands
- including risks and follow-up work in the pull request summary
Documentation needs different criteria. The output should follow the repository structure, avoid unsupported claims and link to the correct source material.
For an incident summary, I would care about an accurate timeline, a clear separation between observations and actions, and assumptions being marked as assumptions.
Success means completing the task in a form that is useful to the next person in the engineering workflow.
Treat guardrail failures as platform feedback
Guardrails should block unsafe behaviour where necessary, but they should also tell the platform team where the system needs work.
Useful signals might include:
- unrelated refactoring
- missing tests
- changes outside the requested scope
- unsafe tool calls
- model access that bypasses the approved gateway
- use of an unapproved model
- missing pull request evidence
- secrets or sensitive data exposure
- unsupported claims in documentation
Some of these should block the workflow. Others are better handled as warnings. Either way, the failure needs to be visible and attributable to the relevant workflow.
When the same guardrail keeps failing, I would look for a platform fix. The repository instructions may need updating, the skill may need a better example, or a validation script may need to catch the problem earlier.
It is also worth checking whether the approved route is unnecessarily difficult. A control that engineers regularly work around may be a poor control, a poor developer experience or both.
Measure whether reusable context works
AI-assisted engineering depends heavily on the context available to the model or agent:
- repository instructions
- agent skills
- architecture decision records
- coding standards
- MCP tools
- source files
- examples
- validation scripts
I would measure whether those sources are used and whether they improve the result.
For example:
- Do pull requests created with a specific skill need fewer review cycles?
- Do repository instructions reduce scope creep?
- Does an MCP integration improve factual accuracy?
- Do validation scripts catch missing tests before review?
- Do reference implementations reduce repeated prompting?
More context is not automatically better. Giving an agent access to every document, repository and API increases cost and makes it harder to understand which sources influenced the result.
The useful target is the smallest authoritative context set that reliably supports the task.
Keep developer confidence in the model
Telemetry will not tell you whether a workflow is frustrating to use.
An output may be technically correct but difficult to trust. A generated pull request may take longer to review than a manual change would have taken to write. An agent may need several rounds of prompting before it understands the task.
I would keep the qualitative measurement lightweight:
- Did this workflow reduce effort?
- Was the output reviewable?
- Would you use the workflow again?
- Where did the agent waste time?
- What context was missing?
- What would make the workflow safer or easier?
This should be a small pulse check, not another survey that disappears into a spreadsheet. Its job is to capture friction that the telemetry cannot see.
Cost per accepted outcome is useful, within limits
One metric worth exploring is cost per accepted engineering outcome.
Depending on the workflow, that could mean:
- cost per accepted pull request
- cost per completed provider upgrade
- cost per reviewed architecture assessment
- cost per incident summary accepted without correction
- cost per generated test retained after review
This is more useful than total token spend because it connects consumption to something the engineering team kept or acted on.
It is not a universal productivity metric. Engineering tasks vary too much in size and difficulty. A £11 architecture review may be excellent value, while a £2 documentation update may still be useless.
I would compare similar workflows over time rather than rank every engineering activity against one another. Reducing everything to one efficiency score will reward cheap, easy work rather than useful work.
Do not make activity the headline
The main risk is building a vanity dashboard full of numbers that do not change any decisions.
I would not lead with:
- total prompts
- total tokens
- generated lines of code
- active users
- agent sessions
- AI-created pull requests
- completion acceptance rates
Those are supporting signals. The headline questions should be closer to:
- Which workflows are improving?
- Which workflows cost a lot but produce little usable output?
- Where is review burden increasing?
- Which skills consistently produce accepted changes?
- Where are guardrails failing?
- Which teams need better defaults or support?
- Where should the platform invest next?
A useful dashboard should lead to an action: improve a skill, tighten a guardrail, update repository instructions, change a default, adjust a quota or support a team.
Otherwise, it is decoration.
What I would build first
For an initial version, I would start with five views.
| View | Purpose |
| Usage by team, repository, workflow and model | Show where activity and cost are coming from |
| Repeated workflow success | Identify which workflows complete reliably |
| Pull request review and rework | Show whether AI reduces effort or moves it into review |
| Guardrail failures | Find weaknesses in skills, policies, tools and repository standards |
| Developer confidence and friction | Capture trust and usability issues that telemetry misses |
I would then choose two or three repeatable workflows and define success properly for each one.
A Terraform provider upgrade, test generation workflow or documentation update is easier to evaluate than “general Copilot usage” because the expected result is clearer.
Once the data for those workflows is trustworthy and leads to useful changes, expand the model.
Known limitations
Attribution will remain difficult. An engineer may start with an agent, make manual changes, use another tool and then open a pull request. Assigning that outcome to one interaction is rarely honest.
Quality is also subjective. Review comments and rework are useful signals, but neither automatically proves that the original output was bad.
Telemetry coverage will vary between IDE tools, hosted agents, custom scripts and shared model APIs. You will probably have gaps.
Teams may also optimise for whatever becomes the target. If pull request acceptance is treated as the main success measure, people may choose smaller and easier tasks instead of more valuable ones.
Use several signals together and keep the workflow visible underneath them. No single number should become an “AI engineering productivity score”.
Frequently asked questions
What is the best metric for AI-assisted software development?
There is no single best metric. I would combine workflow success, review effort, cost, guardrail failures and developer confidence. Usage data supports the analysis, but it is not evidence of value on its own.
Should teams measure AI-generated lines of code?
It can show activity, but it says little about quality. A small change that stays within scope and passes review is usually more useful than a large generated diff that needs substantial rework.
How should token usage be measured?
Measure it by team, repository, workflow, model and task type where possible. Then connect the consumption to an outcome such as an accepted pull request, a completed migration assessment or a validated documentation change.
How do you measure whether a coding agent succeeded?
Define success for the specific workflow before running it. For a provider upgrade, that may include identifying breaking changes, making only relevant updates, running validation and providing review evidence. Other workflows need their own criteria.
The agent completing a run is not the same as the engineering task being completed successfully.
Measure what helps you improve the system
Platform teams do not need a perfect measurement framework on day one. They need enough evidence to see which workflows are worth keeping, where review effort has moved and what needs to change in the platform.
Creating more AI activity is easy. The harder part is showing that the work leads to accepted changes, safer delivery and less wasted review effort.
Measure enough to improve the next run. Anything beyond that needs to earn its place on the dashboard.