The Productivity Metric AI Engineering Is Missing
We're getting very good at measuring how much faster AI can help engineers produce software.
Time to completion.
Pull requests.
Features shipped.
Code generated.
Development cycles.
Engineering hours saved.
Those measurements matter.
If an engineer can complete in an hour what previously required a day, that's real productivity.
If a small team can deliver what once required a much larger organization, that's real leverage.
If an idea can move from concept to production faster because AI eliminates hours of repetitive implementation work, that's valuable.
But I think there's a measurement missing from much of the AI engineering productivity conversation.
What does all of that additional output cost us after it reaches production?
Because engineering productivity doesn't stop when the pull request gets merged.
We're Measuring the Beginning of the Equation
Imagine an engineer is working on a feature that historically takes three days.
AI-assisted development reduces the implementation time to one day.
That's easy to measure.
Two days saved.
Now imagine the organization applies that productivity improvement across fifty engineers.
Suddenly the team has dramatically more development capacity.
That capacity can become:
More features.
More experiments.
More refactoring.
More infrastructure changes.
More dependency upgrades.
More deployments.
More software.
From a development perspective, the productivity improvement is obvious.
But every one of those changes eventually enters a production environment.
And production introduces another set of costs.
Monitoring.
Validation.
Investigation.
Incident response.
Performance analysis.
Dependency management.
Operational coordination.
Human attention.
Those costs don't necessarily scale down simply because development became faster.
Some may scale up.
Faster Development Creates More Opportunities for Change
One of AI's most important effects on engineering may not be that individual tasks become faster.
It may be that the economics of making changes become different.
Things that previously weren't worth engineering time suddenly become practical.
A small optimization that might have sat in the backlog for six months can be implemented.
An internal tool can be built.
A service can be refactored.
A new integration can be tested.
A dependency can be upgraded.
A prototype can become production software.
That's good.
Engineering organizations should be able to do more with their resources.
But every additional change creates another opportunity for production behavior to change too.
The code may be generated faster.
The production environment doesn't become simpler as a result.
The Hidden Operational Tax
Consider a hypothetical AI-assisted change.
The implementation takes twenty minutes.
Tests pass.
The pull request is approved.
The deployment succeeds.
Then latency begins increasing.
Nothing catastrophically fails.
There isn't necessarily an outage.
But something is different.
An engineer investigates.
They open the observability platform.
They examine metrics.
They inspect logs.
They review traces.
They compare behavior before and after the deployment.
They identify the affected dependency.
Another engineer gets involved because that dependency belongs to another team.
Eventually the team understands what happened.
Now ask a different productivity question.
How long did the change actually take?
Twenty minutes?
Or twenty minutes plus the operational work required to understand its consequences?
The answer matters.
Because if we're going to use AI to measure development productivity in minutes saved, we should also understand the engineering minutes being consumed downstream.
Code Generation Is Not System Understanding
AI is remarkably good at helping engineers produce artifacts.
Code.
Tests.
Documentation.
Queries.
Configuration.
Infrastructure definitions.
Even architectural suggestions.
But producing an artifact and understanding the behavior of the resulting system are different problems.
A code-generation model may understand the function it's producing.
It may understand the repository surrounding it.
It may even understand the architectural patterns represented in the codebase.
Production introduces another reality.
Traffic.
State.
Dependencies.
Infrastructure.
Concurrency.
Historical behavior.
External services.
Customer usage.
Failures.
Retries.
Timing.
A change that appears completely reasonable in development can behave differently once it interacts with that environment.
That's not an AI failure.
That's software engineering.
We've dealt with this problem for decades.
AI simply increases the speed at which we can introduce new variables into the equation.
More Output Changes the Operational Denominator
There's another way to think about this.
Suppose an organization has 50 engineers.
Before widespread AI adoption, those engineers produce a certain amount of software change each month.
The organization has built an operational capacity around that rate of change.
Its SRE practices.
Its incident response processes.
Its observability stack.
Its senior engineering expertise.
Its institutional knowledge.
Now AI substantially increases development output without substantially increasing engineering headcount.
The organization hasn't just improved productivity.
It has changed a ratio:
Software change per engineer has increased.
That means operational understanding per engineer must somehow increase too.
Otherwise the organization creates an imbalance.
More production change enters the system.
But roughly the same number of humans remain responsible for understanding what that change does.
That's where the productivity conversation becomes much more interesting.
What Should We Actually Measure?
I don't think the answer is to stop measuring development velocity.
And it certainly isn't to slow down AI adoption.
The leverage is too significant.
Instead, we need a more complete definition of engineering productivity.
We should ask questions like:
How quickly can the organization ship useful software?
How reliably does that software behave?
How much engineering time is spent investigating unexpected production behavior?
How quickly can engineers move from detection to a credible operational hypothesis?
How many people need to become involved before an incident is understood?
How much institutional knowledge is required to interpret production signals?
How frequently does increased development velocity create downstream operational work?
Those measurements connect development productivity with operational reality.
Because the objective isn't simply to produce more software.
It's to produce more valuable software without proportionally increasing the cost of understanding and operating it.
This Becomes More Important With Smaller Teams
There's an uncomfortable possibility embedded in the AI productivity story.
If AI allows organizations to produce more software with fewer people, some companies will eventually choose to operate with smaller engineering organizations.
That creates enormous economic leverage.
It also makes operational knowledge more concentrated.
A senior engineer who understands why a particular service behaves strangely under load becomes more important when fewer people possess that context.
An SRE who recognizes a recurring failure pattern becomes more important.
The engineer who remembers why an architectural decision was made three years ago becomes more important.
Organizations may produce more software while having fewer humans available to interpret the resulting systems.
That means the operational layer has to become more intelligent too.
Not because humans are unnecessary.
Because their judgment becomes more valuable.
The Next Productivity Tool May Not Write Code
This is where I think the AI engineering conversation becomes particularly interesting.
The first generation of AI development tools largely focused on production:
Help engineers write.
Help engineers build.
Help engineers test.
Help engineers ship.
The next productivity opportunity may exist after the software is running.
Help engineers understand.
Help engineers investigate.
Help engineers prioritize.
Help engineers connect operational evidence.
Help engineers decide where to look next.
That's a different category of productivity.
It's not about producing more engineering output.
It's about increasing the effectiveness of engineering attention.
And attention may become one of the scarcest resources in an AI-accelerated engineering organization.
From Telemetry to Operational Intelligence
This is the problem space we're exploring with Signal Audit.
Engineering organizations already have observability.
They already collect metrics, logs, traces, alerts, and deployment information.
The question isn't whether they need more telemetry.
It's whether the operational understanding derived from that telemetry can scale alongside the rate at which software changes.
Signal Audit is designed to sit in that gap.
An observability platform detects something.
Signal Audit interprets the available operational context.
The engineer receives a structured finding:
What happened.
Why it may matter.
What evidence supports the interpretation.
Where investigation should begin.
The engineer still owns the judgment.
But the organization is attempting to increase the amount of useful operational understanding available per engineering cycle.
That's the productivity metric I'm increasingly interested in.
Productivity Should Survive Production
AI will almost certainly allow engineering organizations to build more software.
That's not hypothetical anymore.
The more important question is what happens to the systems those organizations are building.
If development velocity doubles but incident investigation becomes significantly more expensive, the productivity gain isn't as large as it initially appears.
If teams ship more features but senior engineers spend increasing amounts of time reconstructing production context, some of the leverage has simply moved from one part of the engineering lifecycle to another.
And if fewer engineers become responsible for increasingly dynamic production environments, operational understanding becomes an even more important form of leverage.
The goal shouldn't be to slow development down.
It should be to make sure understanding speeds up with it.
Because the productivity metric that ultimately matters isn't how quickly software reaches production.
It's how effectively the engineering organization can build, understand, and operate what reaches production.
Measure what happens after the deployment.
The Signal Audit Vanguard Program is a founder-led, 30-day evaluation designed to understand how operational intelligence performs inside a real engineering environment.
We're working with three engineering organizations using their existing production telemetry to evaluate whether better operational context can help engineering attention scale alongside development velocity.
