Your Most Experienced Engineers Shouldn't Be Human Query Engines
When something unusual happens in production, there's often one person everyone wants in the room.
The engineer who has been there before.
They know the architecture.
They remember why a service behaves strangely under certain conditions.
They know which alerts are usually noise.
They know which dependency looks harmless until it isn't.
They remember the deployment six months ago that produced almost exactly the same behavior.
That kind of experience is extraordinarily valuable.
But look closely at what we often ask these engineers to do during an investigation.
Open the dashboard.
Check the logs.
Look at the traces.
Find the deployment.
Compare timestamps.
Determine ownership.
Look at the dependency.
Remember whether this happened before.
Explain the system to everyone else.
In other words, some of the most experienced—and expensive—people in the engineering organization become human query engines.
That's not the best use of expertise.
THE ENGINEER EVERYONE CALLS
Most engineering organizations have someone like this.
Sometimes it's an SRE.
Sometimes it's a staff engineer.
Sometimes it's the engineer who originally built the service.
Sometimes it's simply the person who has survived enough production incidents to recognize patterns other people haven't encountered yet.
When an incident becomes difficult, that person gets pulled in.
Not necessarily because they're the only person capable of fixing the problem.
Often they're valuable because they can reduce uncertainty faster.
They know where to look.
They know what probably doesn't matter.
They can connect a current signal to historical behavior.
They can distinguish something unusual from something genuinely dangerous.
That's operational judgment.
And it's one of the most valuable resources an engineering organization has.
Which makes the way we spend it worth examining.
EXPERTISE IS OFTEN USED FOR INFORMATION RETRIEVAL
Consider the first ten minutes of a typical investigation.
An alert arrives.
Someone opens the observability platform.
They inspect the metric.
They look at related dashboards.
They search logs.
They inspect traces.
They check recent deployments.
They determine which service owns the dependency.
They ask whether anyone has seen the behavior before.
Then the experienced engineer arrives and starts doing many of the same things—but faster.
That's the part I find interesting.
The organization isn't initially using that engineer for judgment.
It's using them for retrieval.
Find the relevant information.
Remember the relevant history.
Connect the relevant systems.
Reconstruct what happened.
Only after that work is complete does the engineer begin applying the expertise that made them valuable in the first place.
We've accepted that workflow because historically there wasn't a better option.
But increasingly, I think there should be.
INFORMATION AND JUDGMENT ARE DIFFERENT KINDS OF WORK
There is a useful distinction here.
Some operational work is primarily informational.
What changed?
When did it change?
Which service is affected?
Who owns it?
What threshold was crossed?
What related signals appeared around the same time?
Was there a recent deployment?
Which dependencies are involved?
Other operational work requires judgment.
Does this evidence actually support the hypothesis?
How much customer risk are we willing to accept?
Should we roll back?
Should another team be involved?
Is the apparent correlation meaningful or coincidental?
Does this behavior violate what we know about the architecture?
What is the safest next action?
Machines are becoming increasingly capable of helping with the first category.
Humans remain extraordinarily valuable in the second.
The mistake would be treating both categories as though they require the same allocation of human attention.
THE COST OF RECONSTRUCTING CONTEXT
Context reconstruction feels inexpensive because it happens in small increments.
Thirty seconds to check a dashboard.
Two minutes to search logs.
Another minute to find the deployment.
A Slack message asking who owns a service.
A few minutes comparing timestamps.
Someone joins the incident and needs the situation explained again.
Individually, none of these activities seems particularly expensive.
Collectively, they become a significant tax on engineering attention.
And the tax compounds because production investigations rarely involve one person.
If four engineers independently reconstruct overlapping pieces of the same context, the organization pays for that work four times.
If the senior engineer joins twenty minutes later and repeats the process, it pays again.
The issue isn't that engineers shouldn't inspect telemetry.
They absolutely should.
The issue is whether assembling the initial operational picture should continue to require as much manual effort as it does today.
INSTITUTIONAL KNOWLEDGE DOESN'T SCALE EASILY
There's another reason this matters.
Experienced engineers carry context that tooling often doesn't.
They know the history of the system.
Why a particular architectural compromise exists.
Why an alert threshold is configured strangely.
Why a service that appears unrelated is actually important.
Why a behavior that looks dangerous is normal every morning at 9:00.
That knowledge accumulates over years.
It's difficult to document completely.
It's difficult to transfer.
And when an experienced engineer leaves, some of it leaves with them.
This becomes especially important if AI enables organizations to operate with smaller engineering teams.
If fewer engineers become responsible for more software, institutional knowledge becomes more concentrated.
The organization can't afford to use that scarce knowledge inefficiently.
Its most experienced engineers should be spending their attention on the questions only experience can answer.
Not repeatedly collecting information the systems themselves already contain.
THE GOAL ISN'T TO AUTOMATE EXPERTISE
This is where discussions about AI in operations can go wrong.
The objective shouldn't be:
"Let's automate what the senior engineer does."
A senior engineer's value isn't a sequence of deterministic steps that can simply be reproduced.
Experience includes intuition.
Pattern recognition.
Risk assessment.
Business context.
Architectural knowledge.
Skepticism.
Knowing when the available evidence doesn't make sense.
Those are exactly the capabilities we should preserve.
The better objective is:
Remove unnecessary work from the path to expert judgment.
Give the engineer the relevant evidence.
Show the relationships.
Identify the likely investigative direction.
Explain why that direction appears meaningful.
Then let the engineer do what they're actually good at:
Evaluate it.
FROM QUERY ENGINE TO DECISION MAKER
Imagine the same senior engineer joining an incident under a different workflow.
Instead of:
"Latency is high. We're still trying to figure out what's going on."
they receive:
The affected service.
The triggering condition.
The relevant timing.
Recent changes.
Related dependency behavior.
A proposed operational interpretation.
The evidence supporting it.
A recommended place to begin validating the hypothesis.
Now the engineer enters the investigation differently.
They don't begin by reconstructing the situation.
They begin by interrogating it.
Is this interpretation credible?
What is missing?
Does this match how the system actually behaves?
Is the proposed dependency relationship meaningful?
What risk does this create?
What should we do next?
That's a much better use of expertise.
The engineer has moved from human query engine to decision maker.
THIS IS AN ATTENTION-ALLOCATION PROBLEM
Earlier this week, I wrote that engineering organizations increasingly have an attention problem rather than a telemetry problem.
This is another expression of it.
The question isn't only:
Which alert deserves attention?
It's also:
Which work deserves the attention of which engineer?
Not every operational task requires a staff engineer.
Not every piece of context needs to be manually assembled.
Not every alert requires multiple people to independently reconstruct the same evidence.
If machines can perform more of the mechanical interpretation work, human expertise can move toward the decisions where it creates the greatest value.
That's operational leverage.
WHAT SIGNAL AUDIT IS TRYING TO CHANGE
This is one of the workflows we're exploring with Signal Audit.
A production signal arrives.
The observability platform has already done its job.
The telemetry exists.
Instead of simply forwarding that information to another destination, Signal Audit attempts to turn it into an operational finding.
What happened?
Why might it matter?
What evidence supports the interpretation?
What should engineering investigate next?
The engineer receives a starting point.
Not an unquestionable answer.
Not an autonomous remediation.
A starting point.
That's important because the objective isn't to remove engineers from production.
It's to improve where their attention begins.
THE MOST VALUABLE FIVE MINUTES
Imagine an experienced engineer has five minutes available during an emerging incident.
How should those five minutes be spent?
Searching logs?
Finding the last deployment?
Determining service ownership?
Comparing timestamps?
Or:
Evaluating the strongest available hypothesis?
Assessing customer risk?
Challenging the interpretation?
Deciding whether to roll back?
Determining whether the incident needs to escalate?
Those are very different uses of the same five minutes.
And at organizational scale, the distinction matters.
OPERATIONAL INTELLIGENCE SHOULD CREATE LEVERAGE
Engineering organizations have spent years creating leverage around software production.
Automation reduced manual deployment work.
Cloud platforms reduced infrastructure provisioning work.
CI/CD reduced release friction.
Infrastructure as code reduced configuration work.
AI is now reducing implementation work.
The same principle should apply to operational understanding.
If a machine can help assemble context, let it.
If it can identify relationships between signals, let it.
If it can propose a hypothesis supported by available evidence, let it.
Then put the human where human capability is most valuable:
Judgment.
The goal isn't fewer engineers looking at production because machines know better.
It's engineers spending more of their limited attention on the parts of production that actually require engineering.
PROTECT THE JUDGMENT LAYER
The most experienced people in an engineering organization are expensive for a reason.
They've accumulated context.
They've seen systems fail.
They've made mistakes.
They've learned which patterns matter.
They understand consequences that aren't always visible in telemetry.
That expertise shouldn't be consumed unnecessarily by information retrieval.
As production environments become more dynamic and engineering organizations become increasingly leveraged by AI, I think this distinction becomes more important.
Machines are getting better at producing information.
They're getting better at organizing information.
They're getting better at interpreting information.
Humans remain responsible for deciding what to do with it.
That's where we should want our best engineers.
Not acting as human query engines.
Acting as the judgment layer.
Your most experienced engineers shouldn't have to reconstruct the same operational context every time production changes.
The Signal Audit Vanguard Program is a founder-led, 30-day deployment using your existing observability environment and production telemetry.
We're working with three engineering organizations to evaluate whether operational intelligence can reduce context-reconstruction work, focus scarce engineering attention, and give engineers a stronger starting point for production decisions.
