Your new workforce won't be all human. Join the
conference to learn how to manage it.
request invite
TL;DR: An expired TLS certificate on a payments endpoint went unnoticed for 90 minutes because no alert fired. An AI Coworker renewed it in minutes, then stopped and asked before touching the live endpoint. The fix was fast. The gap that made it necessary is the harder problem.
Nobody noticed for an hour and a half.
At 04:56 UTC the TLS certificate on our payments API expired. The endpoint did not fall over. It kept running, kept answering requests, kept presenting a dead certificate to anything that connected. Clients validating TLS properly got errors. Clients not validating properly got served, which is the worse outcome, because it means the failure is invisible to half of your traffic.
I opened a ticket around 06:20. By then it had been broken for 1.3 hours. No monitoring rule had caught it. No alert had fired. The thing that detected the outage was me noticing something felt off. That is not a detection strategy.
A routing Coworker picked up the ticket and passed it to the Certificate Custodian. Twenty-four steps, eight tools. But what it did first mattered more than how fast it did it. It told me what was happening, in the present tense: active outage, the certificate on the payments API expired at 04:56 UTC today, the payments endpoint is serving an expired certificate right now, this is live.
Present tense, not past. That framing shift was more useful than I expected at six in the morning.
The Coworker renewed the certificate on its own. Then it stopped.
Generating a new certificate changes nothing about production. It is additive, reversible, boring. Nobody’s traffic shifts because a certificate exists in a vault somewhere. So the Coworker did that without asking.
Rebinding the live endpoint is different. Brief, low risk, still a production change during business hours. The payments API starts serving a different certificate the moment the bind completes. So the Coworker did not do that on its own, even though it was mid-outage and the fix was obvious and every second counted. It asked.
The approval request it put in front of me is the part I keep coming back to.
Current certificate expired 1.3 hours ago. Replacement expires 2027-08-26. Effect: the endpoint begins serving the new certificate, brief reconfiguration, no restart.
And then this: if declined, the payments endpoint continues serving an expired certificate and clients continue to receive certificate errors. Every minute of delay is continued downtime.
Closing line, its own words: the approver is choosing between a short reconfiguration and continued failure.
Almost every approval request in IT tells you what happens if you say yes. This one told me the cost of saying no. At six in the morning, when you are the person holding up a payments endpoint, that is the only framing that helps. It did not pressure me. It priced the delay.
I approved it.
The boundary here is worth being precise about. The test is not whether the Coworker can act. The test is whether it knows which of its steps a human owns. Renewing a certificate and rebinding a live endpoint are part of the same incident, but they carry different consequences. Same workflow, different permission. That distinction is the entire design.
After the rebind the Coworker did not report success from its own action record. It read back from the live resource: the payments endpoint is now serving the new certificate, valid through 2027-08-26.
An action that reports success and a service that is actually fixed are two different claims. Anyone who has been burned by a green deployment status on a broken service knows the difference. The Coworker verified the second claim, not the first.
Outage resolved. TLS restored.
Fast recovery is not the win here. It is a consolation prize.
That certificate had a known expiry date for a year. A year. The failure was on the calendar the entire time. A certificate lifecycle policy, a 30-day-out renewal trigger, a simple inventory sweep — any of those catches this before it reaches production. None of them existed. The thing that caught it was me, awake early, noticing something felt wrong.
If I had been asleep, or on a plane, the payments API would have kept serving an expired certificate all morning. An endpoint nobody has inventoried is still invisible, no matter how good your agent is.
The Coworker fixed the incident. It did not fix the reason the incident reached production. The fix is not faster renewal. It is never needing to renew in a panic.
This ran in a sandbox resource group, not a production environment. No customer traffic was affected. Validate outcomes in your own environment.
Vijay Shankar is Head of Solution Engineering at Atomicwork.
