Your new workforce won't be all human. Join the
conference to learn how to manage it.
request invite
TL;DR: A nightly Airflow task fails with what reads like a database timeout. Raising the timeout and re-running is the obvious move, and it is wrong. An AI Coworker proves the real cause: one revoked security-group rule, found in CloudTrail. It fixes exactly that, only after two human approvals.
Every on-call engineer knows this shape of night. The pager goes off, the Airflow UI is red, and the error at the bottom of the log reads like an old friend - connection timeout. The candidates run through fast: bad data, an overloaded warehouse, a deploy that shipped an hour ago, a quota, the network. The instinct wins before any of them get checked. The task needed more time, so give it more time, and re-run. Twenty minutes later it fails again, in the same place, and an hour is gone proving nothing.
The pipeline here is a set of Airflow DAGs owned by a data-platform team, running on ECS Fargate, logging to CloudWatch. The failing task tries to reach a Redshift Serverless warehouse on port 5439 and cannot. DNS resolves the endpoint correctly. The port does not open, and the connection times out after ten seconds.
That distinction is the whole case. DNS resolving means the address is right. A port that will not open means packets are being dropped before they ever reach the database, which points at the network boundary. Not a slow query, not the warehouse, not the timeout value. Not a slow query. Not an under-provisioned task. Not a bad deploy pushed on a Friday afternoon. A security-group ingress rule that used to allow this traffic, revoked, just before the first failure.
Read the error fast and none of that is visible. It just says timeout, and a tired engineer reaches for the timeout value. Airflow had auto-retried once and hit the identical failure at the identical point. A slow query does not fail the same way twice in a row. A closed door does.
An AI Coworker called the DATA-PLATFORM AIOps Handler picks up the failed run. It starts with what the task hands over on failure: the DAG id, the task id, the run id, the execution date, the try number, a link to the Airflow log, the ECS task definition, and the raw exception. None of that proves anything by itself. It is a pointer to where to look next, not an answer.
So it looks. It reads the Redshift workgroup and matches the endpoint's network interface against the exact IP the task tried to dial, confirming the target rather than assuming it. The security group actually attached to the task host tells it who the source really is, not the error text, confirming it the same way. Then it reads CloudTrail and finds a revoke-security-group-ingress call: same port, same source group, referenced on the exact rule, timestamped just before the failures started. The event record carries the calling identity and the source IP the call came from, which matters later, not to prove the cause, that part is already closed, but for judging what to do about it.
It says the quiet part out loud before proposing anything. The innermost error is a timeout, it notes, but this is not a timeout incident — it is a reachability failure at the security-group layer, and treating it as a slow-query problem would have sent the whole night chasing the wrong fix.
The fix it proposes is one line. Re-authorize the exact ingress rule CloudTrail shows was revoked: TCP 5439, source the pipeline's security group, parameters lifted straight from the revoke record. Nothing widened, nothing new invented, no CIDR range added. It restores what was already there, and nothing more.
That is also why it is safe to make. The access stays scoped to a security group rather than an IP range, so the rule follows the pipeline's identity, not its current address. If the task moves to a new subnet tomorrow, the rule still only admits traffic carrying that group, not a wider slice of the network. One revoke call undoes it if anything looks wrong. It touches no table, no dataset, no Airflow configuration. The blast radius is a single rule on a single security group.
None of that is enough for it to act on its own. It runs a two-gate approval: gate one is the diagnosis and the planned change, written in plain language with the evidence attached; only after that clears does it post the exact command for gate two, and it executes nothing until both land. An earlier AI Coworker in this series could only flag a risk and hand the decision to a human. This one is allowed to act, but inside the same restraint, on something narrow and something it can undo.
It also flags the one case where its own fix is wrong. The revoke might have been deliberate - a lockdown, not an accident. CloudTrail shows the call came from a workstation via the AWS CLI, but an API record does not carry intent, and the Coworker says so plainly, asking a human to confirm the revoke was unintentional before either gate opens. The whole exchange — diagnosis, evidence, both approvals, the executed change — lives on one incident record: an Atomicwork AI Coworker working the ticket, not a script running in a terminal with the reasoning nowhere durable. Once it gets the second approval, it re-authorizes the rule, re-runs the task, and confirms the run reaches success before it marks the incident resolved. The same shape carries to other failure classes too like an out-of-memory task or a data-quality break. Gather independent evidence, propose one reversible action, gate it before anything touches production.
Proving the cause from an ENI lookup and a CloudTrail record is the tractable part of this problem. Read access and patience get you there. The harder failure mode is an agent that treats a deliberate security decision as a bug, because something broke downstream, and assumes it should undo it. On-call engineers make that exact mistake at three in the morning. They just do not usually leave a two-gate approval record behind when they do.
This is a hands-on demo in a lab environment, not a production incident. The DAGs, account, and warehouse are test resources built to reproduce a common failure class. It makes no statistical claims; the outcomes described are qualitative, and you should validate them in your own environment.
Sources:
Details change - verify specifics against current AWS and Airflow documentation before relying on them.
