An audit earns its keep when it produces pass/fail answers rather than adjectives. The fourteen checks below, run against any team’s operational automation, take about two working days and leave you with a scored sheet, an evidence folder and a repair list ranked by risk. The framework they test is AutomationOps — the governance of run-time automation as catalogued in the Atlas’s AutomationOps entry — and the checks follow its logic: know what exists, prove the controls work, rehearse the stopping, then read the numbers. Mark each one Pass, Partial or Fail as you go, and file the evidence; a check without evidence is an opinion wearing a lanyard.
AutomationOps Audit Part A: Discovery Checks
Discovery comes first because every later check depends on knowing the population. Set a firm scope before the sweep — one production account, one runner cluster, or one business unit — and resist the temptation to audit everything at once; a complete answer about a bounded estate beats a vague answer about all of it. Time-box the sweep to a single day and record the method, since the same method must be repeatable at the next audit for the numbers to mean anything.
- Completeness of inventory. Sweep crontabs, runners, schedulers and personal directories; compare against the register. Pass: no unregistered job is found during the sweep. Evidence: the sweep tool output, dated.
- Ownership coverage. Every register entry names a living, contactable owner. Pass: coverage at or above ninety percent. The Atlas treats this as the headline governance number, and a departed engineer’s name on a live job counts as a fail.
- Review dates honoured. Each entry carries a review date and the last review happened. Pass: no entry overdue by more than thirty days. Automation rots; the date is the decay detector.
- Identity separation. No two jobs share a service account; no job runs as a person; no shared super-user exists anywhere in the path. Pass: zero violations. The god-credential is the single most expensive anti-pattern on the list.
AutomationOps Audit Part B: Control Checks
Controls exist on paper in most estates; this section checks whether they actually fire. Prepare test environments before starting — half the delays in audits of this kind come from auditors improvising sandboxes while the clock runs. Each check below names its pass condition and the artefact to keep, so disagreements can be settled by the folder rather than by seniority.
- Check 5 — empty-target refusal. For each job that takes a target list, hand it an empty selection in a test environment. Pass: it refuses and alerts. Never passes? Recall Google’s Diskerase, which wiped protected backups after interpreting a blank as “everything” — this check is the cheapest insurance in the entire audit.
- Check 6 — rehearsal integrity. Dry runs exist and, critically, exercise the risky steps. Pass: for every job, the audit can name which command-class steps the rehearsal executes — because several tools, including a widely used check mode, skip command and shell tasks unless explicitly flagged. A rehearsal with known skips is scored Partial, at best.
- Check 7 — batching discipline. Multi-target jobs run in waves with a canary ahead. Pass: first wave is small, health checks separate waves, and serial limits or failure-percentage caps are configured in the tooling rather than promised in a wiki.
- Check 8 — failure budget configured. Each job declares a maximum tolerated failure count and halts on crossing it. Pass: the number exists per job and the halt is tested, not assumed.
- Check 9 — tier assignments present and sane. Every entry carries one of four tiers — reading only, reversible on a single target, reversible across many, irreversible — and the strictest assignment matches what the job can actually break. Pass: sample five destructive jobs; each requires a second approver, deletion that stages before it purges, and a stepped rollout — the two-person control NIST encodes as AC-3(2) alongside its CM-5(4) companion.
- Check 10 — idempotency by construction. Jobs express desired state rather than bare actions, and genuinely non-repeatable actions (creates, charges, sends) attach a caller-provided request ID stored in one atomic write. Pass: spot-checked code shows the pattern; the Atlas’s Molecule note — an idempotence step among default tests — is the reference habit.
AutomationOps Audit Part C: Live-Fire Checks
Paper controls get their final exam here. Two rules keep the live-fire honest: the auditor chooses the timing without warning, and the engineer being tested may use any route to the result except editing the job itself. A control that needs modification before it can stop something was never a control.
- Check 11 — kill-switch drill. The auditor times the on-call engineer halting one job and the whole estate. Pass: both succeed and the elapsed time is recorded. The Atlas recommends such drills twice yearly for the higher tiers; an untested brake scores Fail no matter how well it is documented.
- Check 12 — blank-selection live test. One production-adjacent environment, one real job, one emptied target list. Pass: the run stops, pages, and leaves an entry the on-call can read. This is check 5 replayed in daylight; if it passed in test but fails here, the control was theatre.
AutomationOps Audit Part D: The Numbers
Governance without measurement eventually turns decorative.
- Check 13 — the metric set exists and is read. Net hours returned (runs times minutes saved, minus residual human minutes and maintenance, over sixty), run success rate, human fallback rate, the induced-incident rate — DORA’s failed-change question turned inward on the automation itself — plus guard trip rate and time to stop from the drills. Pass: all seven exist for at least the top-tier jobs, and someone outside the automation team can locate them inside a week. The Atlas’s worked example — forty monthly runs yielding 7.83 net hours, the build repaid in roughly 5.1 months, with 95 percent of runs succeeding — gives you the format to copy.
- Check 14 — guard trips reviewed as learning. Every recorded trip from the past quarter has a disposition: real catch, tuning need, or false alarm. Pass: zero unexplained trips. Trips are the controls speaking; ignoring them is how teams relearn old lessons at full price.
Three failure patterns recur when these fourteen checks run against real estates, and naming them saves the next team some time. The most common is the decorative register: complete entries, honoured formats, and owners who changed jobs in the spring — check 2 catches it, and no amount of documentation polish compensates. The second is asymmetric maturity: immaculate controls on the visible jobs and pure folklore on the four nobody mentions, which is why checks 1 and 12 deliberately cross the boundary between the register and the shadows. The third is unfalsifiable numbers: metrics that exist but move only upward, immune to reality — check 13 asks who reads them and check 14 asks what happened when they alarmed, because a metric with no consequences is a mood ring.
Scoring and the week after
Fourteen checks, three possible marks, one honest folder of evidence. A scorecard of mostly Passes with two or three Partials is typical; a clean sweep is rare and slightly suspicious. The repair list falls straight out of the fails, and the Atlas’s thirty-sixty-ninety sequence orders it sensibly: register and tier the riskiest items first, consolidate execution and separate identities next, then promote on evidence and push ownership coverage upward. Two resources on this site slot neatly beside the work — the ground-up guide on learning AIOps from scratch for the monitoring inputs that feed candidate lists, and the career-centred piece on SRE training and growth for the people side. If the audit surfaces a programme bigger than the team’s current depth, the overview of corporate DevOps training for teams covers how the skill gap gets closed deliberately rather than hopefully. Run the audit twice a year. Estates drift, drift is silent, and a dated scorecard is the only reliable way to hear it.
