Harbour Desk is a fictional company, but its problem is real everywhere: a nine-person travel-tech startup in Kochi, booking backwater tours and houseboat stays, with exactly two people handling everything technical. Devika handled infrastructure; Rohan handled the alert inbox, which had begun eating his mornings. So when a vendor demonstrated an agent that could close their phishing queue automatically \u2014 a shiny slice of what the market now calls AIOps for Security \u2014 Rohan was ready to sign before the coffee cooled. Devika wrote one word on the whiteboard: prove. This is the story of what proving took, told month by month, with every number drawn from the field’s public reference material rather than from imagination.<\/p>\n
The demo was genuinely impressive. The agent read a pile of suspicious emails, closed the harmless ones, drafted a one-line reason for each, and finished the afternoon’s queue in minutes. Rohan started calculating saved salaries. Devika asked the vendor two questions that changed the shape of the deal: “Where did you measure those accuracy figures \u2014 on whose mail?” and “If it closes something it shouldn’t, how fast can we pull it back?” The first answer was vague; the second was honest, a one-click reopen. They took the trial anyway, partly because Rohan wanted the hours back and partly because Devika suspected the only way to learn this category was hands-on. But they wrote their own rules for it first, which is where this story becomes useful to anyone else buying in this category:<\/p>\n
Rule one did the heavy lifting. For thirty days the agent suggested and recorded while touching nothing, and its verdicts were scored against labels written by humans who never saw what the agent claimed. Blind labels are the whole game here \u2014 research in this field, spread over 2019 through 2022 (the TESSERACT findings), revealed how test sets carrying spatial and temporal bias could float scores toward a fake 0.99, so Harbour Desk trusted no evaluation that graded itself. Rule two mattered just as much: the rules came before the trust. The market, notably, does not always wait \u2014 Microsoft’s Phishing Triage Agent starts closing the day it runs \u2014 as of October 2026, an observe-only setting remains undocumented for it. Devika’s view, later repeated to every founder who asked, was that a product closing on day one simply means the buyer owes themselves the observation period the vendor skipped.<\/p>\n
Month two produced the agreement, one page long. The agent could recommend \u2014 rank the queue, draft reasons, let Rohan decide. Closing on its own required specific evidence, not vibes: demonstrated misses at or under 2%, measured across at least 150 genuine cases; a holdout set the vendor never touched; canary items planted in the live flow; and the reopen button wired so either of them could pull a wrong closure back in seconds. Ladder language exists for exactly this \u2014 recommending sits at L1, closing at L2, and the evidence gates are published in the Ops Atlas handbook entry for AIOps for Security<\/a>, which Harbour Desk kept open in a browser tab for the entire quarter. They added one clause of their own, after reading about drift: every month, the numbers get re-cut by category, because averages age badly and blended figures hide the categories that fail.<\/p>\n They planted the canaries themselves: fifty crafted threats per quarter, indistinguishable from live traffic to the agent, slipped in where no filtering could dodge them. Month three opened well \u2014 the agent passed every canary, missed nothing in spot checks of its own closures, and the team upgraded it to supervised closing. Then, in the second week of the new arrangement, a canary disappeared. One closed canary, under the agreement, meant a logged incident and a review. Two, over the following fortnight, meant the pattern was real. The third pulled the plug automatically \u2014 their written three-strike rule \u2014 and auto-close ended that afternoon. Nobody argued, which is the quiet benefit of rules written before emotions arrive: the argument happened months earlier, on a calm day, between two colleagues who liked each other.<\/p>\n Here is what separation from the drama revealed. The agent’s closes were still, in raw counts, mostly right; the problem was that a fifty-canary sample can only bound the miss rate near three-over-fifty \u2014 the rule of three in action \u2014 so their scheme detected gross failure, not fine calibration. The honest response was arithmetic, not outrage. They regrouped: agent back to recommending at L1, canaries continued, and a proper blind-labelled evaluation rebuilt the evidence \u2014 because they knew small clean samples prove little, and that demanding a 2% ceiling takes roughly 236 graded cases once one miss exists, about 313 once two do. They also demanded category-level results, having read how a blended score once hid a 42.3% cloud-identity miss rate behind a pleasant 9.1% phishing figure. Harbour Desk’s queue skewed heavily toward booking-confirmation spoofs, and that category, it turned out, was exactly where the agent was weakest. Drift monitors went on the calendar the same week: mail patterns change, attackers change, and a clearance earned in one season cannot be assumed in the next. They also rehearsed the withdrawal once, end to end, so that if a real incident ever demanded the off switch, the motion would be muscle memory rather than improvisation.<\/p>\n By the quarter’s end, the agent had earned back supervised closing under the same three-strike contract, and the two-person team had something more valuable than saved hours: a system they could explain to an auditor in one breath. Their summary, lightly edited, works for any small team eyeing this technology. Trust attaches to actions, not products. Watching costs nothing. Recommending needs labelled volume behind it. Closing needs a miss bound, a holdout, canaries, and that reopen button within reach at all times. Anything irreversible stays human \u2014 the field’s guidance from MITRE keeps a person responsible for judging whatever the machinery outputs. And if a vendor’s product closes from its first day, that is not a feature; it is homework the buyer just inherited. What surprised them most was how little time the discipline cost \u2014 a month of shadow, one written page, fifty planted cases a quarter \u2014 against how much credibility it bought when the first real incident arrived.<\/p>\n Kerala readers will recognise the deeper rhythm in all this: look before you leap is how the backwaters were mapped, and it is how the good stuff gets found \u2014 whether that is the picks in Kerala’s events, travel and local experiences<\/a>, the skills path in a DevOpsSchool career guide<\/a>, or the patient comparison behind choosing a drone and finding the right services<\/a>. Harbour Desk does not exist. The discipline it dramatises does \u2014 and any two-person team can run it, exactly as written, before the first coffee of the demo cools.<\/p>\n","protected":false},"excerpt":{"rendered":" A fictional Kochi travel-tech team road-tests AIOps for Security their own way: a shadow month, planted canaries, a three-strike withdrawal, and a drop to L1.<\/p>\n","protected":false},"author":7,"featured_media":2512,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2058],"tags":[],"class_list":["post-2511","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-travel-tech"],"_links":{"self":[{"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/posts\/2511","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/comments?post=2511"}],"version-history":[{"count":1,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/posts\/2511\/revisions"}],"predecessor-version":[{"id":2534,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/posts\/2511\/revisions\/2534"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/media\/2512"}],"wp:attachment":[{"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/media?parent=2511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/categories?post=2511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/keralaorbit.com\/blog\/wp-json\/wp\/v2\/tags?post=2511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The Week a Canary Got Closed<\/h2>\n
Back Down a Rung, Not Back to Zero<\/h2>\n
What Harbour Desk Would Tell Your Team<\/h2>\n