What if AI ran the crawler end to end — from design to incident response?

The first thing we changed was the only-a-human-can-fix-it structure. Each site used to have its collection method buried inside the program, so any change meant a developer opening the code. We moved that out into 'settings' outside the code. Now what to collect, and how, is decided in settings, not in the program — a safe surface for AI to touch, set up first.

This system doesn't run on one machine but across three servers. The strongest hub server carries 28 collectors plus alerts, self-recovery, and monitoring, while the other two run 22 and 20 collectors and send their results to the hub. Responsibilities are split so a problem on one server doesn't stop the whole thing.

On top of that we added a 'fixes itself' flow. When a site comes back empty several times in a row (five by default), the system treats it as a failure and first alerts the ops channel: 'collection on site X is broken.' It usually takes just a few minutes — the system raises its hand before a person even notices.

It doesn't stop at alerting. AI doesn't fix by guessing. It goes into the site itself to see how the page changed and whether there's a faster, more reliable way to get the data, then re-tunes the collection settings the best way it can. It does what a human would during an incident — except it doesn't wait for a person, starting at midnight or on a weekend all the same.

What we cared about most was keeping AI's fixes from reaching production carelessly. A fix AI makes must first pass an automatic check. On pass, it only files a review request saying 'here's how I fixed it' — it never goes straight to production. We built both a fully automatic path that runs from detection to review request with no human touch, and a semi-automatic path an operator starts with one button. Either way, a human makes the final call. If fixes keep failing, it stops at a set limit and calls a person.

One more thing. Giving AI access doesn't mean letting it do anything. The more autonomously AI acts, the more you have to fence off 'what it can touch, and how far' first — the currently recommended approach, which we followed. The only thing AI can change is a site's collection settings; the system's core and sensitive information are locked away entirely. Content scraped from a site is treated as something to read, never a command, so even if a page hides 'ignore your previous instructions,' AI isn't swayed. And who changed what, when, and who approved it is all kept on record, so it can be retraced later.

We also reworked the invisible plumbing underneath. We swapped the old, slow internals for modern ones so the same servers handle more sites, faster. Programs used to need a person to restart them one by one; now the system revives them the moment they stop. Adding a server or more sites just takes a settings change. And whatever happened in between is all there on the dashboard, site by site.

If everything so far was about making it run, the next part was seeing — at a glance — whether it runs well. We gather the activity records pouring out of dozens of programs into one place and pull out just the signals worth worrying about. When something looks off, we drill straight down to which site had what problem, right down to the raw record.

Finally, we wrote out the rules for how AI should handle this system in advance. We noted each site's quirks and past changes, and bundled the frequent jobs (analyze a site, fix it, add a new site, run a full check) so each can be called with a single command. Adding a new site takes one line — 'add X' — and AI finds the site, confirms it, lays out the settings, checks, docs, and review request, then waits, switched off, for a human's final approval.