Your robots.txt Changed Last Tuesday. You Just Don't Know It Yet

A few months ago I wrote about the five tests that decide whether ChatGPT can read your website. The punchline of that post was that most sites fail at least one of them, and the fix is usually an hour of work.

What I didn’t say in that post is what happens after you fix it.

You fix it, you verify it, you move on. And then, somewhere between three weeks and six months later, it unfixes itself. Quietly. Nobody emails you about it.

How sites break themselves

I’ve now watched this happen enough times to name the patterns.

A managed WordPress host ships a “security enhancement” that adds a Disallow for a list of AI bots. A CDN turns on a bot-fight mode that doesn’t distinguish GPTBot from a scraper farm. A well-meaning developer copies a robots.txt from an old project during a redesign. An SEO plugin update rewrites the file to match its new defaults.

None of these show up in uptime monitoring. The site loads fine. Google still indexes it. The owner checks it in a browser and everything looks right. Meanwhile ChatGPT has been telling people the business doesn’t exist for two months, and the only symptom is a slow leak in a traffic channel most analytics setups don’t even break out.

This is not hypothetical. Cloudflare made blocking AI crawlers the default for new sites in 2025. If you launched or moved a site after that, there is a real chance you opted into invisibility without reading the checkbox.

So I built the thing that watches

The free AI visibility audit on this site has run thousands of scans at this point. It’s a snapshot: thirty seconds, one URL, here’s what’s broken. People run it, fix their stuff, and disappear. Which is fine — that’s what it’s for.

But the snapshot model has a hole in it, and the hole is everything I just described. So I turned the audit engine into a monitor.

It’s called the AI Visibility Monitor, and the deal is simple. You add your site. It scans immediately so you get a baseline. Then every Monday morning it scans again: robots.txt rules for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Google-Extended, plus the twelve answer-engine-readiness signals from the audit — FAQ schema, question-shaped headings, entity grounding, passage structure, all of it.

You do not get a weekly email. Weekly “everything is fine” emails are noise, and noise gets filtered to a folder nobody opens. You get an email when one of two things happens: a crawler’s access flips — allowed to blocked, or blocked back to allowed — or your overall score moves five points or more. Silence means all clear.

When it does email you, it tells you exactly what changed. “PerplexityBot can no longer crawl this site (robots.txt change). Score: 74 → 51.” That’s the whole alert. No digest, no upsell copy, no “your weekly report is ready.”

The parts I’m happy with

The scan engine is the same one the free audit uses, extracted into a shared library so the free tool, the monitor, and the on-demand rescan button all run identical logic. Fix a scoring bug once and it’s fixed everywhere.

Login is a magic link. No passwords. You subscribed with an email; that email gets a fifteen-minute, single-use sign-in link. The session is a signed cookie. There is no credential database to leak because there are no credentials.

Alerts consolidate. The Agency tier watches ten sites, and if three of them break in the same week — say a shared host pushes a bad robots.txt to the whole fleet, which is exactly when you need this — you get one email covering all three, not three separate pings.

The dashboard keeps score history, so when a client asks “is this AI thing actually doing anything,” you can show the line going up after the fixes shipped. Agencies will sell retainers off that chart alone.

What it costs

Solo is $19 a month, one site. Agency is $49 a month, ten sites. Monthly, cancel anytime, no annual lock-in.

For context, the enterprise AEO monitoring tools start around $100 a month and go hard from there. This is deliberately not that. It’s the smoke detector version: cheap, quiet, and exactly loud enough the one time it matters.

If you haven’t run the free audit yet, start there — it’s the same engine, free, no signup. If the results made you fix something, the monitor is how you know it stays fixed. If everything came back clean, the monitor is how you know it stays clean.

Either way: your robots.txt is going to change someday without asking you. Now you’ll know the same morning it happens.