[{"data":1,"prerenderedAt":178},["ShallowReactive",2],{"seo-features/robots-txt-monitoring":3},{"slug":4,"kind":5,"archetype":6,"cluster":7,"navGroup":7,"navLabel":8,"meta":9,"breadcrumbs":14,"hero":22,"sections":62},"features/robots-txt-monitoring","spoke","capability","monitoring","Robots.txt Monitoring",{"title":10,"description":11,"canonicalPath":12,"twitterCard":13},"Robots.txt Monitoring & Disallow Alerts","Catch an accidental Disallow: / the day it ships — before organic traffic collapses three weeks later. Sitewatch monitors robots.txt for deindex regressions.","/features/robots-txt-monitoring","summary_large_image",[15,18,21],{"label":16,"href":17},"Home","/",{"label":19,"href":20},"Features","/features",{"label":8,"href":12},{"eyebrow":8,"headline":23,"intentStatement":24,"bullets":25,"primaryCta":35,"secondaryCta":38,"proofPanel":40},"The most expensive one-line deploy mistake, caught the day it ships","A staging config ships to production and now robots.txt says Disallow: /. Google is about to deindex your client's entire site — and you won't find out until organic traffic collapses three weeks later. Sitewatch catches the change the day it happens.",[26,29,32],{"icon":27,"text":28},"heroicons:no-symbol","Flags Disallow: / as a high-severity regression",{"icon":30,"text":31},"heroicons:exclamation-triangle","Alerts when robots.txt goes from reachable to a server error",{"icon":33,"text":34},"heroicons:clock","Caught on the next check — not weeks later in your rankings",{"label":36,"href":37},"Monitor robots.txt free","/scan",{"label":39,"href":20},"Back to features",{"type":41,"incidentTitle":42,"severity":41,"affectedLabel":43,"detectedLabel":44,"eventLines":45},"incident","Robots.txt regression detected","Affected file","Detected in last check",[46,49,54,58],{"icon":27,"text":47,"time":48,"type":41},"robots.txt — Disallow: / added (site-wide block)","just now",{"icon":50,"text":51,"time":52,"type":53},"heroicons:document-text","Previous: Allow: / (crawlable)","6h ago","ok",{"icon":55,"text":56,"time":48,"type":57},"heroicons:magnifying-glass","Search engines will stop crawling all pages","warning",{"icon":59,"text":60,"time":61,"type":53},"heroicons:check-circle","sitemap.xml — still reachable","1m ago",[63,93,107,128,135,153],{"id":64,"tocLabel":65,"type":66,"eyebrow":67,"heading":68,"items":69},"benefits","Why it matters","benefits-grid","A symptom that lags the cause by weeks","An accidental Disallow: / is invisible until your rankings are gone",[70,73,77,81,85,89],{"icon":27,"title":71,"description":72},"Disallow: / detection","The single most damaging robots.txt change — telling every search engine not to crawl any page — is flagged as high severity the moment it appears.",{"icon":74,"title":75,"description":76},"heroicons:server","Reachability monitoring","If robots.txt goes from a healthy response to a 5xx or times out, Sitewatch flags it (medium severity). A missing or erroring robots.txt confuses crawlers.",{"icon":78,"title":79,"description":80},"heroicons:arrow-path","Regression-based, not noisy","Sitewatch compares against the previously-seen robots.txt. It alerts on a change for the worse — not on every harmless edit — so you only hear about real risk.",{"icon":82,"title":83,"description":84},"heroicons:rocket-launch","The classic deploy accident","Staging environments routinely block crawlers. When that config ships to production, the block ships with it — easy to do in a hurry, and slow to notice because the site still loads fine.",{"icon":86,"title":87,"description":88},"heroicons:chart-bar-square","Protect organic traffic","Lost rankings from an accidental block can take weeks to recover even after the fix. Catching it on day one is the difference between a non-event and a quarter of lost traffic.",{"icon":90,"title":91,"description":92},"heroicons:bolt","In the free scan, too","The robots-disallow-all check runs in the free, no-login public scan as well — so you can spot the problem before you even sign up.",{"id":94,"tocLabel":95,"type":96,"stats":97},"trust","At a glance","trust-strip",[98,101,104],{"value":99,"label":100},"High","Severity for Disallow: /",{"value":102,"label":103},"Day 1","Caught when it deploys",{"value":105,"label":106},"Free + paid","Available on every plan",{"id":108,"tocLabel":109,"type":110,"eyebrow":111,"heading":112,"groups":113},"detection","What we detect","detection-list-grouped","Robots.txt failure modes","How robots.txt quietly destroys your search visibility",[114,121],{"groupLabel":115,"icon":27,"items":116},"Crawl-blocking regressions",[117,118,119,120],"Disallow: / added — a site-wide block on all crawlers","A staging robots.txt shipped to production","A user-agent block that catches Googlebot or Bingbot","Wildcards that disallow far more than intended",{"groupLabel":122,"icon":123,"items":124},"Availability problems","heroicons:server-stack",[125,126,127],"robots.txt returning a 5xx server error","robots.txt newly returning 404 or redirecting away","The file timing out or becoming unreachable",{"id":129,"tocLabel":130,"type":131,"heading":132,"subtext":133,"primaryLabel":134,"primaryHref":37},"cta-mid","","cta-strip","Never get silently deindexed again","Add your site and Sitewatch watches robots.txt on every check. Free plan, no credit card.","Start monitoring free",{"id":136,"tocLabel":137,"type":138,"eyebrow":137,"heading":139,"items":140},"faq","FAQ","faq-accordion","Frequently asked questions",[141,144,147,150],{"question":142,"answer":143},"How does Sitewatch know a robots.txt change is bad?","It compares the current robots.txt against the previously-seen version. A new site-wide Disallow: / is flagged high severity; the file becoming unreachable or returning a server error is flagged medium. Harmless edits — adding a sitemap line, allowing a new path — do not alert.",{"question":145,"answer":146},"How fast will I hear about a Disallow: /?","On the next check after the change ships. Combined with deploy hooks, you can have a check run within seconds of a deploy — so an accidental block is caught immediately, not weeks later when rankings drop.",{"question":148,"answer":149},"Is this in the free public scan?","Yes. The robots-disallow-all check is part of the free, no-login scan as a stateless detection, and part of the paid monitor as a continuous regression check. Run a scan to see your current robots.txt status.",{"question":151,"answer":152},"Why is a wrong robots.txt so damaging?","Because the symptom lags the cause. A Disallow: / does not break the site for visitors — pages still load. But search engines stop crawling, rankings decay over the following weeks, and recovery can take just as long after the fix. The only cheap moment to catch it is the day it ships.",{"id":154,"tocLabel":155,"type":156,"eyebrow":157,"heading":158,"links":159},"related","Related","related-links-grid","Explore more","Related monitoring solutions",[160,165,170,175],{"label":161,"href":162,"description":163,"icon":164},"Sitemap Monitoring","/features/sitemap-monitoring","Catch broken URLs inside your own sitemap.xml.","heroicons:map",{"label":166,"href":167,"description":168,"icon":169},"Website Monitoring","/website-monitoring","Full-page verification beyond uptime checks.","heroicons:shield-check",{"label":171,"href":172,"description":173,"icon":174},"Protection Coverage","/protection-coverage","See the important pages you are not monitoring yet.","heroicons:chart-pie",{"label":176,"href":37,"description":177,"icon":90},"Free Website Health Check","Run a no-login scan — includes the robots.txt check.",1780857882783]