[{"data":1,"prerenderedAt":224},["ShallowReactive",2],{"seo-blog/robots-txt-disallow-all-deindexed":3},{"slug":4,"kind":5,"archetype":5,"cluster":6,"navGroup":6,"navLabel":7,"meta":8,"breadcrumbs":12,"hero":20,"sections":59,"article":221},"blog/robots-txt-disallow-all-deindexed","hub","resources","Robots.txt Disallow All Deindexed My Site",{"title":9,"description":10,"canonicalPath":11},"Robots.txt Disallow All Deindexed My Site: Fix It","A stray \"Disallow: /\" or accidental noindex can quietly drop your whole site from Google. Here's how it happens, how to catch it, and how to recover.","/blog/robots-txt-disallow-all-deindexed",[13,16,19],{"label":14,"href":15},"Home","/",{"label":17,"href":18},"Blog","/blog",{"label":7,"href":11},{"eyebrow":21,"headline":22,"intentStatement":23,"bullets":24,"primaryCta":34,"secondaryCta":37,"proofPanel":40},"SEO incidents","How a Stray \"Disallow: /\" Deindexed an Entire Site","Your site is up. It loads fine. But traffic just fell off a cliff — because one line in robots.txt, or a leftover noindex tag, is telling Google to stay away. This is one of the quietest \"up but broken\" failures there is: nothing errors, nothing crashes, and you usually find out weeks later when rankings are already gone. Here's exactly how it happens, how to confirm it, and how to make sure it never goes unnoticed again.",[25,28,31],{"icon":26,"text":27},"heroicons:no-symbol","How \"Disallow: /\" and noindex silently deindex sites",{"icon":29,"text":30},"heroicons:magnifying-glass","How to confirm whether you're actually blocked",{"icon":32,"text":33},"heroicons:shield-check","How to catch robots.txt changes the day they ship",{"label":35,"href":36},"Monitor your robots.txt","/features/robots-txt-monitoring",{"label":38,"href":39},"Check your site now","/tools/is-my-website-down",{"type":41,"reportTitle":42,"severity":43,"findings":44,"generatedAt":58},"report","robots.txt diff detected","incident",[45,48,51,53,55],{"label":46,"status":47},"HTTP status: 200","pass",{"label":49,"status":50},"robots.txt changed","fail",{"label":52,"status":50},"New rule: Disallow: /",{"label":54,"status":50},"All crawlers blocked",{"label":56,"status":57},"Pages still serving content","info","Site up — but blocked from Google",[60,67,81,115,121,146,168,186,213],{"id":61,"tocLabel":62,"type":63,"eyebrow":64,"heading":65,"html":66},"what-happened","What happens","prose","The failure","How one line takes a whole site off Google","\n\u003Cp>Most teams imagine deindexing as something dramatic — a hack, an outage, a manual penalty. In reality, the most common cause is far more boring: a \u003Ccode>robots.txt\u003C/code> file that, somewhere along the way, picked up this:\u003C/p>\n\u003Cp>\u003Ccode>User-agent: *\u003C/code>\u003Cbr/>\u003Ccode>Disallow: /\u003C/code>\u003C/p>\n\u003Cp>That tells every crawler to stay out of the entire site. The pages still load. Visitors who already have the URL see everything normally. But Google stops crawling, stops refreshing its index, and — over the following days and weeks — quietly drops your pages from search results.\u003C/p>\n\u003Cp>The sibling failure is a leftover \u003Ccode>noindex\u003C/code>. A staging environment ships \u003Ccode><meta name=\"robots\" content=\"noindex\">\u003C/code> on every page, or an \u003Ccode>X-Robots-Tag: noindex\u003C/code> header gets copied to production. Same outcome, even sneakier — robots.txt looks clean, but the pages themselves are begging to be removed.\u003C/p>\n\u003Cp>\u003Cstrong>Why it's an \"up but broken\" problem:\u003C/strong> nothing returns an error. Uptime monitors see 200 OK and report all green. The site is technically perfect. It's just invisible to the one audience that drives your organic traffic. This is exactly the gap \u003Ca href=\"/website-monitoring\">website monitoring\u003C/a> exists to close.\u003C/p>\n",{"id":68,"tocLabel":69,"type":70,"stats":71},"trust","Trust","trust-strip",[72,75,78],{"value":73,"label":74},"20","Detection rules",{"value":76,"label":77},"5–30 min","Check intervals",{"value":79,"label":80},"Free","1 site",{"id":82,"tocLabel":83,"type":84,"eyebrow":85,"heading":86,"items":87},"how-it-happens","How it happens","failure-modes-grid","The usual suspects","The most common ways this gets shipped",[88,93,97,101,106,110],{"icon":89,"title":90,"description":91,"severity":92},"heroicons:arrows-right-left","Staging config promoted to prod","Staging environments block crawlers on purpose with \"Disallow: /\". A deploy copies the staging robots.txt — or the staging env flag — straight to production. Now the live site is blocked.","critical",{"icon":94,"title":95,"description":96,"severity":92},"heroicons:square-3-stack-3d","CMS \"discourage search engines\" toggle","WordPress and many CMSes have a single checkbox that adds a site-wide noindex. A developer ticks it during a rebuild and forgets to untick it before launch.",{"icon":98,"title":99,"description":100,"severity":92},"heroicons:code-bracket","A bad migration or template change","A theme update, framework upgrade, or new build pipeline regenerates robots.txt from a default template — and the default happens to disallow everything.",{"icon":102,"title":103,"description":104,"severity":105},"heroicons:document-duplicate","X-Robots-Tag header at the server/CDN","A noindex header set at the nginx, Apache, or CDN layer applies to every response. robots.txt looks fine, the HTML looks fine, but the header is removing you from the index.","moderate",{"icon":107,"title":108,"description":109,"severity":105},"heroicons:pencil-square","A \"quick fix\" that overreaches","Someone wants to block one folder, writes \"Disallow: /\" by mistake instead of \"Disallow: /admin/\", and ships it. One missing path segment, whole site gone.",{"icon":111,"title":112,"description":113,"severity":114},"heroicons:clock","A temporary block that was never removed","You block crawlers during a big launch or migration \"just for a day\" — and the line stays in robots.txt for three months because nothing reminds you it's there.","low",{"id":116,"tocLabel":117,"type":63,"eyebrow":118,"heading":119,"html":120},"confirm","Confirm it","Diagnosis","How to confirm you're actually blocked","\n\u003Ch3>1. Read your live robots.txt\u003C/h3>\n\u003Cp>Open \u003Ccode>https://yoursite.com/robots.txt\u003C/code> directly in a browser. Look for any \u003Ccode>Disallow: /\u003C/code> under \u003Ccode>User-agent: *\u003C/code>. A blank Disallow (\u003Ccode>Disallow:\u003C/code>) means \"block nothing\" — that's fine. A single slash means \"block everything\" — that's the problem.\u003C/p>\n\u003Ch3>2. Check for a noindex meta tag\u003C/h3>\n\u003Cp>View source on a few key pages and search for \u003Ccode>noindex\u003C/code>. Then check response headers with \u003Ccode>curl -I https://yoursite.com/\u003C/code> and look for an \u003Ccode>X-Robots-Tag: noindex\u003C/code> line. The header overrides nothing-looks-wrong HTML.\u003C/p>\n\u003Ch3>3. Use Google Search Console\u003C/h3>\n\u003Cp>Run the URL Inspection tool on an important page. If it says \"Blocked by robots.txt\" or \"Excluded by 'noindex' tag,\" you have your answer. Check the Pages (Indexing) report for a spike in excluded URLs — that's the smoking gun.\u003C/p>\n\u003Ch3>4. Check how long it's been live\u003C/h3>\n\u003Cp>This is the part that hurts. Most teams discover the block weeks after it shipped, because the only signal was a slow traffic decline. The fix takes five minutes; the lost rankings take months to recover. That gap between \"it broke\" and \"we noticed\" is the entire reason to monitor robots.txt continuously rather than auditing it occasionally — see \u003Ca href=\"/features/robots-txt-monitoring\">robots.txt monitoring\u003C/a>.\u003C/p>\n",{"id":122,"tocLabel":123,"type":124,"eyebrow":125,"heading":126,"steps":127},"fix","Fix & recover","how-it-works-stepper","Recovery","How to fix it and recover indexing",[128,132,137,141],{"number":129,"icon":107,"title":130,"description":131},"1","Remove the block","Delete the \"Disallow: /\" line (or change it to the specific path you meant to block). Remove any site-wide noindex meta tag and the X-Robots-Tag: noindex header. Untick the CMS \"discourage search engines\" box.",{"number":133,"icon":134,"title":135,"description":136},"2","heroicons:arrow-path","Redeploy and verify the live file","Confirm the production robots.txt and page headers are clean — not just your local repo. CDNs cache robots.txt, so purge the cache and re-fetch the live URL to be sure.",{"number":138,"icon":29,"title":139,"description":140},"3","Request re-crawling","In Search Console, use URL Inspection → Request Indexing on key pages, and resubmit your sitemap. This nudges Google to recrawl faster than waiting for the natural cycle.",{"number":142,"icon":143,"title":144,"description":145},"4","heroicons:bell-alert","Add a tripwire so it can't happen silently again","Set up continuous robots.txt and noindex monitoring. The next time a deploy reintroduces a block, you find out in minutes — not when quarterly traffic is already down.",{"id":147,"tocLabel":148,"type":149,"eyebrow":150,"heading":151,"items":152},"prevention","Prevention","benefits-grid","Never again","How to make sure it never goes unnoticed",[153,157,160,164],{"icon":154,"title":155,"description":156},"heroicons:document-magnifying-glass","robots.txt change detection","Sitewatch fetches your robots.txt on every check and diffs it. The moment a \"Disallow: /\" appears — or your whole file changes — you get an alert with the exact diff.",{"icon":26,"title":158,"description":159},"Noindex detection","Monitoring also watches for noindex meta tags and X-Robots-Tag headers on your real pages, catching the version of this failure that robots.txt audits miss entirely.",{"icon":161,"title":162,"description":163},"heroicons:rocket-launch","Post-deploy checks","Most deindexing happens right after a deploy. A check triggered after each ship confirms your crawl directives survived the release.",{"icon":165,"title":166,"description":167},"heroicons:map","Sitemap monitoring alongside it","A blocked site and a broken sitemap often ship together in the same bad migration. Pair robots.txt monitoring with sitemap monitoring to cover the whole crawl path.",{"id":169,"tocLabel":170,"type":171,"heading":172,"items":173},"faq","FAQ","faq-accordion","Common questions",[174,177,180,183],{"question":175,"answer":176},"Does \"Disallow: /\" remove my site from Google immediately?","Not instantly, but quickly. Google stops crawling blocked URLs right away and stops refreshing them in the index. Over the following days to weeks, pages drop out of results. The danger is the delay — by the time traffic visibly falls, the block has often been live for a while.",{"question":178,"answer":179},"What's the difference between robots.txt Disallow and noindex?","Disallow in robots.txt tells crawlers not to fetch a URL. Noindex (a meta tag or X-Robots-Tag header) tells Google not to keep a fetched page in the index. Confusingly, if you block a page with robots.txt, Google may not even see your noindex. For full removal, noindex without blocking the URL is usually the correct approach — but for an accidental block, you simply remove both.",{"question":181,"answer":182},"Why didn't my uptime monitor catch this?","Because nothing was down. The server returned 200 OK, pages rendered, assets loaded. Uptime monitoring only answers \"is it responding?\" — not \"is Google allowed to index it?\" Detecting a crawl-blocking change requires fetching and diffing robots.txt and inspecting page directives, which is what website monitoring does.",{"question":184,"answer":185},"How fast can I recover after fixing it?","The fix is minutes. Recovery is not. Once you remove the block and request re-crawling, Google has to recrawl and re-evaluate every page, then rebuild rankings. Light cases recover in days; a site that was blocked for weeks can take a month or more to fully bounce back. This is why catching it early matters so much.",{"id":187,"tocLabel":188,"type":189,"eyebrow":190,"heading":191,"links":192},"related","Related","related-links-grid","Keep reading","Related resources",[193,196,200,204,209],{"label":194,"href":36,"description":195,"icon":154},"Robots.txt Monitoring","Get alerted the moment your robots.txt changes.",{"label":197,"href":198,"description":199,"icon":165},"Sitemap Monitoring","/features/sitemap-monitoring","Catch broken and missing sitemap URLs.",{"label":201,"href":202,"description":203,"icon":32},"Website Monitoring","/website-monitoring","How Sitewatch catches \"up but broken\" failures.",{"label":205,"href":206,"description":207,"icon":208},"Website Broken After Update","/blog/website-broken-after-update","When a deploy quietly breaks production.","heroicons:document-text",{"label":210,"href":39,"description":211,"icon":212},"Is My Website Down?","Check any URL right now.","heroicons:bolt",{"id":214,"tocLabel":215,"type":216,"heading":217,"subtext":218,"primaryLabel":219,"primaryHref":220},"cta","Get started","cta-strip","Know the moment your site gets blocked","Free plan available. Continuous robots.txt and noindex monitoring — so a bad deploy can't quietly deindex you.","Start free monitoring","https://app.getsitewatch.com",{"headline":22,"datePublished":222,"dateModified":222,"authorName":223},"2026-06-07","Sitewatch Team",1780857881295]