robots.txt blocks Google

What Disallow: / does, and what it does not

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. Rules are grouped under User-agent lines, and a crawler follows the most specific group that names it, falling back to the User-agent: * group. Disallow: / matches every path, so a group containing it, and no Allow line, closes the whole site to that crawler.

Blocking crawling is not the same as removing a page from search. Google can still list a blocked URL it learns about from links, but without a title or description it could read, so the result looks broken. New pages are not discovered, updated content is never fetched, and other crawlers that respect the file see nothing either. A blocked crawler also cannot read a noindex tag on the page, so robots.txt is the wrong tool for keeping pages out of results.

How a staging file reaches production

Staging robots.txt files are meant to say Disallow: /. The problem is that the file lives in the same repository, theme or settings panel as everything else, so it goes live with the launch. In Next.js, a static app/robots.txt or public/robots.txt is deployed as written, and an app/robots.ts that picks its rules from an environment variable serves the blocking version wherever that variable has the wrong value.

On Webflow, the robots.txt field in the site's SEO settings applies to the custom domain, so a Disallow: / pasted there during the build blocks the live site. The separate setting that disables indexing of the webflow.io subdomain only affects the webflow.io address. On WordPress, a physical robots.txt file in the web root takes the place of the virtual one WordPress generates, and SEO plugins such as Yoast and Rank Math include a robots.txt editor. Recent WordPress versions handle the "Discourage search engines" checkbox with a robots meta tag rather than robots.txt, which is covered in the noindex guide.

How Skewcheck detects it

Skewcheck fetches /robots.txt from the host the audited page finally loads from, after redirects, and reads up to the first 64 KB. If the file does not answer 200, or cannot be fetched at all, the check warns with low severity and removes 2 points from the audit score, which starts at 100.

It then groups the lines under their User-agent lines and looks for a group that disallows every path, written as Disallow: /, Disallow: /* or Disallow: /*$, with no non-empty Allow line in the same group. A group that also allows something is not treated as blocking, because Allow can reopen paths. A Googlebot group overrides the * group for Google, so a file that shuts out every crawler but gives Googlebot its own group that does not block passes. Groups for other crawlers, such as Bingbot or Googlebot-Image, do not count on their own.

On a live host, a file that blocks every crawler or Google fails with high severity, removes 10 points and caps the score at 49, and the detail says which one is blocked. On a host Skewcheck treats as staging, such as staging.example.com, a .webflow.io address, a WP Engine environment domain, or a Vercel preview URL containing -git-, it warns with medium severity for 3 points instead, because blocking is right there. A file that blocks nothing site-wide passes. Skewcheck does not test narrower rules, so a Disallow: /services/ that hides the page you audited still passes.

curl -sL https://example.com/robots.txt

Fixing it on your platform

The smallest valid fix is a User-agent: * group with Allow: /, or with an empty Disallow: line, which means the same thing. Keep any narrower rules you need, such as cart or account paths, and add a Sitemap line with the absolute URL of your sitemap. On Webflow, edit the robots.txt field in the SEO settings and republish. On WordPress, delete or edit the physical file, or the plugin's robots.txt editor, whichever one is serving the live file.

In Next.js, generate the file from app/robots.ts so the rules follow the deployment. The version below allows crawling only when the deployment is production, which keeps every preview blocked by default. The catch is that VERCEL_ENV is only set on Vercel. On any other host this exact code blocks the live site, so swap in a variable you control and check the output after the first deploy.

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV !== "production") {
    return { rules: { userAgent: "*", disallow: "/" } };
  }
  return {
    rules: { userAgent: "*", allow: "/", disallow: ["/cart/", "/account/"] },
    sitemap: "https://example.com/sitemap.xml",
  };
}

Edge cases

Each host needs its own file. www.example.com and shop.example.com each read the robots.txt at their own root, so audit each one you launch. Google generally caches robots.txt for up to a day, so a fix can take that long to be noticed, and Search Console has a robots.txt report that shows the version Google last fetched.

A missing file, answering 404, means no restrictions to Google, which is why Skewcheck only warns about it. Server errors are worse: Google treats a robots.txt that answers with a 5xx status as a temporary block on the whole site. If you want to keep a page out of results, leave it crawlable and use noindex instead, so crawlers can actually read the instruction.

Before and after

# Copied from staging
User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml