Noindex left on after launch

What noindex tells a search engine

Search engines look in two places for indexing instructions. The first is a meta tag in the HTML head, either name="robots" for every crawler or name="googlebot" for Google alone. The second is the X-Robots-Tag HTTP response header, which works on any file type, including PDFs and images. A value of noindex, or none, which is shorthand for noindex plus nofollow, asks for the page to be kept out of search results.

Google applies the directive the next time it crawls the page. A new site that launches with noindex never enters the index at all. An existing site that picks it up during a redesign drops out page by page as each URL is recrawled. Nothing looks wrong in the browser, the page loads normally for visitors, and that is why the problem can sit unnoticed until someone asks why the business no longer appears when people search for its name.

Where it comes from

Hiding a staging copy from search engines is good practice, and most platforms make it one switch. WordPress has a checkbox labeled "Discourage search engines from indexing this site" under Settings, Reading, and recent versions turn it into a robots meta tag with noindex, nofollow on every page. Framer page settings include a control for whether a page shows up in search engines. Webflow can turn off indexing of the webflow.io subdomain, and its page settings can keep individual pages out of search. In Next.js, the robots field of the metadata export writes the meta tag, and Vercel sends X-Robots-Tag: noindex on preview deployments by default.

The trouble starts when the switch outlives the build. A WordPress site built on a temporary domain moves to the real one with its database, checkbox included. A Webflow or Framer page hidden during client review is never switched back. A Next.js layout decides robots from an environment variable that is only set on one host, or an agency starter template carries a hard-coded noindex from the last project. The client often finds it weeks later in the page indexing report in Google Search Console, where the affected URLs are listed as excluded by the noindex tag.

How Skewcheck detects it

Skewcheck reads every meta tag named robots or googlebot in the audited page, combining them the way crawlers do, plus the X-Robots-Tag header on the response. A noindex or none token in any of those tags counts. In the header, the directive counts when it has no user-agent prefix or is prefixed with googlebot or *, so X-Robots-Tag: googlebot: noindex is caught and X-Robots-Tag: bingbot: noindex is not. Meta tags addressed to other crawlers are ignored.

On a live host, a match fails the check with high severity and removes 10 points from the audit score, which starts at 100, and caps the score at 49 however clean the rest of the page is. Only the HTTPS check, at 12 points, costs more. The detail line says whether the meta tag, the header, or both carried the directive. With no match, the check passes.

On a host Skewcheck recognizes as staging, the same finding is a medium-severity warning worth 3 points, because noindex is correct there. Staging hosts are localhost and addresses ending in .localhost, .local, .test, .webflow.io, .wpengine.com or .wpenginepowered.com; Vercel addresses containing -git-; Netlify addresses containing a double hyphen; Cloudflare Pages addresses with a subdomain in front of project.pages.dev; and any host whose subdomain part contains the word staging, stage, stg, dev, preview, uat or sandbox, such as staging.example.com or dev-www.example.com. Everything else counts as live, including plain vercel.app, netlify.app and framer.app sites, hash-style Vercel deployment URLs, and test.example.com.

When the page answers with an error (HTTP 400 or above), Skewcheck does not read noindex at all: error pages usually carry it by default, and the HTTP status check reports the real problem. When a bot-protection service such as Cloudflare answers with a challenge page instead of the site, Skewcheck reports that it was turned away and gives no score, because the challenge page's noindex is not the site's.

curl -sI https://example.com | grep -i x-robots-tag
curl -s https://example.com | grep -ioE '<meta[^>]+name="?(robots|googlebot)"?[^>]*>'

Removing it on your platform

Remove the directive rather than piling an override on top of it. Indexing is the default, so a page with no robots meta tag and no X-Robots-Tag header is indexable. In WordPress, untick the search engine visibility checkbox under Settings, Reading, then confirm an SEO plugin is not adding noindex to the page type you audited. In Webflow, check the page's SEO settings and any custom code in the site-wide or page head. In Framer, open each affected page's settings and allow it in search engines again.

In Next.js, look for robots in every layout and page metadata export, and for X-Robots-Tag in next.config, vercel.json, proxy (formerly middleware) and server config such as nginx or a Netlify _headers file. If you gate indexing on an environment variable, key it to production. VERCEL_ENV only exists on Vercel, so if the site moves to another host, the condition below quietly flips every page to noindex unless you set that variable yourself.

import type { Metadata } from "next";

// Index the production deployment only. Preview deployments stay hidden.
const isProduction = process.env.VERCEL_ENV === "production";

export const metadata: Metadata = {
  robots: isProduction
    ? { index: true, follow: true }
    : { index: false, follow: false },
};

After the fix

Search engines only notice the change when they recrawl. Once the live page passes, use the URL Inspection tool in Google Search Console to test the live URL and request indexing for the most important pages, and submit a sitemap so the rest are found. Do not block the same pages in robots.txt while you wait, because a crawler that is not allowed to fetch a page never sees that the noindex has gone.

Check the response visitors actually receive. A CDN or page cache can keep serving HTML with the old meta tag after you change the setting, so purge it. Skewcheck audits one URL at a time, and per-page settings mean the home page can pass while a services page still carries noindex, so audit one page of each template. And keep noindex on staging: the goal is a staging copy that stays hidden and a live site that does not.

Before and after

<!-- Served at https://example.com/cakes/wedding, with X-Robots-Tag: noindex -->
<head>
  <title>Wedding Cakes | Example Bakery</title>
  <meta name="robots" content="noindex, nofollow">
  <meta name="googlebot" content="noindex">
</head>

<!-- Served at https://example.com/cakes/wedding, no X-Robots-Tag header -->
<head>
  <title>Wedding Cakes | Example Bakery</title>
</head>