Blocked due to unauthorized request (401)

What’s actually happening

Googlebot requested the URL and got back HTTP 401 Unauthorized, so it couldn't read or index the page. This shows up the moment a section of the site sits behind HTTP Basic Auth, a staging password, or a login wall that fires before any content loads. Sometimes it's intentional (a staging host that accidentally got submitted) and sometimes a security plugin or WAF started challenging the crawler. Either way Google sees a locked door and leaves.

Common causes

• HTTP Basic/Digest auth on the server or a directory (.htpasswd, nginx auth_basic) covering pages you actually want indexed.
• A staging or pre-production environment that's password-protected but got into the index or a submitted sitemap.
• A security plugin (Wordfence, iThemes) or WAF rule that returns 401 to unrecognized crawlers or under rate pressure.
• An IP allowlist that lets your office through but returns 401 to everyone else, including Googlebot's datacenter IPs.
• A misconfigured auth middleware/redirect that wraps the whole app and 401s any request without a session cookie.

How to fix it

1. Reproduce the 401curl -I https://example.com/page from outside your network. A 401 with a WWW-Authenticate header confirms auth is the blocker. If you only get content when on the office VPN, that's your allowlist locking Googlebot out.
2. Decide: should this be public at all?If it's a staging site that shouldn't be indexed, the fix isn't to open it — it's to remove the URLs from Google. Keep the 401/auth, submit a removal in Search Console, and stop submitting that host in sitemaps. Don't leave staging crawlable.
3. Remove auth from pages that should be publicFor production pages that must rank, drop the auth covering them. Remove the auth_basic block in nginx (or the .htpasswd Require valid-user in Apache) for those paths, or scope the login wall to /admin and /account only — never the whole site.
4. Stop the security plugin/WAF from 401-ing GooglebotIn Wordfence or your WAF, allowlist verified Googlebot by reverse-DNS, not UA string. Check rate-limiting rules — aggressive thresholds can 401 the crawler during a crawl spike.
5. Validate and request indexingAfter curl -I returns 200, run Test Live URL in URL Inspection, then Validate Fix on the issue and Request Indexing for priority pages.