How to Prevent Staging Site from Getting Indexed in Robots.txt

Go to the Best Of the SEO Community.

Luke Mc Leod

Dec 11, 2023, 2:45 AM

Hello,

Quick Q regarding Robots.txt

We have a site “<http://www.hotel.com|www.hotel.com>”

And a staging site “<http://staging.hotel.com|staging.hotel.com>”

What’s the best way to ensure the staging site doesn’t not get indexed.

I was thinking it would be in robots. But I’ve never done a subdomain through this, only a sub folder

Forwarded thread from another channel:

Kim Higdon

Dec 11, 2023, 2:52 AM

WordPress has a box you can check to tell search engine not to scan your site. Go to settings, then Reading to turn it on and off.

shariq.ehsan

Dec 11, 2023, 2:53 AM

Use meta robots tag i.eon each page of staging site.

Olly Torrance

Dec 11, 2023, 3:01 AM

Add a password protection to the staging environment

John Mueller

Dec 11, 2023, 3:28 AM

I'd block staging completely - password or IP level. Using robots.txt means you almost certainly will copy the staging robots.txt to your production site and robot everything (a rite of passage). The same with noindex. Rel-canonical is a bad idea, since it's a signal and doesn't completely prevent staging sites from indexing. Blocking on the server level (password/IP) also means your competitors can't poke around.

Luke Mc Leod

Dec 11, 2023, 3:40 AM

Okay, thank you all.

Regarding password protection. Is this as simple as placing the Wordpress password protection in the site?

Or something a bit more high level

Jono Alderson

Dec 11, 2023, 5:36 AM

There are a range of plugins that'll be a wp site into a "coming soon" or "private" mode that'll prevent logged out access, and noindex everything etc. But then, you're going to have differences between that and the live env, and testing will be a pain. Best off using server-level authentication to require a username and password to access the site.

Jono Alderson

Dec 11, 2023, 5:46 AM

<https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication> - that should be trivial for your developers to implement

Eric Wu

Dec 11, 2023, 5:56 PM

Just to add to the info, I’ve got a full post on StackOverflow that seems have been helpful over the years for folks:

<https://stackoverflow.com/questions/12197979/how-to-prevent-staging-to-be-indexed-in-search-engines/12219987#12219987>

how to prevent staging to be indexed in search engines

I would like my staging web sites to no being indexed by search engines (Google as first).

I have heard Wordpress is good at doing this but I would like to be technology agnostic.

Does the robots...

Luke Mc Leod

Dec 12, 2023, 2:35 AM

Thanks so much Eric, I’ll have a look at this now.

I had one last question.

If I was to enter the url <https://staging.hotel.com/*|https://staging.hotel.com/*> would this remove all the <http://www.hotel.com|www.hotel.com> content from search also?

Dominic Woodman

Dec 12, 2023, 5:09 AM

if you're doing robots.txt (as it looks like from those patterns you've mentioned) you'd want different robots.txt at the root of each subdomain.

But +1 on everyone else suggesting also no-indexing or blocking IPs etc. if you're comfortable doing that.

Channels

agency (3)

ai-for-seo (11)

ai-llms (24)

analytics (13)

biz-of-seo (4)

career (4)

content (15)

cro (4)

data-viz-reporting (4)

datascience (2)

ecommerce (3)

events (1)

freelance (2)

general-seo (21)

in-house (2)

local-seo (12)

local-seo,biz-of-seo (1)

lounge (1)

reading-list-resources (3)

search-console (11)

shameless-plug (1)

technical-seo (37)

tools (1)

Join The Community

The SEO Community is a free organization, with private Slack group and events, for those interested in search and AI optimization to learn, discuss, and meet others in the industry.

How to Prevent Staging Site from Getting Indexed in Robots.txt

Channels

Join The Community

Resources

The SEO Community © 2026

How to Prevent Staging Site from Getting Indexed in Robots.txt

Channels

Sign up to get the newsletter

Join The Community

Resources

The SEO Community © 2026