Cookie Consent by Free Privacy Policy Generator

The Best Of

Go to the Best Of the SEO Community.

Luke Mc Leod
Luke Mc Leod
Dec 11, 2023, 2:45 AM
Forwarded from another channel:
Hello,
Quick Q regarding Robots.txt
We have a site “
And a staging site “
What’s the best way to ensure the staging site doesn’t not get indexed.
I was thinking it would be in robots. But I’ve never done a subdomain through this, only a sub folder
Forwarded thread from another channel:
Kim Higdon
Kim Higdon
Dec 11, 2023, 2:52 AM
WordPress has a box you can check to tell search engine not to scan your site. Go to settings, then Reading to turn it on and off.
shariq.ehsan
shariq.ehsan
Dec 11, 2023, 2:53 AM
Use meta robots tag i.e on each page of staging site.
Olly Torrance
Olly Torrance
Dec 11, 2023, 3:01 AM
Add a password protection to the staging environment
John Mueller
John Mueller
Dec 11, 2023, 3:28 AM
I'd block staging completely - password or IP level. Using robots.txt means you almost certainly will copy the staging robots.txt to your production site and robot everything (a rite of passage). The same with noindex. Rel-canonical is a bad idea, since it's a signal and doesn't completely prevent staging sites from indexing. Blocking on the server level (password/IP) also means your competitors can't poke around.
Luke Mc Leod
Luke Mc Leod
Dec 11, 2023, 3:40 AM
Okay, thank you all.
Regarding password protection. Is this as simple as placing the Wordpress password protection in the site?
Or something a bit more high level
Jono Alderson
Jono Alderson
Dec 11, 2023, 5:36 AM
There are a range of plugins that'll be a wp site into a "coming soon" or "private" mode that'll prevent logged out access, and noindex everything etc. But then, you're going to have differences between that and the live env, and testing will be a pain. Best off using server-level authentication to require a username and password to access the site.
Jono Alderson
Jono Alderson
Dec 11, 2023, 5:46 AM
- that should be trivial for your developers to implement
Eric Wu
Eric Wu
Dec 11, 2023, 5:56 PM
Just to add to the info, I’ve got a full post on StackOverflow that seems have been helpful over the years for folks:
I would like my staging web sites to no being indexed by search engines (Google as first).
I have heard Wordpress is good at doing this but I would like to be technology agnostic.
Does the robots...
Luke Mc Leod
Luke Mc Leod
Dec 12, 2023, 2:35 AM
Thanks so much Eric, I’ll have a look at this now.
I had one last question.
If I was to enter the url would this remove all the content from search also?
Dominic Woodman
Dominic Woodman
Dec 12, 2023, 5:09 AM
if you're doing robots.txt (as it looks like from those patterns you've mentioned) you'd want different robots.txt at the root of each subdomain.
But +1 on everyone else suggesting also no-indexing or blocking IPs etc. if you're comfortable doing that.

Our Values

What we believe in

Building friendships

Kindness

Giving

Elevating others

Creating Signal

Treating each other with respect

What has no home here

Diminishing others

Gatekeeping

Taking without giving back

Spamming others

Arguing

Selling links and guest posts


Sign up for our Newsletter

Join our mailing list for updates

By signing up, you agree to our Privacy Policy and Terms of Service. We may send you occasional newsletters and promotional emails about our products and services. You can opt-out at any time.

Apply now to join our amazing community.

Powered by MODXModx Logo
the blazing fast + secure open source CMS.