Cookie Consent by Free Privacy Policy Generator

How Google Treats Asterisks in Robots.txt Parameters

Go to the Best Of the SEO Community.

Jeremy Rivera
Jeremy Rivera
Dec 19, 2024, 4:05 PM
Uh technical SEO friends -
 
Disallow: /*?*hs_preview=*
Disallow: /*?*hsCacheBuster=*
 
Would Google discard everything after the * and treat this as essentially
 
Disallow: /
Forwarded thread from another channel:
Kyle Faber
Kyle Faber
Dec 19, 2024, 4:08 PM
Nope. Only if the params are present.
Valentin Pletzer
Valentin Pletzer
Dec 20, 2024, 12:40 AM
In Bing Webmaster Tools you can “edit” (those are not real edits) the robots.txt and check against URLs to see what would happen: <https://www.bing.com/webmasters/robotstxttester>
Bing Webmaster Tools
Sign in or signup for Bing Webmaster Tools and improve your site’s performance in search. Get access to free reports, tools and resources.
Tory Gray
Tory Gray
Dec 20, 2024, 6:52 AM
Seconding (thirding?!) the above. Google is *specific* - that’ll only block those specific param urls from getting crawled (not indexed) - whatever order the params are in
John Mueller
John Mueller
Dec 24, 2024, 7:05 AM
I mentioned this somewhere else, I think ???? , but I've seen this when sites do a cookie / bot / login-redirect. The actual / final URL doesn't have the parameters, but one of the intermediate URLs does use them -- then these robots.txt directives will block the whole set.
 
If any URL within a redirect chain is blocked, the whole chain is blocked. Search Console focuses on the "final" URL with inspect URL. I *think* you can trick it by looking at the rendered HTML / console output there. You can also try it in an incognito window with the Googlebot user-agent.

Channels

  • agency (3)
  • ai-for-seo (11)
  • ai-llms (24)
  • analytics (13)
  • biz-of-seo (4)
  • career (4)
  • content (15)
  • cro (4)
  • data-viz-reporting (4)
  • datascience (2)
  • ecommerce (3)
  • events (1)
  • freelance (2)
  • general-seo (21)
  • in-house (2)
  • local-seo (12)
  • local-seo,biz-of-seo (1)
  • lounge (1)
  • reading-list-resources (3)
  • search-console (11)
  • shameless-plug (1)
  • technical-seo (37)
  • tools (1)


    • Join The Community

    The SEO Community is a free organization, with private Slack group and events, for those interested in search and AI optimization to learn, discuss, and meet others in the industry.

    Powered by MODX

    Resources

    The Community Story SEO Articles SEO Jobs SEO Conferences Top SEO Discussions Newsletter Free Tools Learn Claude Code

    The SEO Community © 2026

    Shop Merch Code of Conduct Partnerships Sitemap Update cookies preferences Terms of Service Privacy Policy