Cookie Consent by Free Privacy Policy Generator

The Best Of

Go to the Best Of the SEO Community.

Noah
Noah
Jan 18, 2025, 9:07 PM
Forwarded from another channel:
Forwarded thread from another channel:
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:21 PM
for anyone doing rank tracking, there was a major update Google rolled out globally last night to try and block crawling. This is a ???? to dig into the changes
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:23 PM
where you could use a simple cURL request before to get a SERP, now it returns a bunch of javascript with this message (html included)
> If you're having trouble accessing Google Search, pleaseclick here, or send.
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:32 PM
if you click on the link in the html, it does a redirect that adds an `sg_ss` query parameter
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:33 PM
here's a curl request with that parameter included
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:36 PM
that curl request succeeds with every header removed if the `sg_ss` param is included, and fails if it is removed
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:38 PM
I tried decoding it using Google's standard base64+protobuf encoding that Google uses for most other things ( from @Valentin Pletzer), but it's not valid base64
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:39 PM
I'm still digging in, just thought I'd share this in case anyone else is working on it
Derek Perkins
Derek Perkins
Jan 16, 2025, 12:39 PM
this is the reason that many (most?) rank trackers are having issues today
Ryan Jones
Ryan Jones
Jan 16, 2025, 12:53 PM
this explains why none of the 3rd party APIs i get search results from are working today, and why my own tools are not working.
Valentin Pletzer
Valentin Pletzer
Jan 16, 2025, 1:06 PM
I’m guessing not many tool provider would be comfortable sharing if they have problems. If Google now starts to crack down on scrapers this would be huge. I guess @Barry Schwartz could ask around.
Derek Perkins
Derek Perkins
Jan 16, 2025, 1:08 PM
that's why I'm trying to be transparent, it's useful for everyone
Derek Perkins
Derek Perkins
Jan 16, 2025, 1:17 PM
crawling with Chrome should work, that just increases scraping costs 1-2 orders of magnitude
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 1:21 PM
Well one tool provider did
We've observed a decrease (approximately 2 percentage points) in AIO detection rates through Ziptie's tracking system. It seems Google is blocking AIO checkers in a smart way, far beyond traditional captchas. We are looking for ways to improve the AIO detection rate.
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 1:24 PM
I’m asking other tool providers. Will keep you posted
Derek Perkins
Derek Perkins
Jan 16, 2025, 1:29 PM
VALUE SERP's Status Page - Google Web Search Service Disruption.
Derek Perkins
Derek Perkins
Jan 16, 2025, 1:34 PM
I was hoping to reverse engineer that query param to see if we could skip that redirect page
Ryan Jones
Ryan Jones
Jan 16, 2025, 1:35 PM
and sadly valueserp is used in my tool SERPrecon - so we're down ????
Ryan Jones
Ryan Jones
Jan 16, 2025, 1:57 PM
@Derek Perkins @Barry Schwartz found any other tools impacted? all the major ones tell me "we aren't aware of any impact"
Derek Perkins
Derek Perkins
Jan 16, 2025, 2:06 PM
yes, but not at liberty to share names. I'm skeptical of ones saying they don't have issues
Derek Perkins
Derek Perkins
Jan 16, 2025, 2:12 PM
I just did a manual check with Accuranker and got new data, and I _think_ DataForSEO is working
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:12 PM
dataforseo seems to be working. at least in their playground
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:13 PM
SpaceSERP claims they are up
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:15 PM
but their playground does not work
Derek Perkins
Derek Perkins
Jan 16, 2025, 2:15 PM
I just ran searches in both their playgrounds too, same results
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:17 PM
i wish scrapingrobot had a playground i can test. there's no documentation for their SERP API but i know they have one
Derek Perkins
Derek Perkins
Jan 16, 2025, 2:18 PM
DataForSEO already had some more expensive options to get pixel data and whatnot, that likely used Chrome, so I wonder if they're routing all requests through that infrastructure for now
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:18 PM
can confirm BrightEdge instant is giving me results. so is Moz rank tracker. no idea if they're cached though.
Ryan Jones
Ryan Jones
Jan 16, 2025, 2:19 PM
not aware that semrush/ahrefs have real time SERP tools.
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 2:59 PM
I emailed several asking for statements
Ray Grieselhuber
Ray Grieselhuber
Jan 16, 2025, 5:29 PM
We're not having any issues, we do mostly JS though when scraping
Ray Grieselhuber
Ray Grieselhuber
Jan 16, 2025, 5:31 PM
Transparency is definitely the best approach here, agreed with @Derek Perkins. We're just not seeing much impact yet on our end.
Derek Perkins
Derek Perkins
Jan 16, 2025, 5:38 PM
are you scraping with js enabled?
Derek Perkins
Derek Perkins
Jan 16, 2025, 5:42 PM
I think that's the major differentiator for this, Google started requiring js if certain conditions aren't met (cookies and/or query params)
Ray Grieselhuber
Ray Grieselhuber
Jan 16, 2025, 5:42 PM
Yup that's what it looks like
Patrick Stox
Patrick Stox
Jan 16, 2025, 7:45 PM
Commenting to follow, but I'm not seeing any issues in the data, or customer complaints from Ahrefs users. I just checked and we have recent SERPs and can pull live SERPs.
Patrick Stox
Patrick Stox
Jan 16, 2025, 7:50 PM
If we had any issue, we'd usually have hundreds of complaints by now.
Derek Perkins
Derek Perkins
Jan 16, 2025, 10:03 PM
Fresh in: Google starts intensifying its anti-scraping measures, introducing stricter protections such as IP blocking and CAPTCHAs. Popular SEO tools like… | 32 comments on LinkedIn
Natalia Witczyk on LinkedIn: Fresh in: Google starts intensifying its anti-scraping measures… | 32 comments
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 10:51 PM
@Patrick Stox can I quote you?
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 11:24 PM
Actually I see you commented publicly on LinkedIn. That works for me
Derek Perkins
Derek Perkins
Jan 16, 2025, 11:35 PM
I wonder if the lower volatility in those charts is because those tools haven't gotten as many SERPs recently...
Seppo Puusa
Seppo Puusa
Jan 16, 2025, 11:36 PM
I also think the change in volatility metrics is partly due to data issues
Barry Schwartz
Barry Schwartz
Jan 16, 2025, 11:37 PM
Yes. That is what I am guessing. But I’m wrong about 91% of the time.
Seppo Puusa
Seppo Puusa
Jan 16, 2025, 11:37 PM
@Barry Schwartz I should steal that line and use it with every client recommendation from now on
Barry Schwartz
Barry Schwartz
Jan 17, 2025, 4:15 AM
On Google blocking rank tracking tools - @RangerShay from @Similarweb sent me the following statement which has been added to the story at
X (formerly Twitter): Barry Schwartz (@rustybrick) on X
Seppo Puusa
Seppo Puusa
Jan 17, 2025, 4:16 AM
Data for SEO SERP API timeouts at the moment.
Derek Perkins
Derek Perkins
Jan 17, 2025, 6:56 AM
this is a good perspective I think
@rustybrick PS: This has nothing to do with rank trackers specifically and all to do with LLM training data scrapers.
Ray Grieselhuber
Ray Grieselhuber
Jan 17, 2025, 7:09 AM
We ended up getting blocked last night around 9pm ET, but we're starting to look good this morning.
@Derek Perkins I completely agree with that post. I started noticing Google results (instead of Bing) in ChatGPT results and I think Google was not amused.
Derek Perkins
Derek Perkins
Jan 17, 2025, 9:44 PM
This outage made it to TechCrunch
Google says it has begun requiring users to turn on JavaScript in order to use Google Search, possibly in an effort to block certain SEO tools.
TechCrunch: Google begins requiring JavaScript for Google Search | TechCrunch

Our Values

What we believe in

Building friendships

Kindness

Giving

Elevating others

Creating Signal

Discussing ideas respectfully

What has no home here

Diminishing others

Gatekeeping

Taking without giving back

Spamming others

Arguing

Selling links and guest posts


Sign up for our Newsletter

Join our mailing list for updates

By signing up, you agree to our Privacy Policy and Terms of Service. We may send you occasional newsletters and promotional emails about our products and services. You can opt-out at any time.

Apply now to join our amazing community.

Powered by MODXModx Logo
the blazing fast + secure open source CMS.