Cookie Consent by Free Privacy Policy Generator

The Best Of

Go to the Best Of the SEO Community.

Noah
Noah
Jan 19, 2025, 11:19 AM
Forwarded from another channel:
Forwarded thread from another channel:
Valentin Pletzer
Valentin Pletzer
Dec 24, 2024, 12:18 PM
How to debug this? I’ve checked url inspect already and there are no issues. I see the same drop in the logfiles and it affects all directories the same. There was/is no change/spike in different status codes (e.g. no 5xx or 4xx). There is also no increase in average response times. Only are life altering drop in crawl requests.
I suspect a CDN bot protection gone rogue but it doesn’t look like it. Anything else I can check?
Ian Cappelletti
Ian Cappelletti
Dec 24, 2024, 1:07 PM
are any of the file type crawl stats reports bucking the trend?
Valentin Pletzer
Valentin Pletzer
Dec 24, 2024, 1:14 PM
Not that I could find. I just went through all again just to be sure. “By response”, “By file type”, “By purpose” and “By Googlebot type” are all going down.
“Host status” shows no issues in the last 90 days
Ian Cappelletti
Ian Cappelletti
Dec 24, 2024, 2:31 PM
correlating with any traffic loss?
RP
RP
Dec 24, 2024, 3:25 PM
I had this a month or so ago,.a 10x drop in requests. It has now balanced back out. Sorry I can't add anything technical but just giving some empathy! Has robots.txt been changed?
Valentin Pletzer
Valentin Pletzer
Dec 24, 2024, 11:20 PM
Vertical is news. Traffic-loss is difficult to quantify since it is always very volatile but I believe the traffic should have been much better. One thing I noticed: the news about the evacuation of the Eiffel tower hasn’t been picked up as fast as I thought it would be. That’s why I checked in the first place.
The robots.txt hasn’t been changed. I hope stuff will balance out.
I don’t think that less crawling is necessarily bad but it hasn’t been that low for years now (maybe never?)
Mika Lepistö
Mika Lepistö
Dec 25, 2024, 1:36 AM
Are you using the indexing API?
Valentin Pletzer
Valentin Pletzer
Dec 25, 2024, 1:50 AM
No. Indexing API is officially only for jobs and live video if I recall correctly. Also: Usually Google crawls and indexes our new articles within a minute or two of publishing it. There was no need.
We do make use of the news sitemap of course.
Valentin Pletzer
Valentin Pletzer
Dec 25, 2024, 1:54 AM
I have to add: Looking at the realtime traffic right now all looks good. So maybe it was just a fluke? Maybe Google did crawl way too much before?
Mika Lepistö
Mika Lepistö
Dec 25, 2024, 2:04 AM
Reason I asked was because they have cracked down (supposedly, I haven't seen it first hand) on abuse of the API and wondered if that might be a factor.
Do you have any log access? If so, are the sitemap and new articles being crawled?
I don't work in that vertical so this is all hypothetical, but I can't see a reason to regularly crawl more than what the actual new content is in true news. I'm wondering if that is a recent change that you're seeing and if logs correlate.
Valentin Pletzer
Valentin Pletzer
Dec 25, 2024, 2:15 AM
I appreciate all the ideas! That way I hope I can zoom into the real reasons.
I do have access to the logs. What I see is that the news sitemap get’s crawled about 90 times and hour currently and it used to be around 200 times an hour.
Looking at the logs and evaluating if there is a change between new and old articles getting picked up is a lot more difficult. The logfiles are huge and there is new content all the time. So I am still trying to figure out how to do that efficiently.
Valentin Pletzer
Valentin Pletzer
Dec 25, 2024, 2:16 AM
Just to give a context on why the crawl stats spooked me. This are the desktop and smartphone stats over the last year and everything I have.
I never really looked at it this way but you can see when Google “stopped” crawling with Desktop
Mika Lepistö
Mika Lepistö
Dec 25, 2024, 7:25 AM
Yeah, I see what you mean and understand the concern with the declining trend. Do you have an idea How much new content there is per hour? Has that changed to support the trend?

Our Values

What we believe in

Building friendships

Kindness

Giving

Elevating others

Creating Signal

Discussing ideas respectfully

What has no home here

Diminishing others

Gatekeeping

Taking without giving back

Spamming others

Arguing

Selling links and guest posts


Sign up for our Newsletter

Join our mailing list for updates

By signing up, you agree to our Privacy Policy and Terms of Service. We may send you occasional newsletters and promotional emails about our products and services. You can opt-out at any time.

Apply now to join our amazing community.

Powered by MODXModx Logo
the blazing fast + secure open source CMS.