Optimizing GSC Bulk Export Retention: Adjusting Partition Expiry and Retention Time for Long-Term Data Storage

The Best Of

Mohamed Natheem

Nov 1, 2024, 7:28 AM

Is GSC bulk export is partitioned by default? It says the partition expiry is 60 days. But this way it kinda fails the purpose of using BQ to hold the data for long time.

Can I just change the retention time to none?

Forwarded thread from another channel:

Mohamed Natheem

Nov 1, 2024, 7:28 AM

I found this query somewhere to use:

ALTER TABLE `project.dataset.table`

SET OPTIONS (

partition_expiration_days = <number_of_days>

);

Samantha Torres

Nov 1, 2024, 7:39 AM

You can also set it to NULL - but as John said, storage can incur costs. I'd recommend setting up budget alerts so you're not taken by surprise when a bill comes your way

Mohamed Natheem

Nov 1, 2024, 7:40 AM

Got it. Thank you both.

Mohamed Natheem

Nov 1, 2024, 7:42 AM

That’s the default behaviour of bulk export or is this something new?

Samantha Torres

Nov 1, 2024, 7:45 AM

Default

Nico Brooks

Nov 1, 2024, 10:11 AM

@noahlearner built <https://theseocommunity.com/blog/2024/09/02/skunkworks-google-search-console-bulk-export-and-ai/#calculator|this simple cost calculator> to help you get a sense how how costs will grow over time. And scroll up to the top of the page and check out the free tool and Looker Studio template he built - it's pretty sweet.

Dale McGeorge

Nov 1, 2024, 4:16 PM

The 60 day partition expiry is ONLY the default for <https://cloud.google.com/bigquery/docs/sandbox#limitations|BigQuery sandbox>. i.e. when you have not set up Billing in GCP. This is not specific to the GSC export and it is just the way Google limits costs when you are not paying.

Enable the BigQuery sandbox | Google Cloud

Use the BigQuery sandbox, learn about its limitations, and upgrade and troubleshoot the BigQuery sandbox.

Noah

Nov 1, 2024, 4:22 PM

Yeah having billing turned on was the first prerequisite in the setup I shared.

Samantha Torres

Nov 1, 2024, 5:07 PM

The most recent project I did, billing was definitely on and I still had the partition expiration. Though I think it was for longer than 90, but still not as long as I want.

Noah

Nov 1, 2024, 6:38 PM

very interesting

Mohamed Natheem

Nov 2, 2024, 10:45 PM

the billing was added as Google prompted to add billing when I turned on the bulk export. So, yes even when the billing is added, I still see the 60 days expiry date

Dale McGeorge

Nov 3, 2024, 4:04 PM

Was the card added before or after the export was started?

Mohamed Natheem

Nov 3, 2024, 8:19 PM

It was before getting started. When I turned on bulk export on GSC, it prompted me to add billing.

Dale McGeorge

Nov 3, 2024, 8:27 PM

That sounds like the link to BigQuery was created before billing was active, which means that adding the billing after the table was created won't have any effect. I would expect that if you canceled the export and started it again you wont have the 60 day limitation.