Dataflow and Bigquery pricing | C2C Community
Solved

Dataflow and Bigquery pricing

  • 9 December 2021
  • 3 replies
  • 132 views

Userlevel 1

Hi all, 

I am a little bit confused and I would like to have your clarification. 
About dataflow pricing
When working on dataflow, we have to specify a GCS bucket  for temp_location. So is the use of this bucket subject to a google cloud storage pricing?

About the Bigquery pricing
When I do bacth load from dataflow to bigquery, I have to specify a bucket on google cloud storage. Do I also have to pay for the usage of this bucket? And after the load to bigquery, will the data be deleted from this bucket?

icon

Best answer by guillaume blaquiere 9 December 2021, 16:27

View original

3 replies

Userlevel 6
Badge +15

Hello

 

With dataflow you pay the processing, I mean, the CPU and the Memory that your workers use to process the data. You can set the size and the number of worker that you want per job.

All the additional products that you use, Cloud Storage, BigQuery and other, have their own cost and you are charged accordingly with that cost. i.e. in addition to the Dataflow cost

Note that the temporary file aren’t deleted automatically. I recommend you to put a lifecycle on the temp location bucket to not accumulate too much backlogs.

Userlevel 1

@guillaume blaquiere Thank you for your response! So the temporary file used by dataflow on cloud storage is calculated by the cloud storage pricing.

Does the custom_gcs_temp_location required on the file_loads method to upload a file to bigquery follow this same rule?

Userlevel 6
Badge +15

Yes, same rule, same price, same cleanup to perform.

Reply