s3 to big query data transfer | C2C Community

s3 to big query data transfer

  • 14 September 2021
  • 2 replies

when I use data transfer from S3 to big query it's stores data as duplicate as I try to store last 7 days data. But if I use same transfer from sa360 to big query it updates existing records and appends new one. is there a way i can enable same feature for S3 transfer. or any workaround any sample pub sub model that can remove duplicates.

2 replies

Userlevel 7
Badge +25

@Alfons any advice here?

Userlevel 5
Badge +5

Hi @Indraykk,

I would have a look into Bigquery documentation about S3 transfer. I can see in the docs that migrations from S3 always uses the parameter WRITE_APPEND which does append to the table but does not update the current records and therefore could lead to the duplicate error you are describing.

I don't know how to import and update data in bigquery, the documentation does not say anything about that possibility.

I hope we have some data engineer here at the C2C Community that can help us.