when I use data transfer from S3 to big query it's stores data as duplicate as I try to store last 7 days data. But if I use same transfer from sa360 to big query it updates existing records and appends new one. is there a way i can enable same feature for S3 transfer. or any workaround any sample pub sub model that can remove duplicates.
Hi
I would have a look into Bigquery documentation about S3 transfer. I can see in the docs that migrations from S3 always uses the parameter WRITE_APPEND which does append to the table but does not update the current records and therefore could lead to the duplicate error you are describing.
I don't know how to import and update data in bigquery, the documentation does not say anything about that possibility.
I hope we have some data engineer here at the C2C Community that can help us.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.