Data ingestion using CI / CD pipelines from SalesForce to BigQuery | C2C Community

Data ingestion using CI / CD pipelines from SalesForce to BigQuery

  • 9 April 2021
  • 4 replies

Userlevel 1

Hi Community,


I wondering if someone have the experience implementing Data CI/CD pipelines from SalesForce and other DW flavors to BigQuery, managing versioning, deltas and dynamics pivoting tables, schema updates using GCP tools services (including Dataform).


Thanks in advance 

4 replies

Userlevel 7

@jcescalante hoping you might have some insight to help out @Alberto Vicente on this. 

Alberto, here is the Talk JC did with C2C in December (this is a link from our old blog, our vendor is fixing a bug with this post on our new community platform) 

Userlevel 6
Badge +12

@ilias @Ezekias can lean in to help @Alberto Vicente on the CI/CD and BigQuery side of things here. Should be a number of solid folks to connect him with on the partner side as well @Marcy.Young who play in this lane specifically. 

@yuval is this something your crew works on? 

Userlevel 3
Badge +1

@Alberto Vicente Can you give more details on the CI / CD environment you want to set up?
I will be happy to help you.

Userlevel 1

Thanks for your responses, so sorry for the response delays, here’s the picture:

  • Three different data sources - Salesforce, Teradata and Oracle
  • GCP Data Pipelines for ETL/ELT- Data Fusion and Composer
  • CI/CD - for I.a.a.C Terraform and Cloud Source Repository and Cloud Build for code

Scenario: CI/CD Data as a code and Data ingestion pipelines handling for new dataset/tables/authorized view, existing dataset/tables & its daily deltas, plus dynamics pivoting tables for existing data sources schemas update, makes sense?