C2C Talks: Using Google Cloud’s BigQuery to Move from a 48-Hour Cycle Time to a Mere 7 Minutes | C2C Community

C2C Talks: Using Google Cloud’s BigQuery to Move from a 48-Hour Cycle Time to a Mere 7 Minutes

Categories: Data Analytics C2C Community Spotlight Hybrid and Multicloud Cloud Migration Storage and Data Transfer Databases Session Recording
C2C Talks: Using Google Cloud’s BigQuery to Move from a 48-Hour Cycle Time to a Mere 7 Minutes

Author’s Note: C2C Talks are an opportunity for C2C members to engage through shared experiences and lessons learned. Often there is a short presentation followed by an open discussion to determine best practices and key takeaways.

Juan Carlos Escalante (JC) is a pioneering member of C2C and a vital part of the CTO office at Ipsos. Escalante details how he and his team handled data migration powered by Google Cloud and shares his current challenges, which may not be unlike anything you’re also facing. 

As a global leader in market research, Ipsos has offices in 90 countries and conducts research in more than 150 countries. So, to say its data architecture is challenging barely covers the complexity JC manages each day. 

“Our data architecture on our data pipeline challenges gets complex very quickly, especially for workloads dealing with multiple data sources, and what I describe as hyper-fragmented data delivery requirements,” he said in a recent C2C Talks: Data Migration and Modernization on December 10, 2020.

So, how do they manage a seamless data flow? And how does JC’s data infrastructure landscape look? Hear below.

What was the primary challenge? 
 

Even though the design JC described is popular and widely used in the space, it isn’t without its own set of challenges and siloed data infrastructure rises to the top.

“The resilience of siloed data infrastructure platforms that we see scattered across the company translates to longer cycle times and more friction to pivot and react to changing business requirements,” he said. 

Hear JC explain the full challenge below. What resonates with you? Share it with us!

 

 

How did you use Google Cloud as a solution? 
 

By leveraging Google Cloud, JC and his team have unlocked new opportunities to simplify how different groups come into a data infrastructure platform and serve or solve their specific needs.

“We all have different products and services that we have available within Google Cloud Platform,” he said. “Very quickly, we've been able to test and deploy proofs of concept that have moved rapidly towards production.”

Some examples of the benefits JC and his team have found by using the Google Cloud Platform product, BigQuery include: 

  1. Reduced cycle time or processing time from 48 hours to seven minutes
  2. Data harmony across teams

Hear JC explain how BigQuery helped reach these successful milestones.

 

Since it's going so well, what's next? 
 

The goal is to think bigger and determine how JC and his team can transform end-to-end data platform architecture. 

“The next step we want to take in our data architecture journey is to bring design patterns that are common and are used widely in software development and bringing those patterns into our data engineering practices,” he said. 

On that list is version control for data pipelines—hear JC explain why. 

Also, JC is working with his team to plan for the future of data architecture and analytics on a global scale, which he says will be a multi-cloud environment. Hear him explain why below.

 

Questions from the C2C Community
 

1. Are the business analysts running their daily job through the BigQuery interface? Or do they use a different application that's pulling from BigQuery?

For JC’s organization, some teams got up to speed very quickly, while others need a little more coaching, so they’ll be putting together some custom development with Tableau. Hear JC’s full answer below.

 

Hear how they use Google Sheets to manage the data exported from Big Query.

 

2. I have the feeling that my databases are way more similar than yours because my database is not talking about those things. It's just a handful of tables. So it's easier for us to monitor a handful of tables. But how do you monitor triggers?

This question led to a more in-depth discussion, so JC offered to set up a time to discuss further separately, which is just one of the beautiful benefits of being a part of the C2C community. Check out what JC said to attack the question with some clarity below. We’ll update you with their progress as it becomes available!

 

3. What data visualization tools do JC and his team use?

“Basically, the answer is we're using everything under the sun. We do have some Spotfire footprint, we have Tableau, we have Looker, and we have ClixSense. We also have custom development visualization developments,” he said.

“My team is gravitating more towards Tableau, but we have to be mindful that whatever data architecture design we come up with, it has to be decoupled, flexible, and it has to be data engine and data visualization agnostic because we do get a request to support the next visualization,” he warned. 

Hear about how JC manages the overlap with Looker and Tableau and why he likes them both. 

 

 

Extra Credit

 

JC and his team used the two articles from Thoughtworks, linked below, to inform their decision-making and what they used as a guide for modernizing their data architecture. He recommends checking them out. 

  1. How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh by Zhamak Dehghani, Thoughtworks, May 2019
  2. Data Mesh Principles and Logical Architecture by Zhamak Dehghani, Thoughtworks, December 2020

 

We want to hear from you!

 

There is so much more to discuss, so connect with us and share your Google Cloud story. You might get featured in the next installment! Get in touch with Content Manager Sabina Bhasin at sabina.bhasin@c2cgobal.com if you’re interested.

Rather chat with your peers? Join our C2C Connect chat rooms! Reach out to Director of Community Danny Pancratz at danny.pancratz@c2cglobal.com.

 

Be the first to reply!