Orchestration often refers to the automated configuration, coordination, and management of computer systems and services.
Google Cloud Platform offers a number of tools and services for orchestration:
- Cloud Scheduler for schedule-driven single-service orchestration
- Workflows for complex multi-service orchestration
- Cloud Composer for orchestration of your data workloads
One image is like one thousand words. Don’t you agree? The article is great but the diagram is even better!
I was asked by a customer about a batch orchestration tool. As a K8s person my first instinct was Argo / Flux / Tekton but I looked at Workflows and it looks very promising especially for users who do not want tomanage K8s (even with autopilot). There are some features that could make Workflows much better like waiting for a callback but there is a great blog post: 3 common serverless patterns to build with Workflows with some interesting ideas on how to emulate callbacks.
The article is really great
@yuval! I remember the impression that made me when I read it.
Thanks for sharing it.
I haven’t used Argo / Flux / Tekton. Can you tell us more about them?
And I suppose you have signed up on our event with Kelsey, as a K8s person that you are, right? 😉
I’m not fully agree with this diagram. You can perform ETL/ELT with Workflows. you have less visibility, less control, less control management, but if you call only services, such as BigQuery or Dataflows (with template) it’s only API (thus services) and Workflow fit perfectly this use case.
Composer offers entreprise grade workflow management with more controls and features, and more cost also (min $400 per months).
i’m sure that
@noziere from l’Oreal company can explain more their design and why they move from composer to Workflows in their data pipeline today.
@noziere could give us more insights on that.