Join on August 17th at 9AM PT for a live session with Google Cloud experts on Dataflow observability, monitoring, and troubleshooting.
Google Cloud expert will provide an overview of all the observability experiences in Dataflow and discuss more on the new features. They will also look at the common symptoms or issues reported by customers for streaming pipelines and discuss how to use the new observability features and tools to troubleshoot those.
To participate in the live session on August 17 at 9 a.m. Pacific Time, click on the video link below.
Video chapter:
03:02 Evolution of data and the value gap
06:21 Why Dataflow?
08:44 What is Dataflow?
10:25 What is observability? Core Dataflow observability features:
11:48 Job visualizers
14:38 Job metrics
21:06 Cloud Error Reporting integration
22:19 Cloud Profiler integration
23:33 Dataflow insights (recommendations)
25:21 Datadog integration
26:54 Where to start?
29:23 Live demo Troubleshooting common scenarios
48:31 Job slows down with increase in data volume
49:50 Quota issues, throttling
51:19 OOMs (out of memory)
52:29 Hot keys, uneven CPU load across workers
57:13 User code issues
59:16 Job failed to start
1:01:00 Job is not autoscaling as expected
1:04:52 Dataflow observability roadmap
