Dataflow Observability, Monitoring, and Troubleshooting | C2C Community

Dataflow Observability, Monitoring, and Troubleshooting

  • 15 August 2022
  • 0 replies

Userlevel 7
Badge +21

Join on August 17th at 9AM PT for a live session with Google Cloud experts on Dataflow observability, monitoring, and troubleshooting.

Google Cloud expert will provide an overview of all the observability experiences in Dataflow and discuss more on the new features. They will also look at the common symptoms or issues reported by customers for streaming pipelines and discuss how to use the new observability features and tools to troubleshoot those.

To participate in the live session on August 17 at 9 a.m. Pacific Time, click on the video link below.



Video chapter:

03:02 Evolution of data and the value gap

06:21 Why Dataflow?

08:44 What is Dataflow?

10:25 What is observability? Core Dataflow observability features:

11:48 Job visualizers

14:38 Job metrics

21:06 Cloud Error Reporting integration

22:19 Cloud Profiler integration

23:33 Dataflow insights (recommendations)

25:21 Datadog integration

26:54 Where to start?

29:23 Live demo Troubleshooting common scenarios

48:31 Job slows down with increase in data volume

49:50 Quota issues, throttling

51:19 OOMs (out of memory)

52:29 Hot keys, uneven CPU load across workers

57:13 User code issues

59:16 Job failed to start

1:01:00 Job is not autoscaling as expected

1:04:52 Dataflow observability roadmap

0 replies

Be the first to reply!