Traffic drops to zero on existing instances when scale out event happens

Hello @Idan .

Is this a TCP or HTTP/S load balancer? The backend is GKE or a managed instance group?
Let me know if you want to have a Google Meet session to talk about the issue.

Yuval, can we have a short session to show you this issue and maybe get some insights?
This prevents us from using any scaling topology in our solution as we are traffic affected each time scale out event happens .

@Idan Is it solved?

Also, do you know it just by seeing the graphics or looking inside?

Maybe the observability gives some false positive.

1. Application configuration: Examine your application's configuration to verify if it relies on specific instance identifiers or if there are any limitations causing traffic to drop when new instances are added. For example, if your application uses IP-based whitelisting or caching based on instance identifiers, it may not recognize the new instances until the cache expires or the whitelist is updated.

2. Database or session management: If your application relies on a shared database or session management system, ensure that the new instances can connect to and access the necessary resources. If there are limitations on the number of connections or sessions, adjust the configuration accordingly to accommodate the additional instances.

3. Monitoring and logging: Implement comprehensive monitoring and logging to gain visibility into your infrastructure and application performance. Monitor the traffic, instance health, and any relevant metrics during and after the scale-out event. Look for any anomalies or errors that may indicate the cause of the traffic drop.

4. Test and validate: Conduct thorough testing and validation to ensure that your application can handle the expected traffic after a scale-out event. Simulate traffic spikes or increased loads to observe how the system responds and identify any issues that may lead to traffic drops.