Traffic drops to zero on existing instances when scale out event happens | C2C Community
Question

Traffic drops to zero on existing instances when scale out event happens


Userlevel 1

While using internal load balancer, scale out event happens (due to traffic metric activation), a new instance is created.
At this point I noticed that all existing instances do not get traffic AT ALL from the load balancer for at least 10 seconds (traffic drops to zero).
This cause a sever traffic failures to the backend and only after a few minutes when the scale out event completes the old instanced get traffic back.
 


7 replies

Userlevel 4
Badge +3

Hello @Idan .

Is this a TCP or HTTP/S load balancer? The backend is GKE or a managed instance group?
Let me know if you want to have a Google Meet session to talk about the issue.

Userlevel 1

Hi Yuval,
As for your questions:
1. This is an HTTPS load balancer
2. The backend is Managed instance group
Can we schedule a Google Meet for next week so I can show a live demo of this issue?

Userlevel 1

Yuval, can we have a short session to show you this issue and maybe get some insights?
This prevents us from using any scaling topology in our solution as we are traffic affected each time scale out event happens .

Seeing the same behavior in my lab. A short while after my MIG scales out, traffic to all existing instances is temporarily drained (for about 10-20 seconds). No clue why it happens yet.

Userlevel 6
Badge +11

@Idan Is it solved?

Also, do you know it just by seeing the graphics or looking inside?

Maybe the observability gives some false positive.

Badge +1

you gotta check the trafic

  1. Application configuration: Examine your application's configuration to verify if it relies on specific instance identifiers or if there are any limitations causing traffic to drop when new instances are added. For example, if your application uses IP-based whitelisting or caching based on instance identifiers, it may not recognize the new instances until the cache expires or the whitelist is updated.

  2. Database or session management: If your application relies on a shared database or session management system, ensure that the new instances can connect to and access the necessary resources. If there are limitations on the number of connections or sessions, adjust the configuration accordingly to accommodate the additional instances.

  3. Monitoring and logging: Implement comprehensive monitoring and logging to gain visibility into your infrastructure and application performance. Monitor the traffic, instance health, and any relevant metrics during and after the scale-out event. Look for any anomalies or errors that may indicate the cause of the traffic drop.

  4. Test and validate: Conduct thorough testing and validation to ensure that your application can handle the expected traffic after a scale-out event. Simulate traffic spikes or increased loads to observe how the system responds and identify any issues that may lead to traffic drops. 

 

Reply