Kubernetes DDoS: Dangers of K8s Auto Scaling by Gilad David Maayan
Distributed denial of service (DDoS) is an old attack pattern, and there are well-known solutions for protecting against DDoS attacks. However, in a cloud-native world, applications behave differently, and attackers are finding ways to exploit this behavior. Applications running within a Kubernetes cluster can scale up and down on demand. So what happens when DDoS traffic hits this type of application?
I’ll describe new types of DDoS attacks known as burst attacks and EDoS attacks, show how they impact Kubernetes applications, and provide best practices you can use to protect your Kubernetes clusters.
Cloud Native Denial of Service Attacks: EDoS and The YoYo Attack Pattern
Burst attacks are a new type of DDoS attack. In a burst attack, the victim is subjected to multiple bursts of traffic lasting from a few seconds to a few minutes each, with total attack duration varying from a few hours to a few days. Burst attacks are now feasible as recent improvements in botnet technology have enabled them to work in a planned and synchronized manner.
When executed properly, burst attacks are cost-effective for attackers and can use advanced resource controls to divert attack traffic to multiple endpoint targets. It also confuses traditional DDoS mitigation and detection solutions.
Modern cloud native applications are immune to traditional DDoS attacks because they have automated scalability mechanisms. When a denial of service attack hits these applications with fake traffic, they can add more virtual machines (VMs) to handle the additional traffic. This means DDoS does not necessarily result in performance degradation or downtime. However, it becomes an Economic Denial of Sustainability attack (EDoS), where the application owner wastes resources by scaling up their application to process fake traffic.
Recent research (Ben David and Bremler Barr, 2021) describes a type of burst attack known as the YoYo attack. A YoYo attack is a burst attack that deliberately targets auto scaling of VMs in cloud platforms. Attackers deliver DDoS traffic in bursts, causing the application to scale up with each burst and then scale down when the burst ends.
Scaling events can take several minutes and result in performance degradation. Attackers can do this again and again, causing multiple scaling events. Repetitive scaling can hurt application performance and even result in downtime. So if the bursts are timed correctly, they can severely disrupt cloud native applications. Another aspect of this type of attack is that it “tricks” a cloud native application into scaling up but then stops the traffic, causing additional waste of cloud resources.
The researchers conducted an experiment using the Google Cloud Platform, proving that a YoYo attack can be effective against containerized applications and Kubernetes. Even though Kubernetes has more nuanced auto scaling mechanisms, it is still dependent on scaling of cloud VMs. It is still not fully understood how these types of attacks will interact with Kubernetes auto scaling, and attackers may find ways to specifically exploit Kubernetes mechanisms in the future.
What is Kubernetes Autoscaling?
Kubernetes is an open-source container orchestration platform. You can use Kubernetes to manage containers and microservices across various environments, including public and private clouds, as well as multi-cloud and hybrid cloud implementations.
Kubernetes helps automate various orchestration tasks, including deployment, scheduling, maintenance, operations, and scaling across multiple clusters of nodes. Kubernetes runs containers on a common shared operating system on a host machine while keeping each container isolated. Containers remain isolated unless you connect them.
Kubernetes is designed for scalability, providing many features that enable applications and the host infrastructure to scale in and out. You can specify metrics that influence automated scaling (autoscaling) processes, including demand and efficiency.
Kubernetes supports the following autoscaling types:
- Vertical pod autoscaler (VPA) – automatically decreases or increases the resource limits on a certain pod.
- Horizontal pod autoscaler (HPA) – automatically decreases or increases the number of pod instances.
- Cluster autoscaler (CA) – automatically decreases or increases the number of nodes within a certain node pool according to pod scheduling.
Kubernetes DDoS Mitigation Strategies
As explained above, cloud native architectures like Kubernetes convert DDoS attacks into economic denial of sustainability (EDoS) attacks. The attack does not necessarily impact application performance or cause downtime, but it can waste resources for application owners by causing them to needlessly scale up the application.
With this in mind, here are a few best practices that can help you harden Kubernetes to limit the impact of DDoS and EDoS attacks.
Use IP Allow/Deny Lists
At the network level, use the IP addresses of ingress traffic to decide if it should be allowed to access your cluster or not. The best scenario is to have an allowlist of known client IPs. If this is not possible, you can use a denylist of known bad traffic sources using threat intelligence databases. Yet another approach is to use a web application firewall (WAF), which can inspect application traffic and block malicious traffic patterns.
Connection Limits
Set a hard limit on the number of connections an external client can create to your cluster. This can help both with malicious DDoS clients and misconfigured clients that send a large number of requests accidentally.
However, it can be challenging to determine the most appropriate value for the connection limit. In some Kubernetes environments, you may need to set one limit for all traffic sources, which can be problematic. Some clients share the same IP address, for example, if they are using a NAT. This means that limiting the number of connections can block legitimate users.
Rate Limiting
You can use a Kubernetes ingress controller to limit the number of requests in a specific timeframe – for example, requests per minute or second. You can implement this by limiting requests per client IP, as it appears in the HTTP header, or by limiting requests at the service host level.
Limiting Simultaneous Requests
Even if each individual client has a rate limit, your cluster can still be overwhelmed by high concurrency of requests. As above, this can occur with malicious clients, misconfigured clients, or even a natural burst of traffic. Determine a level of concurrent requests which is not reasonable for your cluster to withstand or which will result in excessive cloud costs, and set a hard limit on your external traffic.
Audit Logging
Use access logs to identify a DDoS attack at its early stages and respond to it. If your organization has a security information and event management (SIEM) system, you can ingest Kubernetes access logs into the SIEM. This allows the SIEM to notify security teams of abnormal traffic to your Kubernetes cluster. You can also leverage access logs in a Kubernetes ingress controller to automate remediation strategies for traffic patterns that may indicate an attack.
Conclusion
In this article, I explained new types of DDoS attacks that can affect cloud native applications and Kubernetes clusters. I also introduced the concept of EDoS attacks, which do not result in denial of service, instead wasting resources for application owners by tricking them into scaling up their applications.
I covered several ways you can harden your Kubernetes clusters to protect against DDoS and EDoS attacks:
- Use IP allow/deny lists – preventing harmful traffic from connecting to your cluster in the first place.
- Apply connection limits – limiting the number of connections allowed from any specific client.
- Rate limiting – within a specific connect, limiting the rate at which clients can send requests to the cluster.
- Limiting concurrency – placing a limit on the amount of traffic that can hit your Kubernetes cluster at any given time.
- Audit logging – leveraging access logs to identify DDoS attacks and automate remediation strategies.
I hope this will be useful as you prepare for the next generation of denial of service attacks.
ABOUT THE AUTHOR:
Gilad David Maayan is a technology writer who has worked with over 150 technology companies including SAP, Imperva, Samsung NEXT, NetApp and Ixia, producing technical and thought leadership content that elucidates technical solutions for developers and IT leadership. Today he heads Agile SEO, the leading marketing agency in the technology industry.
LinkedIn: https://www.linkedin.com/in/giladdavidmaayan/
Twitter: https://twitter.com/gilad_maayan
FB: https://www.facebook.com/gilad.maayan
