Kubernetes Exposed: Exploiting the Kubelet API

Kubelet API is a vital component in Kubernetes clusters that manages pods and their containers on each node. While it is not typically intended for direct user interaction, many DevOps teams may utilize the Kubelet API for debugging and direct node communication. However, exposing the Kubelet API to the public internet while enabling anonymous unauthenticated requests can lead to severe security implications, including unauthorized access and potential data breaches.  

In this blog, we describe real-world attacks observed through our honeypot setup, showing that attackers aim to steal secrets to gain full control over clusters and move laterally across the cloud provider account. We highlight the techniques these attackers utilize. You will learn how to protect your Kubernetes clusters against such attacks and ensure a robust security posture. 

What is Kubelet API? 

The Kubelet API is a crucial component in Kubernetes clusters, acting as the primary agent running on each node, responsible for managing pods and their containers. It communicates with the Kubernetes control plane to ensure that the containers are running as expected. By default, the Kubelet API is not available to the internet and serves as an internal undocumented API. Sometimes DevOps teams utilize the Kubelet API to debug the nodes directly, bypassing the control plane and directly communicate with a node. To communicate directly with the Kubelet API, you need to take the following steps: 

1. Configure kubelet Options: Modify the kubelet startup options to allow anonymous and unauthenticated access. Specifically, you can use the following flags:
   • --anonymous-auth=true: This flag enables requests that are not rejected by other configured authentication methods to be treated as anonymous requests. 
   • --authentication-mode=anonymous: This option sets the authentication mode to anonymous, bypassing other authentication checks. 
2. Disable Authorization: Disable authorization by using this flag: 
   • --authorization-mode=AlwaysAllow: This setting allows all requests without performing any authorization checks. 
3. Network Configuration: Ensure the kubelet is reachable from the desired networks. This might involve setting up network rules or modifying firewalls to allow access to the default kubelet API port (10250). 

When this happens and the Kubelet API is exposed to the public internet, it can present significant security threats, such as unauthorized access, potential exploitation of vulnerabilities, and data breaches.  

Figure 1: Kubelet API operation

Attacks in the Wild 

While this exposure (misconfiguration) is well documented, we desired to learn more about concurrent campaigns against K8s clusters, that particularly target the Kubelet API. 

Aqua Nautilus has been running a Kubelet API honeypot that involves connecting the Kubelet API to the internet and allowing anyone to send anonymous, unauthenticated API requests. In this section, we review the attacks documented by our monitoring tools. Using Tracee, we collect various logs and run our logic to detect cyberattacks. Tracee is an open-source runtime security and forensics tool for Linux, built to address common Linux security issues. By leveraging the advantages of Linux extended Berkeley Packet Filter (eBPF) technology to trace systems and applications at runtime, Tracee analyzes collected events to detect suspicious behavioral patterns.  

This section teaches us about the different attacks and techniques observed in the wild. 

Internal mapping campaign 

One of the basic campaigns we recorded was an internal mapping of the K8s cluster. Below we display a compilation of various attacks that conduct execution into a container and running manual commands such as: 

• Environment discovery and modifications: Below are various commands that were sent to our honeypot, these represent environment discovery and modification attempts on the node. 

Figure 2: Various commands aimed to discover the attacked server

This first command is a shell completion code for the specified shell (bash, zsh, fish, or powershell). The rest are basic Linux commands used to discover the environment, list directory, username of current user, list the groups of the current user and list the running processes. 

• Network discovery

As can be seen in Figure 3 below, the threat actor is installing curl and zmap to allow downloading tools and scanning the network of the K8s cluster. Next, deleting the zmap blocklist. While zmap is configured to ignore internal ranges, the threat actor want to scan this range in the kubernetes cluster, thus deleting the default configuration. 

Figure 3: compilation of network discovery commands

The threat actor is also using the command-line utility ‘iptables’, which is a used to configure the Linux kernel’s built-in firewall, which is part of the Netfilter framework. It allows system administrators to define rules for how incoming and outgoing network traffic should be handled. These rules can filter, modify, forward, or reject packets based on various criteria such as source and destination IP addresses, ports, and protocols. 

Below is a summary of the commands used in these attacks: 

• ip a: Displays all network interfaces and IP addresses. This is used by attackers to gather information about the network interfaces and IP addresses of the compromised system. 
• ip -o -4 addr show dev eth0 scope global: Shows IPv4 address information for eth0 with global scope. Attackers use this to obtain specific details about the primary network interface, helping them understand the network configuration and the role of the compromised system in the network. 

• ip6tables –version: Displays the version of ip6tables. Attackers use this to check if IPv6 packet filtering is available and to understand the capabilities of the firewall on the compromised system. 

• ip6tables-legacy-save: Saves current IPv6 firewall rules. Used by attackers to save the current state of the firewall rules before making modifications, allowing them to restore the original rules if needed. 

• iptables-legacy-save: Saves current IPv4 firewall rules. Used by attackers to save the current state of the firewall rules before making modifications, allowing them to restore the original rules if needed. 

• iptables-restore -w 5 -W 100000 --noflush –counters: Restores iptables rules with specific options. Attackers use this to restore a modified set of firewall rules, possibly to allow certain types of traffic or to bypass existing firewall restrictions. 

• iptables-save -t filter: Saves current filter table rules. Attackers use this to back up the filter rules, which define the default packet filtering policies. 

• iptables-save -t nat: Saves current nat table rules. Attackers use this to back up the NAT rules, which control how network addresses are translated and manipulated. 

To sum up, the threat actor is gathering detailed information about the network configuration and firewall settings. Modifying or disabling firewall rules to facilitate their activities, such as maintaining persistence, exfiltrating data, or moving laterally within a network. Lastly, ensuring they can revert any changes made to the firewall settings to avoid detection or maintain stealth.  

After adjusting network rules the threat actor is scanning from within the container. The targeted range is 172.20.0.1/16, which is commonly used in Kubernetes for pod IP addresses, service IP addresses, or possibly node IP addresses depending on the configuration. The threat actor is utilizing zmap or nmap to scan the internal network, hoping to find internal services. 

• Secrets Collection: 

Threat actors are trying to collect secrets from various locations on the node. 

Figure 4: Internal mapping campaign and the rest in this category.

The most common attempt is to iterate over the running containers and extract the ServiceAccount token, by querying the runningpods or pods API and then trying to collect the token from each running container. 

We also saw a usage in the excellent kubeletctl tool by Cyberark. Which gathered monitoring and exploitation capabilities of the kueblet API. 

This is a nice trick used to see if the file exists, utilizing the du (disk usage) command the threat actor is checking the file space usage estimation, avoiding to check the file directly and possibly triggering alerts. This command is skipping directories on different filesystems and inspecting the specific pod, while displaying the result in bytes. The threat actor is checking the configuration and tokens of installed utilities (flannel, codedns, proxy). The nice command is used to modified scheduling priority, in this case nice -n 19, run with the lowest priority. 

Figure 5: Commands meant to find secrets on the node

Similarly, the threat actor and as illustrated in Figure 6 below, is trying to find configuration files and tokens. 

Figure 6: Commands meant to find secrets on the node

This great writeup by Bin Liu thoroughly explains about the risk of secrets in Kubernetes and introduces some tools that can help manage secrets in K8s clusters. It appears that all the techniques above (except replacing the configuration and env files) appear in his blog. 

The ‘F’ Gang Campaign 

Utilizing the Kubelet API, a new threat actor was able to run a new container and execute a command that downloads and executes a malicious script (f593). 

Figure 7: Download of main payload

Below is the shell script (f593), and as you can see it contains 2 additional shell scripts f401 and f402. Hence, we named the campaign the ‘f’ gand.

Figure 8: Download of secondary payload

The first file (f401) is set to download a binary named cron, which is actually a cryptominer (MD5= 86f2790c04ccd113a564cc074efbcdfd).

Figure 9: the content of file f401

The f402 files is set to execute the cryptominer.

Figure 10: the content of file f402

We are not sure why the threat actor chose to use one script that downloads the cryptominer and another that executes it.  

The cryptominer was first seen in VirusTotal on December 18th, 2023. The other files, and IP addresses in this attack weren’t seen in the wild before, so it looks like the campaign first started in December 2023. 

TeamTNT Kubelet Campaigns 

TeamTNT campaigns against Kubernetes clusters started in 2021 (for instance pwn and kubelet). The targeting of Kubelet API started back in 2021 and is still running today. 

The campaign was designed to target Docker and Kubernetes APIs. In Figure 11 below, you can see the scanrand() function which randomly selects an IP range and scans various ports, including port 10250, which is allocated to Kubelet APIs. If the scan discovers potential targets, the pwnkube() function is executed.

Figure 11: The function scanrand() in TeamTNT’s campaign

In Figure 12 below, we can see the pwnkube() function which scans for any running pods and collects the namespace, pod and container names. Per each collected triad it runs various commands to exploit the running containers. 

Figure 12: The function pwnkube() in TeamTNT’s campaign. These illustrate the initial exploit of the misconfigured Kubelet API

In Figure 13 below, you can see the code that runs on the exploited containers, downloading commands, and execution of a cryptominer. 

Figure 13: The xmr payload drop script in TeamTNT’s campaign

In our Kubelet honeypot we identified the TeamTNT campaign. Since it was first started, it was modified a bit. On our honeypot we identified the basic code execution as above, but in addition we also discovered some other commands executed. 

Kubelet Exposure in the Wild 

In the past few years there were many great write-ups on exploitation of Kubelet APIs. These publications reported about significant numbers of connected to the internet Kublet APIs. For instance, research in 2021 by Eduardo Bautello reported about 103K exposed Kubelet APIs and another research 2022 by Trand Micro reported about 243K exposed kubelet APIs. 

We utilized Shodan (Internet connected instances search engine) to search for internet connected and exposed Kubelet API instances. We reconstructed the Shodan query that appeared in the first report (“port:10250 ssl:true 404”) and found 287K Kubelet APIs connected to the internet. This shows that the increased trend of internet connected Kubelet APIs continues. While a 0-day exploit may lead to significant campaign against K8s clusters, the scope of exposure remains unclear at this point. 

We, therefore, asked how many Kubelet APIs allow anonymous unauthenticated requests, namely how many APIs can be exploited by attackers. We found that 100 Kubelet APIs are both connected to the internet and can be exploited by attackers. 27 of them (27%) enabled communication with the node, while 9 (9%) were identified as honeypots (low or high interaction). 

On average these 27 nodes ran 11.1 pods, when the one with the most pods ran 44 of them. Of the 27 nodes 15 returned at least one ServiceAccount token, and 8 both returned ServiceAccount token and had the control plane exposed to the internet. In this case the control plane enables using the exposed ServiceAccount, and possibly privilege escalation to further control over the cluster. 

In 4 cases of exposed Kubelet API there was also Kubernetes Dashboard connected to the cluster and 3 cases a self-hosted container registry which contained dozens to hundreds of container images.  

In the past, there was a high number of exposed and exploitable kubelet APIs. However, it appears that organizations have taken matters into their own hands and made significant changes. We found only 27 nodes that are accessible and exploitable from the outside. 

Microsoft’s Threat Matrix for Kubernetes 

We often utilize the MITRE ATT&CK framework to map the components in the attacks we review. In this case, however, MITRE don’t offer an ATT&CK framework for Kubernetes clusters. Luckily Microsoft created one of their own in 04/2020 and updated it on 03/2021. Below we highlight the techniques in this blog: 

(*) Colored in green are new categories added in this blog by Aqua Nautilus

Misconfigured Kubelet API (Initial Access): This is a new suggestion, that doesn’t appear in the Microsoft matrix. It appears to be missing as attackers actively exploit misconfigured kubelet APIs that are connected to the internet. 

Execute payload from running pods (Defense evasion): The attackers are utilizing running pods and containers that are known to the cluster operations, thus reducing the chance detection because the pods and containers are known to the defenders, as opposed to new pods, which may raise suspicion.  

Summary and Mitigation 

The Kubelet API, while crucial for managing containers in Kubernetes clusters, can pose significant security risks if exposed to the public internet. Through our honeypot setup, we’ve observed various attack techniques, including environment discovery, network scanning, and attempts to collect secrets. These activities demonstrate the potential for unauthorized access, data breaches, and disruption of services. 

To mitigate these risks: 

1. Restrict Access: By default, Kubelet APIs aren’t supposed to allow anonymous unauthenticated requests. According to best practice, they shouldn’t be exposed to the internet. Ensure the Kubelet API doesn’t allow anonymous unauthenticated requests and is not exposed to the public internet. Use network policies and firewalls to limit access to trusted sources.
2. Authentication and Authorization: Implement robust authentication and authorization mechanisms to control who can access the Kubelet API.
3. Monitor and Audit: Continuously monitor and audit Kubelet API access logs for suspicious activities. Use tools like Tracee to detect and respond to potential threats.
4. Security Patching: Keep Kubernetes and its components up to date with the latest security patches to protect against known vulnerabilities.
5. Least Privilege: Follow the principle of least privilege, ensuring that users and services have only the minimum permissions necessary to perform their tasks. 
6. Automate Kubernetes Posture Management (KSPM): Ensure that best practices for hardening your clusters and preventing drift are continuously monitored and maintained using KSPM. 

By implementing these measures, you can significantly reduce the risk of exposing your Kubernetes clusters to potential attacks, ensuring a more secure and resilient infrastructure.