In this research, we uncovered several vulnerabilities and security flaws within the Prometheus ecosystem. These findings span across three major areas: information disclosure, denial-of-service (DoS), and code execution. We found that exposed Prometheus servers or exporters, often lacking proper authentication, allowed attackers to easily gather sensitive information, such as credentials and API keys.
Additionally, we identified an alarming risk of DoS attacks stemming from the exposure of pprof debugging endpoints, which, when exploited, could overwhelm and crash Prometheus servers, Kubernetes pods and other hosts.
Furthermore, our investigation revealed a remote code execution risk due to a vulnerability called “RepoJacking”, where malicious exporters could be introduced through abandoned or renamed GitHub repositories.
Our findings highlight that at least 336,000 servers expose their Prometheus servers and exporters to the internet—a practice that poses significant security risks. It is crucial to restrict public access to these servers, as attackers can easily exploit this exposure to target organizations.
What is Prometheus?
Prometheus is an open source monitoring and alerting toolkit that has become a cornerstone of modern monitoring strategies. Its core architecture includes scraping metrics from targets at regular intervals, storing them locally in a time-series database, and providing a powerful query language, PromQL, for real-time data analysis. The system is modular, with components such as Alertmanager for handling alerts and Grafana for visualization, making it ideal for dynamic environments like Kubernetes.
Figure 1: Prometheus Stack
Exporters are installed on various systems and serve to collect metrics from the monitored endpoints, making it possible for Prometheus to scrape and store data from systems, applications, or services that don’t natively expose metrics in the Prometheus format.
Our Discoveries
According to the Prometheus documentation it is presumed that untrusted users can view information about Prometheus, specifically Prometheus HTTP endpoint, logs and debugging information. Although Prometheus supports user authentication, we wondered about the above-mentioned disclaimer and asked ourselves if practitioners indeed expose the Prometheus server or its exporters to the internet without authentication.
We used Shodan, a search engine designed to identify internet-connected devices, to analyze the presence of publicly accessible Prometheus servers and exporters. Our findings revealed over 296,000 internet-facing exporters and 40,000 Prometheus servers, totaling approximately 336,000 servers were exposed to potential risks.
Figure 2: Prometheus exporters in Shodan
Figure 3: Prometheus servers in Shodan
Information Disclosure Through Exposed Prometheus Servers and Metrics Endpoints
The concept of information disclosure through publicly accessible Prometheus servers or exporters is not new and has been previously highlighted by researchers Andrey Polkovnychenko and Shachar Menashe from JFrog, as well as Miguel Hernández and David de Torres Huerta from Sysdig, in their reports.
Although these research studies aimed to raise awareness about the risks of exposing Prometheus servers and exporters, the number of exposed instances remains alarmingly high, with over 336,000 exposed assets. This widespread exposure has resulted in users inadvertently revealing numerous sensitive secrets through their Prometheus components.
When Prometheus servers or exporters are connected to the public internet without authentication, they introduce a significant risk. Such misconfigurations allow anyone to query the exposed environments to list labels or metrics. Attackers can exploit this access to gather seemingly trivial data and, with the help of secret-scanning tools, uncover sensitive information, including credentials, passwords, authentication tokens, and API keys.
As shown in Figure 4, this is far from a theoretical risk. Unauthenticated Prometheus servers enable direct querying of internal data, potentially exposing secrets that attackers can exploit to gain an initial foothold in various organizations.
Figure 4: Secrets Exposed in Prometheus Servers on Port 9090
We observed another example of information disclosure through an exposed /metrics endpoint of Node Exporter. In some cases, this endpoint can reveal internal API endpoints. This type of exposure can inadvertently allow attackers to expand their attack surface, gain access to sensitive data, and learn and exploit internal backend functionalities that were not intended for public use.
In addition, exposed Prometheus servers and the /metrics endpoint can reveal subdomains, Docker registries, images, and other information about a company. For instance, we discovered an unauthenticated Prometheus instance associated with Skoda, one of the biggest European car manufacturers. In addition to exposing Docker registries and images linked to Skoda, this Prometheus server also revealed subdomains and paths of Skoda through the kube_ingress_path metrics.
Figure 5: Exposed Prometheus server of Skoda
We reported our findings to Skoda’s security team, and they mitigated the risk by blocking access to this instance.
DoS in Prometheus: Exploiting /debug/pprof Endpoints
In this research, we conducted an analysis of Prometheus and its key components, including prometheus/prometheus, prometheus/alertmanager, prometheus/pushgateway, prometheus/node_exporter, and others. A particular focus was placed on the Go debugging interface, known as the pprof package, which is commonly used for performance profiling. Specifically, the http/pprof package, when imported (by default it is embedded in most Prometheus components), provides the /debug/pprof endpoint as part of an HTTP server for profiling data. For instance, you can access the pprof endpoint via http://<<IP_ADDRESS:9090-9100>>/debug/pprof/ for Prometheus server and node exporter.
Exploiting the ‘/debug/pprof’ Endpoint
Misconfigured Prometheus servers and exporters exposed to the internet provide HTTP access to the pprof endpoint, which is enabled by default in most Prometheus components. As shown in Figure 4, accessing the /debug/pprof endpoint grants access to various functionalities, such as /debug/pprof/heap for heap profiling, /debug/pprof/trace for tracing, and more.
Figure 6: An exposed Prometheus server/Node exporter enabling access to the ‘/debug/pprof’
The exposed /debug/pprof endpoint poses significant security risks. While it is designed to assist users in profiling remote hosts, attackers can exploit it to execute Denial of Service (DoS) attacks. For example, sending multiple simultaneous requests to endpoints like /debug/pprof/heap?seconds={i} forces the server to perform intensive profiling operations, consuming excessive CPU and memory resources. This can degrade the system’s performance, potentially leading to service instability or outages, especially if proper access controls or resource limits are not in place.
Additional risks associated with the exposed /debug/pprof endpoint are detailed in the following blog.
We created a Proof of Concept available on the following Gist. This script demonstrates how a DoS attack can be executed against an exposed /debug/pprof endpoint.
In our research, we tested the script on various Prometheus components and deployments, including the Node Exporter, which collects system metrics like CPU, memory, and disk usage for Prometheus monitoring. We used a single standard VM. Our tests targeting the /debug/pprof debugging endpoint were divided into two categories:
Host-Level Deployment
When the Node Exporter is installed directly on a host, attackers can exploit the /debug/pprof endpoint with DoS requests, rendering the host and its services inaccessible. In cloud-hosted environments, such as AWS EC2 instances, our tests demonstrated that a DoS attack caused a loss of communication with the host, requiring a machine reboot to restore functionality.
Pod-Level Deployment (Kubernetes)
When Node Exporter runs in a Kubernetes pod, attackers can repeatedly crash the pod using the /debug/pprof endpoint. Our tests resulted in (depend on how Kubernetes is configured):
- Increased Operational Overhead: Frequent pod failures require constant attention from administrators.
- Degraded Cluster Performance: Kubernetes attempts to restart failing pods, impacting the performance of other workloads.
- Resource Exhaustion: Pods failing due to OOMKilled or evictions unnecessarily consume resources.
Prior Warnings
Later, we discovered that the exposure of /debug/pprof had been flagged as a security risk in a audit conducted by Cure53 for the Prometheus security team and noted in GitHub issues.
Similar risks were recognized and addressed by other CNCF projects, such as Kubernetes, which resolved related vulnerabilities, for example CVE-2019-11248.
Disclosure Process with Prometheus
Exposing Prometheus, its components, and various exporters to the internet without authentication is considered poor practice. However, the /debug/pprof endpoint introduces a particularly concerning risk: the ability to directly impact the host machine/pod and serve as a vector for DoS attacks. In our view, this vulnerability demands attention and mitigation.
When we disclosed our findings to the Prometheus security team, their response was:
“Supporting good production practices trumps protecting users from gross misconfigurations.”
Although there is an intention enabling users to configure different endpoints differently, including switching them off, it is still in the backlog and not in available to users. Meanwhile we strongly advise not to expose Prometheus components without authentication as our findings above suggest that this exposure introduces significant security risks.
Code Execution via RepoJacking in Prometheus Exporters
During our research, we discovered found that several Prometheus exporters are vulnerable to RepoJacking. This illustrates that RepoJacking remains an actual risk in many open-source projects.
GitHub RepoJacking is a type of supply chain attack that allows attackers to takeover GitHub projects’ dependencies or an entire project to run malicious code on whoever uses these projects. It occurs when an owner deletes or renames a repository. If this repository is still referenced by other internal or external projects, an attacker can create a new repository with the same old name (changed or deleted) and by doing so take control over the referenced name. Now any new users who will use the old or deleted name will download code from the repository that is controlled by the attack, who can introduce malicious code and execute it in the environments of whoever users the old name (using old links to the original project). You can read more about this vulnerability in our blog, along with some mitigation steps by GitHub in the following article.
In the case of Prometheus, we discovered that several exporters listed in its official documentation were vulnerable to RepoJacking. An attacker could claim the now-available usernames referenced in the documentation, recreate an exporter with the same name, and host a malicious version. Unsuspecting users following the documentation could unknowingly clone and deploy this malicious exporter, leading to remote code execution on their systems.
Below are examples of exporters that were vulnerable:
| Category | Exporter Name |
| HTTP | Nginx VTS Exporter |
| HTTP | Tinyproxy Exporter |
| FinOps | AWS Cost Exporter |
| FinOps | Azure Cost Exporter |
| FinOps | Kubernetes Cost Exporter |
| APIs | Rancher Exporter |
| APIs | Docker Hub Exporter |
| APIs | Docker Cloud Exporter |
We can discuss two use cases from this list. Consider the “AWS Cost Exporter” as an example. When someone clicks the aws-cost-exporter link, GitHub redirects them to the account electrolux-oss and the repository aws-cost-exporter, even though the original link points to a different account, opensourceelectrolux, with the same repository name. Here’s how the links appear:
- Original link: https://github.com/opensourceelectrolux/aws-cost-exporter
- Redirect to: https://github.com/electrolux-oss/aws-cost-exporter
Since the username opensourceelectrolux is currently available for claim, attackers could potentially take over this username and host a malicious exporter under the original address. If someone clicks the original link, they will unknowingly redirect the exporter hosted in the attacker’s repository.
Another exporter may also be at risk. The link to the exporter under the username hnlq715 currently leads to a 404 (Not Found) error. Attempting to claim this account is not possible, and it’s unclear who currently controls it. This account could already have been seized, posing a potential risk to users.
We reported this risk to the Prometheus security team and submitted a pull request, which was merged, to address the issue.
Summary and Mitigation
In this research we highlighted vulnerabilities and flaws in the Prometheus stack. We highlight the risks associated with exposing Prometheus servers and exporters to the internet without authentication, which expose sensitive information and can be exploited to launch DoS attacks or even execute arbitrary code through compromised exporters.
As mitigation we recommend organizations the following guidelines:
- Authentication and Authorization: Prometheus servers and exporters should be protected with proper authentication mechanisms, ensuring that only authorized users can access sensitive data or system internals.
- Limit External Exposure: Prometheus should be deployed in a manner that minimizes exposure to the public internet. In cases where external access is necessary, secure communication channels (e.g., VPNs or private networks) should be used to limit who can access the services.
- Monitor and Secure Debugging Endpoints: The
/debug/pprofendpoint, which can be exploited for DoS attacks, should not be exposed publicly. Access to debugging and profiling endpoints should be limited to internal use only or disabled in production environments. - Limit Resource Exhaustion: We strongly recommend placing guardrails that limit resources exhaustion such as CPU and RAM limitations that can prevent or minimize attackers’ ability to DoS against such exposure.
- Inspect Open-Source Links to avoid RepoJacking: To prevent supply chain attacks such as RepoJacking make sure you are 100% positive that the links from where you download the project are the correct and original ones.
