When you’re running Kubernetes, how do you know whether it’s configured securely? Kubernetes is a complex system, with several control plane components, each of which has numerous configuration parameters. In some cases, it’s clear that a parameter will have an impact on Kubernetes security – for example, providing keys and certificates for setting up secure connections between components. In other cases, the security impact is far less obvious. Did you know that attackers are known to deliberately attempt to flood your audit logs so that they wrap, attempting to cover their tracks? This means that even your audit log configuration settings have a bearing on security.
With dozens or hundreds of different configuration parameters across the system, it’s challenging for Kubernetes administrators to know whether they have set them all up correctly with good security practices in mind. Fortunately, there is guidance available in the form of the Center for Internet Security’s benchmark recommendations.
CIS Kubernetes Benchmark
The Center for Internet Security publishes a series of Benchmarks with advice on how to configure software according to security best practices.
The Benchmark documents follow a standard format, with instructions on how to audit (that is, how to determine whether your configuration matches the recommendation), and how to remediate in the case where your setup doesn’t pass that audit test. There is also background on why a setting matters, and on the possible impacts of changing a setting to meet the recommendation.
You may well be wondering how the CIS establishes these recommendations. The answer is that the recommendations are written collaboratively by a group of volunteer security experts from across the Kubernetes community. We’re now at v1.4.0 of the CIS Kubernetes Benchmark, with the benchmark authors working to keep the document up to date as Kubernetes evolves.
Testing configurations with kube-bench
The Kubernetes benchmark includes over 200 pages of recommended tests, so it’s impractical to run them by hand even just once – and the reality is that you should be running tests on every node in your cluster. Further, it’s a good idea to run them on a regular basis to spot configuration drift.
At Aqua Security we first released our open source tool kube-bench back in 2017, automating the CIS Kubernetes tests. Since then we’ve had over 100k downloads of the kube-bench Docker image, and contributions from dozens of people across the community. We’ve also incorporated kube-bench into the Aqua Container Security Platform – currently the only container security tool that’s CIS Certified for the Kubernetes Benchmark.
However, not every test defined by the CIS Benchmark is applicable for every distribution of Kubernetes. Some distributions, especially when they are offered as a managed service, have compensating controls that fall outside the scope of the CIS Benchmark. This results in failed tests suggesting non-secure configuration settings on Kubernetes, when in fact the security issues are addressed elsewhere in the overall system. These are false negatives, so to avoid this it’s helpful to be able to modify the test suite to skip those tests that aren’t relevant in a given environment.
Since the beginning kube-bench’s YAML test file format has made it relatively easy to tweak the tests to meet their needs, but that requires the user to have a very good understanding and a firm opinion on which tests they want to add, remove or modify for their particular environment.
Openshift benchmark tests
Openshift is a great example where there are several aspects of security that are addressed outside the scope of the Kubernetes benchmark. Furthermore, the audit commands need to be adapted as Openshift has different command line tools compared with vanilla Kubernetes.
Working in conjunction with Red Hat, who advised on the changes necessary to tune the CIS tests for an OpenShift platform, we have recently extended the kube-bench test suite to include a set of configuration files for OpenShift. The current test files cover OCP 3.10 and 3.11. It’s worth pointing out that at this stage the OpenShift-specific tests don’t have the CIS stamp of approval, but there are moves afoot to broaden the scope of the CIS recommendations to cover the changes needed for different distributions and environments.
“One of the things that’s always been a challenge when writing for the Kubernetes ecosystem is the sheer variety of options available,” explains Rory McCune of NCC, one of the CIS Kubernetes Benchmark authors. “At the moment the CIS Benchmark has focused on a common baseline, but we’re hoping to be able to expand that with help from distribution and service vendors in the future.”
How Kubernetes is configured
One of the challenges with these recommended tests is a reflection of the significant changes in Kubernetes since the CIS Kubernetes benchmark was first released. Back then, all the config parameters were set at run time on each executable, so you could reliably see the settings by running a ps command and seeing what settings were configured. This is the basis for the majority of the audit commands in the Benchmark, as you can see in the example above.
However, Kubernetes now supports reading parameters for some components from a YAML or JSON format config file. Both the CIS Benchmark and kube-bench need changes to reflect the fact that not only is this possible, it’s an increasingly commonly-used approach. Notably, the config file approach is used by Amazon’s EKS. You’ll see updates to both the benchmark and the kube-bench tool to enable accurate benchmark testing when this config file approach is in use.
Joining the community
If you would like to get involved with the CIS Kubernetes Benchmark, your starting point is to join the community at the CIS WorkBench. We’re also very happy to receive your issues and pull requests for kube-bench!