Kubernetes on VMware: What are the Options?

How Does Kubernetes Run on VMware?

Kubernetes is an open-source container orchestration platform that automates the management of containerized applications. VMware provides virtualization platforms used by a majority of enterprises. As containers become more popular, a growing requirement is to deploy containers and manage them alongside traditional virtual machines, on VMware infrastructure.

In 2019, VMware started supporting Kubernetes as part of its vSphere virtualization platform, which includes the ESXi hypervisor. It is now possible to run containers directly on ESXi, or use the Tanzu Kubernetes Grid platform to manage standard Kubernetes clusters, compatible with upstream Kubernetes development. Tanzu makes it possible to run Kubernetes in production, managing large numbers of Kubernetes clusters across VMware infrastructure, bare metal servers, and public clouds.

In this article, you will learn:

• Running Kubernetes Alongside VMware Workloads
• How Does vSphere with Kubernetes Work?
• Supervisor vs. Guest Clusters
• Running Kubernetes in a Multi Cloud Environment with Tanzu Kubernetes Grid Integrated Edition (TKGI)
• Tanzu Kubernetes Grid Integrated Edition Features
• Tanzu Kubernetes Grid Integrated Edition vs Tanzu Kubernetes Grid

Running Kubernetes Alongside VMware Workloads

VMware vSphere is VMware’s flagship virtualization platform. Since version 7, vSphere fully supports Kubernetes. It is built to support developers, who are familiar with Kubernetes, and IT staff who are familiar with vSphere system constructs:

• For developers, vSphere looks and acts like regular Kubernetes. They can use normal Kubernetes constructs and declarative configuration to request compute resources, storage, networking, and high availability. In the background, the requests are fulfilled via vSphere resources, but developers do not need to be aware of vSphere infrastructure.
• For IT administrators, vSphere works exactly like in the past, only with new workload management capabilities. vSphere Admins can work with “namespaces”, which behave just like regular virtualized workloads, but are also visible within Kubernetes. This is how admins can manage the security, resources, and networking available to the Kubernetes environment.

How Does vSphere with Kubernetes Work?

vSphere has been deeply integrated with Kubernetes, by adding the Kubernetes APIs as a new control plane. This lets Kubernetes users consume services seamlessly from the VMware environment, just like they would in a public cloud. vSphere can now manage workloads consistently, whether they are containers, applications, or virtual machines.

The ESXi hypervisor, which is at the core of vSphere, now includes the Kubernetes API, as well as the Spherelet, a management agent based on the Kubernetes Kubelet. This lets the ESXi hypervisor act as a native Kubernetes node, which can join Kubernetes clusters. 

ESXi hosts can run containers directly on the hypervisor. This is made possible by a new container runtime called CRX, which is provided as part of vSphere. This approach does not require loading a full Linux guest OS, instead it uses a highly optimized Linux kernel and lightweight init process. 

Within Kubernetes, these containers can be accessed as part of a vSphere Pod Service. The vSphere Pod Service lets you run vSphere containers in Kubernetes, but they are not fully conformant Kubernetes clusters.

Supervisor vs. Guest Clusters

vSphere lets users run two types of Kubernetes clusters:

• Supervisor cluster—a special kind of Kubernetes cluster that runs worker nodes directly on the ESXi hypervisor (not on Linux). Container workloads run in vSphere Pods, and offer the same security, high availability and performance of ESXi. However, a supervisor cluster is not a standard Kubernetes cluster, and cannot work seamlessly with existing Kubernetes deployments. 
• Tanzu Kubernetes Cluster (guest cluster)—this is a cluster, based on VMware infrastructure, which is compatible with upstream Kubernetes. A Tanzu cluster runs on virtual machines, not vSphere Pods. It uses the open source Cluster API project, which uses the VM Operator to manage virtual machine workflows for the cluster.

Running Kubernetes in a Multi Cloud Environment with Tanzu Kubernetes Grid Integrated Edition (TKGI)

VMware Tanzu Kubernetes Grid Integrated Edition is a VMware platform that makes it possible to run Kubernetes on heterogeneous multi-cloud environments, including public clouds and on-premises VMware environments. It supports both day 1 (initial cluster deployment) and day 2 operations (patching, upgrades, and high availability).

The main goal of TKGI is to expose Kubernetes in standard form, making Tanzu clusters fully compatible with existing Kubernetes deployments and upstream Kubernetes development. Developers work with the native Kubernetes CLI and APIs just like if they were deploying Kubernetes locally or on a public cloud.  

The platform lets you deploy Kubernetes clusters on:

• On-premises bare metal
• OpenShift
• On-premises VMware vSphere environments
• Public clouds including Amazon, Microsoft Azure, and Google Cloud Platform, Amazon EC2, and Microsoft Azure

TKGI provides robust management tools, including Tanzu Mission Control, which manages Kubernetes clusters on one pane of glass, whether they reside on vSphere, PKS, OpenShift, or public cloud services. Mission Control also provides policies that govern user access, resource quotas, backups, and many other aspects of a cluster, in a unified way across clouds.

TKGI comes pre-integrated with a full stack of solutions, including:

• Cloud Foudry BOSH for cluster lifecycle management
• VMware NSX-T for pod networking and load balancing
• Harbor as cloud native repository handling container images, vulnerability scanning, etc.
• Docker Community Edition (Docker-CE) as container engine
• Velero for backup and disaster recovery of Kubernetes clusters
• Prometheus for Kubernetes monitoring, with Grafana for visualization
• Sonobuoy for diagnosing the state of Kubernetes clusters.
• Fluentbit for log processing and forwarding

Tanzu Kubernetes Grid Integrated Edition Features

Here are some of the key features of Tanzu Kubernetes Integrated Edition (formerly Enterprise PKS).

• High availability—supports multi-AZ replication and multi-master etcd deployments. Monitors health of all underlying virtual machines and heals VMs upon failure. It manages rolling upgrades for fleets of Kubernetes clusters with no downtime. 
• Persistent storage—supports both stateless and stateful applications. Provides both the vSphere Cloud Provider storage plugin and standard CSI storage, supporting Persistent Volumes (PV), Persistent Volume Claims (PVC), Storage Classes and Stateful Sets. All these constructs can be used to provision vSphere storage including vSAN.
• Kubernetes lifecycle management—provides central control over the entire lifecycle of Kubernetes clusters running on bare metal, VMware vSphere or in the public cloud.
• Unified access management—centralized management for cluster access, letting the organization define in one place which teams can access which clusters, as an added management layer above Active Directory groups.
• Security and configuration management—defining security and network policies at scale across multiple clusters. Allows administrators to manage policies by grouping Kubernetes clusters together in workspaces, divided according to functional teams or development stages, and applying policies to all clusters in a group at once.

Related content: read our guide to Kubernetes security best practices ›

Tanzu Kubernetes Grid Integrated Edition vs Tanzu Kubernetes Grid

Alongside TKGI, VMware also provides Tanzu Kubernetes Grid (TKG). This is a separate offering with a deeper vSphere integration, but which is less suitable for multi-cloud deployment.

The following table summarizes three flavors of the VMware Tanzu platform.