Kubernetes on AWS

Kubernetes on AWS

Learn about Kubernetes Node Components, Kubernetes vs. ECS, deploying on AWS using Kops, Kubernetes EKS, launching Kubernetes on EC2 using Rancher, scheduling Resources using Terraform, and more.

In this page: everything you need to know about Kubernetes on AWS



Overview

Amazon Web Services (AWS) is a popular cloud provider option for Kubernetes deployments, as it allows unlimited scaling of an enterprise containerized application clusters. AWS’ region availability all around the world means Kubernetes clusters can benefit from very low latencies. Additionally, the wide range of AWS services like S3 for raw storage or RDS for relational databases, it becomes easy to use Kubernetes for both stateless and stateful workloads integrated with native AWS services.

Kubernetes Components and Architecture

A Kubernetes node runs the kube-proxy and kubelet components together with a container runtime engine, typically Docker.


1. kube-proxy

This Kubernetes proxy service runs on each node in the cluster and works as a load balancer for services running on a single worker node. Proxy service watches the master node for addition and removal of services/endpoints and does load balancing via simple UDP, TCP stream forwarding and round-robin across a set of backend services. The proxy service is also responsible for virtual IP implementation for regular services.


2. kubelet

kubelet is the lowest level component of Kubernetes that runs on each node in the cluster, it’s kind of a process watcher for containers. kubelet works on pod object specifications. Since those declare a desired state, kubelet works to make sure that the current state matches the desired state.


3. cAdvisor

cAdvisor is responsible for collecting, aggregating, processing and exporting metrics for instance memory, CPU and network data among others about containers that are running on a certain node. It has a built-in web interface as well to visualize the metrics.

Kubernetes vs. ECS

Here we’ll look at the differences between deploying and running your own Kubernetes clusters (for example on AWS EC2 instances) versus AWS ECS service, Amazon’s own container orchestration service. Both options present advantages and disadvantages, depending on which use case and operational requirements.

ECS is a highly scalable, high performance container orchestration service that supports Docker and allows running and scaling containerized applications on AWS with minimal configuration.

Ease of Setup

Tools like kops or kubeadm can greatly ease the task of deploying a Kubernetes cluster on AWS EC2. These tools make extensive use of AWS’ APIs to automate the process.

On the other hand, for those who prefer a GUI, setting up an ECS cluster via AWS’ dashboard is quite straightforward.

Networking

Kubernetes has a built-in service discovery with a particular networking model which automatically assigns a new IP address to every pod. Also, a new subnet gets created for each new Kubernetes node. However, multi-availability zones (multi-AZ) support poses a challenge as a single Kubernetes cluster cannot spread across multiple availability zones. Instead, to support that use case, one must use Kubernetes federation model with multiple clusters that must register themselves in the federation.

On the other hand, ECS has built-in service discovery which integrates natively with AWS Elastic Load Balancing (ELB) and Application Load Balancers. Each container created on ECS gets a new port number, requiring application awareness of those ports. Finally, there are no scaling issues for ECS between regions and availability zones since this feature is natively managed by AWS.

Vendor Lock-in

Given Kubernetes’ community and open source nature, it naturally supports multiple cloud vendors (despite its Google origins) and even bare metal implementations. A given Kubernetes configuration is mostly portable between providers. However, when the applications running on Kubernetes are deeply integrated with AWS or other cloud services, that will increase the complexity of migrating to a different cloud provider. Note that this is not much different from migrating a regular application running on a cloud provider without Kubernetes.

On the other hand, ECS is a native AWS service, thus it can only run clusters and containers on AWS cloud. So in essence this locks in the applications to the AWS ecosystem as the ECS container setup cannot be easily migrated to another cloud provider or on-premises.

Fault Tolerance and Orchestrator Resilience

Kubernetes provides fault tolerance at the pod and container level, so if one containerized application fails it will be automatically restarted. To ensure orchestrator resiliency (VM-level fault tolerance), you can use Autoscaling Groups so AWS will automatically restart crashed VM instances and ensure a target number of instances is maintained. It is a good practice to put all master instances in one autoscaling group to ensure a quorum is maintained. The same can be done for worker nodes.

On the other hand, ECS transparently manages your cluster and ensures it is highly available. If a node in the cluster goes down, it will automatically be re-created and brought back to service. You only intervene when you want to change your physical setup, either growing or reducing the cluster size.

Third-party Tools

A growing number of third party tools (for example for DNS management or networking) developed by the community are available to extend Kubernetes functionalities.

On the other hand, although ECS provides a rich set of APIs, it is still a closed platform, thus not as prone to extensions via third party tools. Requests for new features and bug fixes will depend on Amazon’s roadmap and priorities.

Which One to Choose?

ECS excels when it comes to ease of deployment and maintenance. ECS does all the work behind the scenes. ECS is ideal for quick setup and providing multi-AZ HA out-of-the-box.

On the other hand, Kubernetes provides the flexibility to migrate or share the same configuration between on-premises and multiple cloud providers. It is also possible to extend its functionalities with custom tools, besides making use of a wealth of open source, community-led solutions to ease setup and deployment.

Getting Started with a Cluster on AWS EC2

Although it would be possible to set up a cluster by creating EC2 instances and installing all Kubernetes components on them, this could be quite error-prone and time-consuming. Many tools are available to start deploying a Kubernetes cluster on AWS, in particular:

  • Kubernetes Operations (kops) - This is a production grade tool used to install, upgrade and manage Kubernetes on AWS. It has support for different operating systems like Ubuntu, Debian, CentOS and RHEL. This tool has the ability to generate Terraform templates and support custom Kubernetes add-ons.

  • Tectonic Installer - This tool was developed by CoreOS to facilitate deployment of secure and highly available Kubernetes clusters on different infrastructure such as AWS, Azure, OpenStack, Google Cloud, and also bare metal. CoreOS is the base operating system for the nodes in a Kubernetes cluster installed by this tool.

  • kube-aws - This is a tool by Kubernetes Incubator which can create, update and destroy highly available Kubernetes clusters on AWS backed by multi-AZ deployment. This deployment tool is powered by various AWS services including CloudFormation, Key Management Service, Auto Scaling, Spot Fleet, EC2, ELB, S3, etc and an existing VPC can be used for the deployment.

In the next section we provide concrete steps to deploy a cluster using kops.

For further reading, see Kubernetes Documentation: Running Kubernetes on AWS EC2

Deploying Kubernetes on AWS Using Kops

Kops is a production grade tool used to install, upgrade, and operate highly available Kubernetes clusters on AWS and other cloud platforms using the command line. Kops is capable of generating Terraform templates with support for multiple CNI networking plugins and custom Kubernetes add-ons.

Installing a Kubernetes Cluster on AWS

Before proceeding, make sure to have installed kubectl , kops , and AWS cli tools.


Configure AWS Client with Access Credentials

Make sure AWS IAM user has the following permissions for kops to function properly:

- AmazonEC2FullAccess
- AmazonRoute53FullAccess
- AmazonS3FullAccess
- IAMFullAccess
- AmazonVPCFullAccess

Configure AWS cli with this user’s credentials by running:

# aws configure


Create S3 Bucket for Cluster State Storage

Create a dedicated S3 bucket which will be used by kops to store the state representing the cluster. We’ll name this bucket my-cluster-state :

# aws s3api create-bucket --bucket my-cluster-state

Make sure to activate bucket versioning to be able to later recover or revert to a previous state:

# aws s3api put-bucket-versioning --bucket my-cluster-state --versioning-configuration Status=Enabled


DNS Setup

On the DNS side, you can go with either public or private DNS. For public DNS, a valid top-level domain or subdomain is required to create the cluster. DNS is required by worker nodes to discover the master and by the master to discover all the etcd servers. For a domain whose registrar is not AWS, create a Route 53 hosted zone on AWS and change nameserver records on your registrar accordingly.

In this example we’ll be using a simple, private DNS to create a gossip-based cluster . The only requirement to set this up is for our cluster name to end with k8s.local .


Creating the Kubernetes Cluster

Note that all kops commands below that include --yes option can be run first without it to just show which changes would take place (for example, which AWS resources will get created or destroyed when running the command with --yes option).

The following command will create a 1 master (an m3.medium instance) and 2 nodes (two t2.medium instances) cluster in us-west-2a availability zone:

# kops create cluster \ --name my-cluster.k8s.local \ --zones us-west-2a \ --dns private \ --master-size=m3.medium \ --master-count=1 \ --node-size=t2.medium \ --node-count=2 \ --state s3://my-cluster-state \ --yes

Some of the command options in the above example have default values: --master-size , --master-count , --node-size , and --node-count . We’ve used the default values so the end result would be the same if we hadn’t specified those options. Also note that kops will create one master node in each availability zone specified, so this option: --zones us-west-2a,us-west-2b would result in 2 master nodes, one in each of the two zones (even if --master-count was not specified in the command line).

Note that cluster creation may take a while as instances must boot, download the standard Kubernetes components and reach a "ready" state. Kops provides a command to check the state of the cluster and check it’s ready:

# kops validate cluster --state=s3://my-cluster-state Using cluster from kubectl context: my-cluster.k8s.local Validating cluster my-cluster.k8s.local INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master m3.medium 1 1 us-west-2a nodes Node t2.medium 2 2 us-west-2a NODE STATUS NAME ROLE READY ip-172-20-32-203.us-west-2.compute.internal node True ip-172-20-36-109.us-west-2.compute.internal node True ip-172-20-61-137.us-west-2.compute.internal master True Your cluster my-cluster.k8s.local is ready

If you want to make some changes to the cluster, do so by running:

# kops edit cluster my-cluster.k8s.local # kops update cluster my-cluster.k8s.local --yes


Upgrading the Cluster to a Later Kubernetes Release

Kops can upgrade an existing cluster (master and nodes) to the latest recommended release of Kubernetes without having to specify the exact version. Kops supports rolling cluster upgrades where the master and worker nodes are upgraded one by one.

1. Update Kubernetes

# kops upgrade cluster \ --name $NAME \ --state s3://my-cluster-state \ --yes

2. Update the state store to match the cluster state.

# kops update cluster \ --name my-cluster.k8s.local \ --state s3://my-cluster-state \ --yes

3. Perform the rolling update.

# kops rolling-update cluster \ --name my-cluster.k8s.local \ --state s3://my-cluster-state \ --yes

This will perform updates on all instances in the cluster, first master and then workers.


Delete the Cluster

To destroy an existing cluster that we used for experimenting or trials for example, we can run:

# kops delete cluster my-cluster.k8s.local \ --state=s3://my-cluster-state \ --yes
For further reading, see AWS Documentation: Manage Kubernetes Clusters on AWS Using Kops

Using Kubernetes EKS Managed Service

Amazon Elastic Container Service for Kubernetes (EKS) is a fully managed service that takes care of all the cluster setup and creation, ensuring multi-AZ support on all clusters and automatic replacement of unhealthy instances (master or worker nodes). It also patches and upgrades clusters to the latest recommended Kubernetes release without requiring any intervention.

While EKS provides similar levels of integration with other Amazon services as ECS, it relies on Kubernetes open orchestration model instead of an AWS specific model. This increases the portability of clusters deployed on EKS to other cloud providers. The key contention for such a migration would be the level of coupling with native AWS services, but at least the orchestration side would be easier.

By default clusters in EKS consist of 3 masters spread across 3 different availability zones to protect against the failure of a single AWS availability zone:

Worker nodes are launched on the AWS user’s own EC2 instances, thus not shared with other tenants. In order to use tools such as kubectl , access to master instances must be set up via IAM authenticated public endpoints or through AWS PrivateLink . With AWS PrivateLink, masters appear as an elastic network interface with private IP addresses in the Amazon VPC. This allows to the masters and the EKS service directly from the Amazon VPC, without using public IP addresses or requiring the traffic to traverse the internet.

Amazon EKS also integrates tightly with other AWS services such as ELB for load balancing, or AWS CloudTrail for logging.

Standing up a new Kubernetes cluster with EKS can be done simply using the AWS Management Console. After getting access to the cluster, containerized applications can be scheduled in the new cluster in the same fashion as with any other Kubernetes installation:

For further reading, see AWS documentation: Amazon EKS

Launching Kubernetes on EC2 Using Rancher

Rancher is a complete container management platform that eases deployment of Kubernetes and containers. Rancher natively supports Kubernetes and allows users to control its features through a simple UI, including updates to the latest stable release. It integrates with LDAP, AD, and GitHub for authentication. Rancher also provides an application catalog with over 90 popular Docker applications .

Setting Up Rancher in AWS

Rancher (the application) runs on RancherOS, which is available as an Amazon Machine Image (AMI), and thus can be deployed on any EC2 instance.


Create RancherOS Instance on EC2

After installing and configuring AWS CLI tool, proceed to create an EC2 instance using RancherOS AMI. Check RancherOS documentation for AMI ids for each region. For example this command:


$ aws ec2 run-instances --image-id ami-12db887d --count 1 --instance-type t2.micro --key-name my-key-pair --security-groups my-sg

will create one new t2.micro EC2 instance with RancherOS on ap-south-1 AWS region. Make sure to use the correct key name and security group. Also make sure the security group enables traffic to TCP port 8080 to the new instance.


Start Rancher Server

When the new instance is ready, just connect using ssh and start the Rancher server:


$ sudo docker  run  --name rancher-server -d --restart=unless-stopped \  
  -p 8080:8080 rancher/server:stable

This might take a few minutes. Once done, the UI can be accessed on port 8080 of the EC2 instance . Since by default anyone can access Rancher’s UI and API, it is recommended to set up access control.

Creating a Kubernetes cluster via Rancher in AWS


Configure Kubernetes environment template

An environment in Rancher is a logical entity for sharing deployments and resources with different sets of users. Environments are created from templates. Rancher’s application catalogue already includes templates for Kubernetes that can be selected and modified to configure, among other: disabling add-ons (Rancher installs by default: Helm , Dashboard and SkyDNS ), enabling backups , and selecting the cloud provider for managing load balancers, nodes and networking routes.


Create the Kubernetes Cluster (environment)

Adding a Kubernetes environment is just a matter of selecting the adequately configured template for our use case and inputting the cluster name. If access control is turned on, we can add members and select their membership role . Anyone added to the membership list would have access to the environment.


Add Hosts to Kubernetes Cluster

We need to add at least one host to the newly created Kubernetes environment. In this case, the hosts will be previously created AWS EC2 instances.

Once the first host has been added, Rancher will automatically start the deployment of the infrastructure (master) including Kubernetes services (i.e. kubelet, etcd, kube-proxy, etc). Hosts that will be used as Kubernetes nodes will require TCP ports 10250 and 10255 to be open for kubectl. Make sure to review the full list of Rancher requirements for the hosts .

It might take a few minutes for the Kubernetes cluster setup/update to complete, after adding hosts to Kubernetes environment:

Deploying Applications in the Kubernetes Cluster

Once the cluster is ready containerized applications can be deployed using either the Rancher application catalog or kubectl. kubectl needs to be configured via the Rancher UI in order for deployment information to become visible in Rancher’s dashboards.

For further reading, see Rancher documentation: Kubernetes

Scheduling Kubernetes Resources on AWS Using Terraform

Terraform is an infrastructure-as-code tool used for building, changing, and versioning infrastructure safely and efficiently. It can be used to deploy containerized applications into an properly configured Kubernetes cluster running in AWS.

Terraform uses its own configuration language and by default looks for resource specifications in the same directory where the terraform commands are being executed.

Specifying Kubernetes as Provider

Terraform needs to be informed of the Kubernetes cluster configuration. For example this Terraform configuration file (extension .tf ):

provider "kubernetes" { host = "https://104.196.242.174" username = "my-user" password = "my-password" client_certificate = "${file("~/.kube/client-cert.pem")}" client_key = "${file("~/.kube/client-key.pem")}" cluster_ca_certificate = "${file("~/.kube/cluster-ca-cert.pem")}" }

tells Terraform that the Kubernetes master is located at the host IP address and provides the AWS credentials and certificates to ssh to it. We can then install the Terraform plugin for Kubernetes provider by running the command:

# terraform init

Specifying and Deploying Pods and Services

Kubernetes resources like pods and services can be created using Terraform’s configuration language (which then gets translated transparently by Terraform to actual Kubernetes specifications). For example to create a single (nginx) pod and a service selecting this pod:

resource "kubernetes_pod" "nginx" { metadata { name = "nginx-example" labels { App = "nginx" } } spec { container { image = "nginx:1.7.8" name = "example" port { container_port = 80 } } } } resource "kubernetes_service" "nginx" { metadata { name = "nginx-example" } spec { selector { App = "${kubernetes_pod.nginx.metadata.0.labels.App}" } port { port = 80 target_port = 80 } type = "LoadBalancer" } }

Running the terraform plan command will display the list of actions (resources to destroy, change or create) that Terraform will perform based on the above configuration:

+ kubernetes_pod.nginx + kubernetes_service.nginx Plan: 2 to add, 0 to change, 0 to destroy.

Note that the whole configuration of both the pod and service are displayed, but were omitted here for brevity.

Finally, to actually deploy the pod and service, the terraform apply command must be executed. This will create resources via the API in the right order, supplying any defaults as necessary and waiting for resources to finish provisioning to the point when it can either present useful attributes or a failure (with reason) to the user.

For further reading, see Terraform documentation: Getting Started with Kubernetes provider

Other Options for Deploying Kubernetes in the Cloud

Besides the Kubernetes deployment options already covered, there are other tools that can be used to deploy Kubernetes on public clouds like AWS. Each tool has its unique features and building blocks:


  • Heptio - Heptio provides a solution based on CloudFormation and kubeadm to deploy Kubernetes on AWS, and supports multi-AZ. Heptio is suitable for users already familiar with CloudFormation AWS orchestration tool.

  • Kismatic Enterprise Toolkit (KET) - KET is a collection of tools with sensible defaults which are production-ready to create enterprise-tuned clusters of Kubernetes. The goal with this toolkit is to make it easy for organizations to install and manage their Kubernetes infrastructure and clusters.

  • kubeadm - The kubeadm project is focused on a solution to build a simple cluster on AWS using Terraform. It is an adequate tool for tests and proof-of-concepts only as it doesn’t support multi-AZ and other advanced features.

  • OpenShift - This is a Red Hat platform-as-a-service product for container-based deployment and management of software. There is an open source version called OpenShift Origin which adds developer and operations-centric tools on top of Kubernetes to enable rapid application development, easy deployment and scaling, and long-term lifecycle maintenance for small and large teams.

  • Stackpoint - This is a web based solution that provides a user friendly platform to provision Kubernetes on various cloud providers such as AWS, Google Cloud Platform, Microsoft Azure and Digital Ocean. This is a good tool for those using more than one cloud provider and would like a single place for managing their multi-cloud Kubernetes deployments.

  • Tack - This is a terraform module that can be used to create Kubernetes clusters which run on any version of CoreOS on AWS. Supports multi-AZ deployments of worker nodes that are able to auto-scale.

  • Tectonic - Tectonic enables an automated installation of Kubernetes with the goals of being secure by default, quick and easy to install clusters, highly available, modular and customizable. It also focuses on portability (runs on any operating system), and flexibility of deployment to multiple cloud providers such as AWS, Google Cloud Platform, or Microsoft Azur.

Summary

AWS is a premiere solution for running cloud native apps, but setting up Kubernetes to run on it can be complex. This is where Kubernetes deployment tools like Kops come in. Amazon itself offers alternatives to decrease the operational overhead of setting up Kubernetes: on one hand there’s the Elastic Container Service (ECS) which is a container orchestration service focused on high availability (HA) out-of-the-box but is not portable to other infrastructure providers. There’s also the Amazon Elastic Container Service for Kubernetes (EKS) which is compatible with existing Kubernetes configurations and provides HA across availability zones by default. Rancher and Terraform are tools than can help accelerate deployment of applications in Kubernetes clusters. Rancher’s forte is their application catalog that allows deploying standard and custom applications with a few clicks. Terraform can be useful for adopting a unified infrastructure configuration language across providers.


Further Reading



Top Kubernetes AWS Tutorials from the Community

Tutorial by: AWS

Length: Short

Can help you learn: Kubernetes lifecycle management with Kops

Tutorial Steps:

  1. Kubernetes and Kops overview

  2. DNS configuration

  3. Creating a Kubernetes cluster

  4. Upgrading a Kubernetes cluster

  5. Deleting a Kubernetes cluster

Tutorial by: Codefresh

Length: Short

Can help you learn: Creating a Kubernetes cluster on AWS using Kops and deploying apps using Codefresh

Tutorial Steps:

  1. Setup environment
  2. Deploy Kubernetes on AWS
  3. Manage Kubernetes and deploy containers using Codefresh UI

Tutorial by: Datawire Inc

Length: Short

Can help you learn: How to use Terraform to deploy a production-quality Kubernetes cluster

Tutorial steps:

  1. Technical design of Kubernetes on AWS
  2. AWS EC2 setup for Kubernetes
  3. Using Kops and Terraform to generate the cluster
  4. Reducing cost of AWS instances used in the cluster

Tutorial by: Deepak Vohra

Length: Short

Can help you learn: Deploying Kubernetes on AWS with CloudFormation and CoreOS

Tutorial steps:

  1. Prepare AWS environment
  2. Create a CloudFormation stack for the Kubernetes cluster
  3. Configuring DNS

Tutorial by: OpenCredo

Length: Short

Can help you learn: Deploying Kubernetes on AWS with Terraform and Ansible

Tutorial steps:

  1. Prepare AWS environment using Ansible
  2. Deploy and manage Kubernetes using Terraform templates


Kubernetes on AWS Videos





  • No labels