What Are Kubernetes Jobs?
Kubernetes is a container orchestration platform. It provides multiple options for controlling and managing pods and containers, using dynamic objects like Deployment, StatefulSet, and ReplicaSet. Each of these objects is operated by a controller. When a pod fails, the controller restarts or reschedules the pod to keep its services running.
A Kubernetes Job or CronJob defines a certain task (called a Job), and ensures that a certain number of pods shut down when the Job is complete. This can help automate and manage application workflows in Kubernetes.
Get background about basic Kubernetes concepts in our guides to:
In this article:
Kubernetes Job Types
Kubernetes provides three types of Jobs:
- Multiple parallel Jobs (work queues)—running multiple tasks in parallel. In many cases, it is impractical to complete one task and then start another. It can be much more efficient, and make better use of available computing resources, to support parallelism. To run multiple parallel Jobs, the
.spec.completionsfield in the Job declaration should not be undefined (it then defaults to 1).
- Parallel Jobs with a fixed number of completions—these are tasks that occur concurrently but run a limited number of times before they are considered done. You can define the number of times using the
.spec.completionsfield in the Job declaration. It is also possible to add indexes to assign each pod a fraction of the overall task for completion. Another option is to set the
spec.parallelismfield to define the number of parallel Jobs.
- Non-parallel Jobs—a Job that runs on one pod at a time. The Job launches one pod, and more pods are formed if there are startup failures. When the pod completes successfully, the Job is considered complete.
Two important points regarding Job declaration fields:
spec.completionscannot be zero, since a Job cannot run less than zero times.
spec.parallelismcan be zero, and in this case, the Job is scaled down to zero, meaning it is paused until the number is incremented.
- For both of these fields, the default value is 1, and you cannot use negative values.
Kubernetes Jobs Examples
The following are examples of a Kubernetes Job and CronJob.
Kubernetes Job Example
The following Job config computes and prints out Pi to 1000 places. It should take about 10 seconds to run:
apiVersion: batch/v1 kind: Job metadata: name: pi spec: template: spec: containers: —name: pi image: pic1 command: ["pic1", "-Mbignum=bpi", "-wle", "print bpi(1000)"] restartPolicy: Never backoffLimit: 4
You can use the following command to run this Job:
kubectl apply -f https://kubernetes.io/examples/controllers/Job.yaml
The output should look like this:
The pod definition will begin with a typical Kubernetes config kind, metadata, and apiVersion. The spec includes a pod template identical to the pod definition, minus the apiVersion and kind fields.
Note that you cannot set the restart policy to “always.” The Job mustn’t restart the pod after successful termination. The available restart policy options are “on failure” and “never.”
Kubernetes CronJob Example
A CronJob is useful for performing periodic, scheduled tasks like generating backups and reports. It is important to configure each task, so it recurs indefinitely. You can schedule CronJobs to execute at regular intervals: i.e., daily, weekly, or monthly. You define the time during each interval that the CronJob should initiate.
The following example of a CronJob manifest will print the current time with a message saying “Working” every hour during business operating hours:
apiVersion: batch/v1 kind: CronJob metadata: name: working spec: schedule: "* * * * *" JobTemplate: spec: template: spec: containers: —name: working image: busybox:1.29 imagePullPolicy: IfNotPresent command: —/bin/sh —-c —date; echo Working from the Kubernetes cluster restartPolicy: OnFailure
The CronJob’s schedule should follow this syntax:
# ┌───────────── hour (8—21) # │ ┌───────────── day of the month (1—31) # │ │ ┌───────────── month (1—12) # │ │ │ ┌───────────── day of the week (2—6) (Monday to Friday; # │ │ │ │ # │ │ │ │ # │ │ │ │ # * * * *
Kubernetes Job Failure and Concurrency Considerations
Kubernetes Jobs are highly suitable for services that are sensitive to shutdown and need to run only once, or a specified number of times. If the pod fails, the Kubernetes Job restarts the pod’s containers, restarting the operation.
A Job can restart or reschedules a pod to another node for the following reasons:
- The node crashed, was rebooted or upgraded
- The pod consumed more resources than the node
spec.template.spec.restartPolicy field governs how Kubernetes will behave when a pod in a Job needs to be restarted:
- If you set
OnFailure, and there are repeated failures of the pod, the Jobs controller will retry the pod at increasing time intervals—10 seconds, 20 seconds, 40 seconds, etc. After six minutes, the controller gives up and stops trying to restart the pod. This is the default behavior—you can customize it using the
- If you set
OnFailure, the container shuts down when the backoff limit is reached. This can make it difficult to track down the cause of a Job failure. In this case, you can set
restartPolicyto Never to start debugging the problem.
- Note that even if you set both .spec.parallelism and .spec.completions to 1 and
Never, this does not guarantee that the Job won’t run the process multiple times. This means that you need to ensure any process running as part of a Job can support premature termination and multiple instances running in parallel.
Kubernetes Job Deletion and Cleanup
When a Kubernetes Job terminates, neither the Job nor the created Pods are automatically deleted. You must manually delete them. The Jobs mechanism works this way to allow you to continue to view logs and status of completed Jobs and their pods.
You can delete a Job using kubectl like this:
kubectl delete jobs <job-name>
The above command deletes the specified Job and all its child pods. Alternatively, you can delete only the Job object but keep its pods alive:
kubectl delete jobs <job-name> cascade=false
Kubernetes Security with Aqua
Aqua Cloud Native Security Platform with the feature Kubernetes Security Posture Management (KSPM) ensures ongoing secure configuration with built-in CIS benchmarks, least privilege RBAC, pen-testing, and pod deployment policies.
Protect container-based cloud native applications from development to production, using the industry’s most advanced container security solution.