Edit This Page

Determine the Reason for Pod Failure

This page shows how to write and read a Container termination message.

Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.

• Before you begin
• Writing and reading a termination message
• Setting the termination log file
• What’s next

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube, or you can use one of these Kubernetes playgrounds:

• Katacoda
• Play with Kubernetes

Writing and reading a termination message

In this exercise, you create a Pod that runs one container. The configuration file specifies a command that runs when the container starts.

termination.yaml
apiVersion: v1 kind: Pod metadata: name: termination-demo spec: containers: - name: termination-demo-container image: debian command: ["/bin/sh"] args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"]

1. Create a Pod based on the YAML configuration file:

    kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/termination.yaml
   

   In the YAML file, in the cmd and args fields, you can see that the container sleeps for 10 seconds and then writes “Sleep expired” to the /dev/termination-log file. After the container writes the “Sleep expired” message, it terminates.

2. Display information about the Pod:

    kubectl get pod termination-demo
   

   Repeat the preceding command until the Pod is no longer running.

3. Display detailed information about the Pod:

    kubectl get pod --output=yaml
   

   The output includes the “Sleep expired” message:

    apiVersion: v1
    kind: Pod
    ...
        lastState:
          terminated:
            containerID: ...
            exitCode: 0
            finishedAt: ...
            message: |
              Sleep expired
            ...
   

4. Use a Go template to filter the output so that it includes only the termination message:

    kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"

Setting the termination log file

By default Kubernetes retrieves termination messages from /dev/termination-log. To change this to a different file, specify a terminationMessagePath field for your Container.

For example, suppose your Container writes termination messages to /tmp/my-log, and you want Kubernetes to retrieve those messages. Set terminationMessagePath as shown here:

apiVersion: v1
kind: Pod
metadata:
  name: msg-path-demo
spec:
  containers:
  - name: msg-path-demo-container
    image: debian
    terminationMessagePath: "/tmp/my-log"

What’s next

• See the terminationMessagePath field in Container.
• Learn about retrieving logs.
• Learn about Go templates.