Example: Deploying Cassandra with Stateful Sets

This tutorial shows you how to develop a native cloud Cassandra deployment on Kubernetes. In this instance, a custom Cassandra SeedProvider enables Cassandra to discover new Cassandra nodes as they join the cluster.

Deploying stateful distributed applications, like Cassandra, within a clustered environment can be challenging. StatefulSets greatly simplify this process. Please read about StatefulSets for more information about the features used in this tutorial.

Cassandra Docker

The pods use the gcr.io/google-samples/cassandra:v12 image from Google’s container registry. The docker is based on debian:jessie and includes OpenJDK 8. This image includes a standard Cassandra installation from the Apache Debian repo. By using environment variables you can change values that are inserted into cassandra.yaml.

ENV VAR	DEFAULT VALUE
CASSANDRA_CLUSTER_NAME	‘Test Cluster’
CASSANDRA_NUM_TOKENS	32
CASSANDRA_RPC_ADDRESS	0.0.0.0

Objectives
Before you begin
- Additional Minikube Setup Instructions
Creating a Cassandra Headless Service
- Validating (optional)
Using a StatefulSet to Create a Cassandra Ring
Validating The Cassandra StatefulSet
Modifying the Cassandra StatefulSet
Cleaning up
What’s next

Objectives

Create and Validate a Cassandra headless Service.
Use a StatefulSet to create a Cassandra ring.
Validate the StatefulSet.
Modify the StatefulSet.
Delete the StatefulSet and its Pods.

Before you begin

To complete this tutorial, you should already have a basic familiarity with Pods, Services, and StatefulSets. In addition, you should:

Install and Configure the kubectl command line
Download cassandra-service.yaml and cassandra-statefulset.yaml
Have a supported Kubernetes Cluster running

Note: Please read the getting started guides if you do not already have a cluster.

Additional Minikube Setup Instructions

Warning: Minikube defaults to 1024MB of memory and 1 CPU which results in an insufficient resource errors.

To avoid these errors, run minikube with:

minikube start --memory 5120 --cpus=4

Creating a Cassandra Headless Service

A Kubernetes Service describes a set of Pods that perform the same task.

The following Service is used for DNS lookups between Cassandra pods and clients within the Kubernetes Cluster.

cd to the folder you saved the .yaml files.
Create a Service to track all Cassandra StatefulSet Nodes from the following .yaml file:

kubectl create -f cassandra-service.yaml

`cassandra-service.yaml`
`apiVersion: v1 kind: Service metadata: labels: app: cassandra name: cassandra spec: clusterIP: None ports: - port: 9042 selector: app: cassandra`

Validating (optional)

Get the Cassandra Service.

kubectl get svc cassandra

The response should be

NAME        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   None         <none>        9042/TCP   45s

If anything else returns, the service was not successfully created. Read Debug Services for common issues.

Using a StatefulSet to Create a Cassandra Ring

The StatefulSet manifest, included below, creates a Cassandra ring that consists of three pods.

Note: This example uses the default provisioner for Minikube. Please update the following StatefulSet for the cloud you are working with.

Update the StatefulSet if necessary.
Create the Cassandra StatefulSet from the following .yaml file:

kubectl create -f cassandra-statefulset.yaml

`cassandra-statefulset.yaml`
apiVersion: "apps/v1beta1" kind: StatefulSet metadata: name: cassandra spec: serviceName: cassandra replicas: 3 template: metadata: labels: app: cassandra spec: containers: - name: cassandra image: gcr.io/google-samples/cassandra:v12 imagePullPolicy: Always ports: - containerPort: 7000 name: intra-node - containerPort: 7001 name: tls-intra-node - containerPort: 7199 name: jmx - containerPort: 9042 name: cql resources: limits: cpu: "500m" memory: 1Gi requests: cpu: "500m" memory: 1Gi securityContext: capabilities: add: - IPC_LOCK lifecycle: preStop: exec: command: ["/bin/sh", "-c", "PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done"] env: - name: MAX_HEAP_SIZE value: 512M - name: HEAP_NEWSIZE value: 100M - name: CASSANDRA_SEEDS value: "cassandra-0.cassandra.default.svc.cluster.local" - name: CASSANDRA_CLUSTER_NAME value: "K8Demo" - name: CASSANDRA_DC value: "DC1-K8Demo" - name: CASSANDRA_RACK value: "Rack1-K8Demo" - name: CASSANDRA_AUTO_BOOTSTRAP value: "false" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP readinessProbe: exec: command: - /bin/bash - -c - /ready-probe.sh initialDelaySeconds: 15 timeoutSeconds: 5 # These volume mounts are persistent. They are like inline claims, # but not exactly because the names need to match exactly one of # the stateful pod volumes. volumeMounts: - name: cassandra-data mountPath: /cassandra_data # These are converted to volume claims by the controller # and mounted at the paths mentioned above. # do not use these in production until ssd GCEPersistentDisk or other ssd pd volumeClaimTemplates: - metadata: name: cassandra-data annotations: volume.beta.kubernetes.io/storage-class: fast spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi --- kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: k8s.io/minikube-hostpath parameters: type: pd-ssd

cassandra-statefulset.yaml Copy cassandra-statefulset.yaml to clipboard

apiVersion: "apps/v1beta1"
kind: StatefulSet
metadata:
  name: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v12
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
           cpu: "500m"
           memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done"]
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: CASSANDRA_AUTO_BOOTSTRAP
            value: "false"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
      annotations:
        volume.beta.kubernetes.io/storage-class: fast
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd

Validating The Cassandra StatefulSet

Get the Cassandra StatefulSet:

kubectl get statefulset cassandra

The response should be

NAME        DESIRED   CURRENT   AGE
cassandra   3         0         13s

The StatefulSet resource deploys pods sequentially.

Get the Pods to see the ordered creation status:

kubectl get pods -l="app=cassandra"
NAME          READY     STATUS              RESTARTS   AGE
cassandra-0   1/1       Running             0          1m
cassandra-1   0/1       ContainerCreating   0          8s

Note: It can take up to ten minutes for all three pods to deploy.

Once all pods are deployed, the same command returns:

kubectl get pods -l="app=cassandra"
NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          10m
cassandra-1   1/1       Running   0          9m
cassandra-2   1/1       Running   0          8m

Run the Cassandra utility nodetool to display the status of the ring.

kubectl exec cassandra-0 -- nodetool status

Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.4.2.4  65.26 KiB  32           63.7%             a9d27f81-6783-461d-8583-87de2589133e  Rack1-K8Demo
UN  10.4.0.4  102.04 KiB  32           66.7%             5559a58c-8b03-47ad-bc32-c621708dc2e4  Rack1-K8Demo
UN  10.4.1.4  83.06 KiB  32           69.6%             9dce943c-581d-4c0e-9543-f519969cc805  Rack1-K8Demo

Modifying the Cassandra StatefulSet

Use kubectl edit to modify the size of of a Cassandra StatefulSet.

Run the following command:

kubectl edit statefulset cassandra

This command opens an editor in your terminal. The line you need to change is Replicas.

Note: The following sample is an excerpt of the StatefulSet file.

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  creationTimestamp: 2016-08-13T18:40:58Z
  generation: 1
  labels:
    app: cassandra
  name: cassandra
  namespace: default
  resourceVersion: "323"
  selfLink: /apis/apps/v1beta1/namespaces/default/statefulsets/cassandra
  uid: 7a219483-6185-11e6-a910-42010a8a0fc0
spec:
  replicas: 3

Increase the number of replicas to 4, and then save the manifest.

The StatefulSet now contains 4 pods.
Get the Cassandra StatefulSet to verify:

kubectl get statefulset cassandra

The response should be

NAME        DESIRED   CURRENT   AGE
cassandra   4         4         36m

Cleaning up

Deleting or scaling a StatefulSet down does not delete the volumes associated with the StatefulSet. This ensures safety first: your data is more valuable than an auto purge of all related StatefulSet resources.

Warning: Depending on the storage class and reclaim policy, deleting the Persistent Volume Claims may cause the associated volumes to also be deleted. Never assume you’ll be able to access data if its volume claims are deleted.

Run the following commands to delete everything in a StatefulSet:

grace=$(kubectl get po cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \
  && kubectl delete statefulset -l app=cassandra \
  && echo "Sleeping $grace" \
  && sleep $grace \
  && kubectl delete pvc -l app=cassandra

Run the following command to delete the Cassandra Service.

kubectl delete service -l app=cassandra

What’s next

Learn how to Scale a StatefulSet.
Learn more about the KubernetesSeedProvider
See more custom Seed Provider Configurations

Create an Issue Edit this Page

Tutorials

Detailed walkthroughs of common Kubernetes operations and workflows.

Example: Deploying Cassandra with Stateful Sets

Objectives

Before you begin

Additional Minikube Setup Instructions

Creating a Cassandra Headless Service

Validating (optional)

Using a StatefulSet to Create a Cassandra Ring

Validating The Cassandra StatefulSet

Modifying the Cassandra StatefulSet

Cleaning up

What’s next