Turorial - Understanding Kubernetes volumes

imt

1. Objective

Understand different types of Kubernetes volumes.

This tutorial directly uses a subset of the examples of the book “Kubernetes in action” written by Marko Lukša. All examples of the book can be found here.

2. Volume of type emptyDir

We are going to create a pod with 2 containers: a Nginx web server, and an HTML generator that generates a file index.html

Create a file 9-fortune-pod.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: fortune
spec:
  volumes:
  - name: html
    emptyDir: {}
  containers:
  - image: luksa/fortune
    name: html-generator
    volumeMounts:
    - name: html
      mountPath: /var/htdocs
  - image: nginx:alpine
    name: web-server
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
      readOnly: true
    ports:
    - containerPort: 80
      protocol: TCP

> kubectl create -f 9-fortune-pod.yaml

The pod contains two containers and a single volume that’s mounted in both of them, yet at different paths.

To access the Nginx server externally we can forward the port (you can also create a service):

> kubectl port-forward fortune 8080:80

In another terminal we can request the web server:

> curl http://localhost:8080

You will get a different message every ten seconds.

3. Volume of type gcePersistentDisk

Get the cluster list from gcloud.

> gcloud container clusters list

NAME  LOCATION       MASTER_VERSION      MASTER_IP     MACHINE_TYPE   NODE_VERSION        NUM_NODES  STATUS
zeus  us-central1-c  1.27.8-gke.1067004  34.70.44.223  n1-standard-2  1.27.8-gke.1067004  3          RUNNING

Create a compute disk of size 1GB in the zone us-central1-c called mongodb

> gcloud compute disks create --size=1GiB --zone=us-central1-c mongodb

WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.
Created [https://www.googleapis.com/compute/v1/projects/login-kubernetes/zones/us-central1-c/disks/mongodb].
NAME     ZONE           SIZE_GB  TYPE         STATUS
mongodb  us-central1-c  1        pd-standard  READY

New disks are unformatted. You must format and mount a disk before it
can be used. You can find instructions on how to do this at:

https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting

Create a file 10-mongodb-pod-gcepd.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: mongodb
spec:
  volumes:
  - name: mongodb-data
    gcePersistentDisk:
      pdName: mongodb
      fsType: ext4
  containers:
  - image: mongo
    name: mongodb
    volumeMounts:
    - name: mongodb-data
      mountPath: /data/db
    ports:
    - containerPort: 27017
      protocol: TCP

> kubectl create -f 10-mongodb-pod-gcepd.yaml

We are going to add entries in the mongodb disk. To do so we execute bash on the pod.

> kubectl exec -it mongodb — bash

We start mongosh

mongodb> mongosh

We create a new store

mongodb-mongosh> use mystore

We add one entry in our store

mongodb-mongosh> db.foo.insertOne({name:'foo'})

We check that the entry has been added

mongodb-mongosh> db.foo.find()

Exit mongosh and exit the bash on the pod.

Now we can delete the pod and create it again to see if we find again our entry.

kubectl delete pod mongodb

`> kubectl create -f 10-mongodb-pod-gcepd.yaml`

> kubectl exec -it mongodb — bash

mongodb> mongosh

mongodb-mongosh> use mystore

mongodb-mongosh> db.foo.find()

4. Volume claim

4.1. Static volume claim

With the type of volume persistentVolumeClaim we can avoid having references and knowledge about volumes within pods specification. We first need to define a persistent volume. This is typically defined by an admin of the cluster.

Create the manifest 11-mongodb-pv-gcepd.yaml with the following content:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-pv
spec:
  capacity:
    storage: 1Gi
  accessModes: # how the PV can be accessed by nodes/pods
    - ReadWriteOnce
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain # what to do with the PV when the claim is released
  gcePersistentDisk: # name, type, location etc. of the PV
    pdName: mongodb
    fsType: ext4

> kubectl create -f 11-mongodb-pv-gcepd.yaml

> kubectl get pv

NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mongodb-pv   1Gi        RWO,ROX        Retain           Available                                   6s

Second, as a user/developer we have to define a volume claim. A volume claim is a resource created independently of the pod.

Create a manifest 12-mongodb-pvc.yaml with the following content:

apiVersion: v1
kind: PersistentVolumeClaim # a claim for storage
metadata:
  name: mongodb-pvc
spec:
  resources:
    requests:
      storage: 1Gi # requested storage (have to be compatible with the PV)
  accessModes:
  - ReadWriteOnce # have to be compatible with the PV
  storageClassName: "" # we will use that later

> kubectl create -f 12-mongodb-pvc.yaml

> kubectl get pvc

NAME          STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb-pvc   Bound    mongodb-pv   1Gi        RWO,ROX                       5s

> kubectl get pv

NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
mongodb-pv   1Gi        RWO,ROX        Retain           Bound    default/mongodb-pvc                           69s

Finally, in the pod specification, we need to specify a volume of type persistentVolumeClaim and a reference to our previously created PVC resource.

Create a manifest 13-mongodb-pod-pvc.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: mongodb
spec:
  volumes:
  - name: mongodb-data
    persistentVolumeClaim: # type persistentVolumeClaim
      claimName: mongodb-pvc # the name of the PVC to use
  containers:
  - image: mongo
    name: mongodb
    volumeMounts:
    - name: mongodb-data
      mountPath: /data/db
    ports:
    - containerPort: 27017
      protocol: TCP

> kubectl create -f 13-mongodb-pod-pvc.yaml

> kubectl exec -it mongodb — bash

mongodb> mongosh

mongodb-mongosh> use mystore

mongodb-mongosh> db.foo.find()

  • Nothing in the pod specification relates to knowledge on the hardware or available volumes.

  • PV and PVC can be reused on different clusters.

4.2. dynamic volume claim

This is a mechanism to use by administrators to dynamically create persistent storages. Two elements have to be created:

  • a PersistentVolume provisioner, which is a piece of code (following a given interface) to dynamically create disks

  • a StorageClass definition within Kuebernetes that uses the required PV provisioner and that is used within the pod specification of users/developers

Let’s define a StorageClass that uses an existing PV provisioner on GCP.

Create the manifest 14-storageclass-fast-gcepd.yaml with the following content:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd # an existing PV provisioner on GCE is used
parameters:
  type: pd-ssd

> kubectl create -f 14-storageclass-fast-gcepd.yaml

Now we’ll define the user PVC 15-mongodb-pvc-dp.yaml as follows

apiVersion: v1
kind: PersistentVolumeClaim #PVC definition
metadata:
  name: mongodb-pvc2
spec:
  storageClassName: fast # the storage class fast is used
  resources:
    requests:
      storage: 100Mi
  accessModes:
    - ReadWriteOnce

> kubectl create -f 15-mongodb-pvc-dp.yaml

> kubectl get pvc mongodb-pvc2

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb-pvc2   Bound    pvc-3188465f-78d0-45be-a984-900c142d3aaa   1Gi        RWO            fast           32s

> kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
mongodb-pv                                 1Gi        RWO,ROX        Retain           Bound    default/mongodb-pvc                            16m
pvc-3188465f-78d0-45be-a984-900c142d3aaa   1Gi        RWO            Delete           Bound    default/mongodb-pvc2   fast                    40s

> gcloud compute disks list

NAME                                          LOCATION       LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
mongodb                                       us-central1-c  zone            1        pd-standard  READY
pvc-3188465f-78d0-45be-a984-900c142d3aaa      us-central1-c  zone            1        pd-ssd       READY

You should see the dynamically created disk.

4.3. dynamic provisioning without storage class

> kubectl get sc

NAME                     PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
premium-rwo              pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   2m35s
standard                 kubernetes.io/gce-pd    Delete          Immediate              true                   2m33s
standard-rwo (default)   pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   2m34s

We can see that the standard storage class uses the provisioner kubernetes.io/gce-pd. This is of course because we are using a GKE Kubernetes cluster that the standard StorageClass is using a GCE provisioner.

To get more details about this default storage class type the following command:

> kubectl get sc standard -o yaml

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    components.gke.io/component-name: pdcsi
    components.gke.io/component-version: 0.16.18
    components.gke.io/layer: addon
    storageclass.kubernetes.io/is-default-class: "false"
  creationTimestamp: "2024-03-04T12:55:21Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: standard
  resourceVersion: "798"
  uid: 8237240a-70db-427e-adbf-31eba9136376
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate

Let’s create a pod with the StorageClass set to standard:

Create the following 16-mongodb-pvc-dp-nostorageclass.yaml manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-pvc3
spec:
  storageClassName: standard
  resources:
    requests:
      storage: 100Mi
  accessModes:
    - ReadWriteOnce

> kubectl create -f 16-mongodb-pvc-dp-nostorageclass.yaml

> kubectl get pvc mongodb-pvc3

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb-pvc3   Bound    pvc-621038be-8ba6-4b46-b818-703b5671ac85   1Gi        RWO            standard       64s

> kubectl get pv pvc-621038be-8ba6-4b46-b818-703b5671ac85

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-621038be-8ba6-4b46-b818-703b5671ac85   1Gi        RWO            Delete           Bound    default/mongodb-pvc3   standard                84s

> gcloud compute disks list

NAME                                          LOCATION       LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
mongodb                                       us-central1-c  zone            1        pd-standard  READY
pvc-3188465f-78d0-45be-a984-900c142d3aaa      us-central1-c  zone            1        pd-ssd       READY
pvc-621038be-8ba6-4b46-b818-703b5671ac85      us-central1-c  zone            1        pd-standard  READY

5. Clean and Destroy your persistent disk

Delete your created resources (you know how to do it) and created the persistent disk, for example with the following command:

> gcloud compute disks delete --zone=us-central1-c mongodb