Локальный том и тома в Kubernetes

Sometimes, you don’t have block volumes provisioning available on your Kubernetes cluster. In particular when your cluster is running on premise / outside the clouds providers. In this article i‘m gonna introduce you to one great feature called “local” volume and how to provision automatically local Pv with the local volume static provisionner.

Creating the local volume persistent volume claim

Local volumes must be statically created as a persistent volume claim (PVC)
to be accessed by the pod.

Prerequisites

• Persistent volumes have been created using the local volume provisioner.

Procedure

1. Create the PVC using the corresponding storage class:

   kind: PersistentVolumeClaimapiVersion: v1metadata: name: local-pvc-name (1)spec: accessModes: - ReadWriteOnce volumeMode: Filesystem (2) resources: requests: storage: 100Gi (3) storageClassName: local-sc (4)

   1	Name of the PVC.
   2	The type of the PVC. Defaults to Filesystem.
   3	The amount of storage available to the PVC.
   4	Name of the storage class required by the claim.

2. Create the PVC in the OpenShift Container Platform cluster, specifying the file
   you just created:

   $ oc create -f <local-pvc>.yaml

Attach the local claim

After a local volume has been mapped to a persistent volume claim
it can be specified inside of a resource.

Prerequisites

• A persistent volume claim exists in the same namespace.

Procedure

1. Include the defined claim in the resource spec. The following example
   declares the persistent volume claim inside a pod:

   apiVersion: v1kind: Podspec: ... containers: volumeMounts: - name: local-disks (1) mountPath: /data (2) volumes: - name: localpvc persistentVolumeClaim: claimName: local-pvc-name (3)

   1	The name of the volume to mount.
   2	The path inside the pod where the volume is mounted. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host /dev/pts files. It is safe to mount the host by using /host.
   3	The name of the existing persistent volume claim to use.

2. Create the resource in the OpenShift Container Platform cluster, specifying the file
   you just created:

   $ oc create -f <local-pod>.yaml

• Installing the Local Storage Operator
• Provisioning local volumes by using the Local Storage Operator
• Provisioning local volumes without the Local Storage Operator
• Creating the local volume persistent volume claim
• Attach the local claim
• Automating discovery and provisioning for local storage devices
• Using tolerations with Local Storage Operator pods
• Deleting the Local Storage Operator resources
  • Removing a local volume or local volume set
  • Uninstalling the Local Storage Operator

OpenShift Container Platform can be provisioned with persistent storage by using
local volumes. Local persistent volumes allow you to access local storage
devices, such as a disk or partition, by using the standard
persistent volume claim interface.

Local volumes can be used without manually scheduling pods to nodes
because the system is aware of the volume node constraints. However,
local volumes are still subject to the availability of the underlying node
and are not suitable for all applications.

kind: PersistentVolumeapiVersion: v1metadata: name: local labels: type: localspec: storageClassName: "" capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/data" persistentVolumeReclaimPolicy: Recycle

kubectl create -f local-persistent-volume-recycle.yaml

kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
local 2Gi RWO Recycle Available 33m

kind: PersistentVolumeClaimapiVersion: v1metadata: name: volume-claimspec: storageClassName: "" accessModes: - ReadWriteOnce resources: requests: storage: 1Gi

kubectl create -f volume-claim.yaml

kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
local 2Gi RWO Recycle Bound project/volume-claim 37m

kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
volume-claim Bound local 2Gi RWO 1m

---kind: PodapiVersion: v1metadata: name: nginx namespace: default labels:spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: "http-server" volumeMounts: - mountPath: "/usr/share/nginx/html" name: html volumes: - name: html persistentVolumeClaim: claimName: volume-claim

kubectl create -f nginx-pod-pvc.yaml

kubectl get pod nginx -o yaml | grep IP hostIP: 10.10.10.86 podIP: 172.30.5.2
curl 172.30.5.2:80
403 Forbidden

echo "Welcome to $(hostname)" > /data/index.html

 curl 172.30.5.2:80 Welcome to kubew05

kubectl delete pod nginx
pod "nginx" deleted
kubectl create -f nginx-pod-pvc.yaml
pod "nginx" created

kubectl get pod nginx -o yaml | grep podIP podIP: 172.30.5.2
curl 172.30.5.2
Welcome to kubew05

kubectl delete pvc volume-claim

kubectl get pv persistent-volume
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
local 2Gi RWO Recycle Available 57m

kind: PersistentVolumeapiVersion: v1metadata: name: local-retain labels: type: localspec: storageClassName: "" capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/data" persistentVolumeReclaimPolicy: Retain

kubectl create -f local-persistent-volume-retain.yaml
kubectl create -f volume-claim.yaml

kubectl exec -it nginx bash
root@nginx:/# echo "Hello World" > /usr/share/nginx/html/index.html
root@nginx:/# exit

kubectl delete pvc volume-claim

kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS AGE
local-retain 2Gi RWO Retain Released project/volume-claim 3m

apiVersion: v1kind: PersistentVolumemetadata: name: nfs-volumespec: storageClassName: "" capacity: storage: 1Gi accessModes: - ReadWriteOnce nfs: path: "/mnt/nfs" server: fileserver persistentVolumeReclaimPolicy: Recycle

kubectl create -f nfs-persistent-volume.yaml
persistentvolume "nfs" created
kubectl get pv nfs -o wide
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-volume 1Gi RWO Recycle Available 7s

kubectl create -f volume-claim.yaml

kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-volume 1Gi RWO Recycle Bound project/volume-claim 5m
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
volume-claim Bound nfs-volume 1Gi RWO 9s

apiVersion: extensions/v1beta1kind: Deploymentmetadata: generation: 1 labels: run: nginx name: nginx-pvcspec: replicas: 3 selector: matchLabels: run: nginx strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: run: nginx spec: containers: - image: nginx:latest imagePullPolicy: IfNotPresent name: nginx ports: - containerPort: 80 protocol: TCP name: "http-server" volumeMounts: - mountPath: "/usr/share/nginx/html" name: html volumes: - name: html persistentVolumeClaim: claimName: volume-claim dnsPolicy: ClusterFirst restartPolicy: Always

kubectl create -f nginx-pvc-template.yaml

kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-pvc-3474572923-3cxnf 1/1 Running 0 2m 10.38.5.89 kubew05
nginx-pvc-3474572923-6cr28 1/1 Running 0 6s 10.38.3.140 kubew03
nginx-pvc-3474572923-z17ls 1/1 Running 0 2m 10.38.5.90 kubew05

kubectl exec -it nginx-pvc-3474572923-3cxnf bash
root@nginx-pvc-3474572923-3cxnf:/# echo 'Hello from NFS!' > /usr/share/nginx/html/index.html
root@nginx-pvc-3474572923-3cxnf:/# exit

curl 10.38.5.89
Hello from NFS!
curl 10.38.5.90
Hello from NFS!
curl 10.38.3.140
Hello from NFS!

kind: PersistentVolumeClaimapiVersion: v1metadata: name: pvc-volume-selectorspec: storageClassName: "" accessModes: - ReadWriteMany resources: requests: storage: 1Gi selector: matchLabels: volumeName: "share01"

kubectl create -f pvc-volume-selector.yaml

kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-volume-selector Pending 5s
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
share01 1Gi RWX Recycle Available
share02 1Gi RWX Recycle Available 

kubectl label pv share00 volumeName="share01"
persistentvolume "share01" labeled

kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
share01 1Gi RWX Recycle Bound project/pvc-volume-selector
share02 1Gi RWX Recycle Available 

kind: StorageClassapiVersion: storage.k8s.io/v1beta1metadata: name: glusterfs labels:provisioner: kubernetes.io/glusterfsreclaimPolicy: Deleteparameters: resturl: "http://heketi:8080" volumetype: "replicate:3"

kubectl create -f gluster-storage-class.yaml
kubectl get sc
NAME PROVISIONER
glusterfs-storage-class kubernetes.io/glusterfs

kind: StorageClassapiVersion: storage.k8s.io/v1beta1metadata: name: default-storage-class labels: annotations: storageclass.kubernetes.io/is-default-class: "true"provisioner: kubernetes.io/glusterfsreclaimPolicy: Deleteparameters: resturl: "http://heketi:8080"

kubectl get sc
NAME PROVISIONER
default-storage-class (default) kubernetes.io/glusterfs
glusterfs-storage-class kubernetes.io/glusterfs

kind: StorageClassapiVersion: storage.k8s.io/v1beta1metadata: name: glusterfs-storage-class labels:provisioner: kubernetes.io/glusterfsreclaimPolicy: Deleteparameters: resturl: "http://heketi:8080" volumetype: "replicate:3"

kubectl create -f gluster-storage-class.yaml

yum install -y glusterfs-fuse

kind: PersistentVolumeClaimapiVersion: v1metadata: name: apache-volume-claimspec: accessModes: - ReadWriteMany resources: requests: storage: 500Mi storageClassName: glusterfs-storage-class

apiVersion: v1kind: Podmetadata: name: apache-gluster-pod labels: name: apache-gluster-podspec: containers: - name: apache-gluster-pod image: centos/httpd:latest ports: - name: web containerPort: 80 protocol: TCP volumeMounts: - mountPath: "/var/www/html" name: html volumes: - name: html persistentVolumeClaim: claimName: apache-volume-claim dnsPolicy: ClusterFirst restartPolicy: Always

kubectl create -f glusterfs-storage-class.yaml
kubectl create -f pvc-gluster.yaml
kubectl create -f apache-pod-pvc.yaml

kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
apache-volume-claim Bound pvc-4af76e0f 1Gi RWX glusterfs-storage-class 7m

kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS
pvc-4af76e0f 1Gi RWX Delete Bound apache-volume-claim glusterfs-storage-class

heketi-cli --server http://heketi:8080 volume list
Id:7ce4d0cbc77fe36b84ca26a5e4172dbe Name:vol_7ce4d0cbc77fe36b84ca26a5e4172dbe ...

kubectl delete pvc apache-volume-claim
kubectl get pvc,pv
No resources found.

kind: PersistentVolumeClaimapiVersion: v1metadata: name: redis-data-claimspec: storageClassName: default accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

kubectl create -f redis-data-claim.yaml
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-data-claim Bound pvc-eaab62e3 10Gi RWO default 1m

apiVersion: apps/v1beta1kind: Deploymentmetadata: name: redis-deploymentspec: replicas: 1 template: metadata: labels: name: redis role: master name: redis spec: containers: - name: redis image: kubernetes/redis:v1 env: - name: MASTER value: "true" ports: - containerPort: 6379 volumeMounts: - mountPath: /redis-master-data name: redis-data volumes: - name: redis-data persistentVolumeClaim: claimName: redis-data-claim

apiVersion: v1kind: Servicemetadata: name: redisspec: ports: - port: 6379 targetPort: 6379 nodePort: 31079 name: http type: NodePort selector: name: redis

kubectl create -f redis-deployment.yaml
kubectl create -f redis-service.yaml

kubectl get pods -a -o wide
NAME READY STATUS RESTARTS AGE IP NODE
redis-deployment-75466795f6-thtx4 1/1 Running 0 30s 10.38.5.62 kubew05

yum install -y nmap-ncat
nc -v kubew05 31079
Ncat: Version 6.40
Ncat: Connected to kubew05:31079.
ping
+PONG
set greetings "Hello from Redis!"
+OK
get greetings
$17
Hello from Redis!
logout

apiVersion: batch/v1kind: Jobmetadata: name: redis-benchspec: template: metadata: name: bench spec: containers: - name: bench image: clue/redis-benchmark restartPolicy: Never

kubectl create -f redis-benchmark.yaml

kubectl get job
NAME DESIRED SUCCESSFUL AGE
redis-bench 1 1 48s

kubectl get pods -a
NAME READY STATUS RESTARTS AGE
redis-bench-jgbj8 0/1 Completed 0 1m
redis-deployment-75466795f6-thtx4 1/1 Running 0 16m
kubectl logs redis-bench-jgbj8

Provisioning local volumes without the Local Storage Operator

Local volumes cannot be created by dynamic provisioning. Instead, persistent volumes can be created by defining the persistent volume (PV) in an object definition. The local volume provisioner looks for any file system or block volume devices at the paths specified in the defined resource.

Prerequisites

• Local disks are attached to the OpenShift Container Platform nodes.

Procedure

1. Define the PV. Create a file, such as example-pv-filesystem.yaml or example-pv-block.yaml, with the PersistentVolume object definition. This resource must define the nodes and paths to the local volumes.

   Do not use different storage class names for the same device. Doing so will create multiple PVs.

   example-pv-filesystem.yaml

   apiVersion: v1kind: PersistentVolumemetadata: name: example-pv-filesystemspec: capacity: storage: 100Gi volumeMode: Filesystem (1) accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage (2) local: path: /dev/xvdf (3) nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - example-node

   1	The volume mode, either Filesystem or Block, that defines the type of PVs.
   2	The name of the storage class to use when creating PV resources. Use a storage class that uniquely identifies this set of PVs.
   3	The path containing a list of local storage devices to choose from, or a directory. You can only specify a directory with Filesystem volumeMode.

   A raw block volume (volumeMode: block) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices.

   example-pv-block.yaml

   apiVersion: v1kind: PersistentVolumemetadata: name: example-pv-blockspec: capacity: storage: 100Gi volumeMode: Block (1) accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage (2) local: path: /dev/xvdf (3) nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - example-node

   1	The volume mode, either Filesystem or Block, that defines the type of PVs.
   2	The name of the storage class to use when creating PV resources. Be sure to use a storage class that uniquely identifies this set of PVs.
   3	The path containing a list of local storage devices to choose from.

2. Create the PV resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <example-pv>.yaml

3. Verify that the local PV was created:

   Example output

   NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
   example-pv-filesystem 100Gi RWO Delete Available local-storage 3m47s
   example-pv1 1Gi RWO Delete Bound local-storage/pvc1 local-storage 12h
   example-pv2 1Gi RWO Delete Bound local-storage/pvc2 local-storage 12h
   example-pv3 1Gi RWO Delete Bound local-storage/pvc3 local-storage 12h

Prepare your local disk on all workers nodes

In my case i’m using lvm to create 1 disk volume called prometheus with an ext4 filesystem on each workers nodes. This disk is mounted on /disks/local-provsionner by the Fstab at boot. You can use standard partitions also.

On each workers nodes :

$ lvcreate -n prometheus -L 60G vg0

$ mkfs.ext4 /dev/vg0/prometheus

$ vi /etc/fstab
/dev/vg0/prometheus /disks/local-provisioner/prometheus ext4 defaults 0 0

$ mkdir -p /disks/local-provisioner/prometheus

$ mount -a

Why Local-volume-static-provisionner ?

The local volume static provisioner manages the PersistentVolume lifecycle for pre-allocated disks by detecting and creating PVs for each local disk on the host, and cleaning up the disks when released. It does not support dynamic provisioning.

Hostpath vs local volume

Local persistent volume ensure that a pod with persistent data is always scheduled to the same worker node :

The biggest difference is that the Kubernetes scheduler understands which node a Local Persistent Volume belongs to. With HostPath volumes, a pod referencing a HostPath volume may be moved by the scheduler to a different node resulting in data loss. But with Local Persistent Volumes, the Kubernetes scheduler ensures that a pod using a Local Persistent Volume is always scheduled to the same node.

Deploy local-static-provisionner

Create a storage class called local-prometheus :

$ git clone https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner.git

$ cp deployment/kubernetes/example/default_example_storageclass.yaml prometheus-storageclass.yaml

$ vi prometheus-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-prometheus
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

$ kubectl apply -f prometheus-storageclass.yaml

Customize Helm local-provisioner configuration :

$ cp helm/provisioner/values.yaml values.yaml

$ vi values.yaml
[...]
classes:
- name: local-prometheus
hostDir: /disks/local-provisioner
volumeMode: Filesystem
fsType: ext4
namePattern: “*"
[...]

In this example I’m using 1 storage class but you can add other storage class for different type of configuration (hd / ssd / etc..) if needed.

Deploy local-volume-static-provisioner with Helm :

$ helm install -f values.yaml -n kube-system localprovi ./helm/provisioner/

$ kubectl get pods -n kube-system | grep localprovi
localprovi-provisioner-8klbn 1/1 Running 0 40s
localprovi-provisioner-lvchq 1/1 Running 0 40s
localprovi-provisioner-ndwjd 1/1 Running 0 40s

Check volumes discovery :

$ kubectl logs localprovi-provisioner-8klbn -n kube-system
Found new volume at host path “/disks/local-provisioner/prometheus” with capacity 63145029632, creating Local PV “local-pv-1d3f8952”, required volumeMode “Filesystem”
Created PV “local-pv-1d3f8952” for volume at “/disks/local-provisioner/prometheus”

$ kubectl get pv | grep local-prom
local-pv-1d3f8952 58Gi RWO Delete Available local-prometheus
local-pv-4de2a9e7 58Gi RWO Delete Available local-prometheus
local-pv-73241d24 58Gi RWO Delete Available local-prometheus

Configure Prometheus-operator helm values to use our new local provisionner storageclass

$ vi values.yaml
prometheus
replicas: 3
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: local-prometheus
accessModes: [“ReadWriteOnce”]
resources:
requests:
storage: 58Gi

$ helm install -n monitoring prometheus stable/prometheus-operator -f values.yaml

$ kubectl get pvc -n monitoring
prometheus-oper-db-0 Bound local-pv-4de2a9e7 58Gi RWO local-prometheus 40s
prometheus-oper-db-1 Bound local-pv-4de2a9e7 58Gi RWO local-prometheus 40s
prometheus-oper-db-2 Bound local-pv-4de2a9e7 58Gi RWO local-prometheus 40s

Awesome ! each replicas of Prometheus is bounded to a local Pv 🙂

If you delete a replica, the scheduler automatically reschedule the replica to the same node.

What is local volume ?

A local volume is a mounted storage device like a disk, a partition or even a simple directory. It allow us to use static PV in Kubernetes. Local volumes are stable since Kubernetes 1.14 but the local static provisionner is supported by the Kubernetes SIG.

A simple use case example

To demonstrate how to implement a dynamic local provisioner, i will use a simple common use case : adding persistent volumes to a Prometheus-operator helm deployment composed of 3 prometheus-server statefullset replicas. What we want here is that the local provisioner create 1 local volume for each replicas on each nodes and keep the pods assigned to the same node during all the lifecycle of the deployment. That mean if the pod die and is recreated, the scheduler will assign the same worker node for the pods to maintain data consistency.

Using tolerations with Local Storage Operator pods

Taints can be applied to nodes to prevent them from running general workloads. To allow the Local Storage Operator to use tainted nodes, you must add tolerations to the Pod or DaemonSet definition. This allows the created resources to run on these tainted nodes.

You apply tolerations to the Local Storage Operator pod through the LocalVolume resource
and apply taints to a node through the node specification. A taint on a node instructs the node to repel all pods that do not tolerate the taint. Using a specific taint that is not on other pods ensures that the Local Storage Operator pod can also run on that node.

Prerequisites

• The Local Storage Operator is installed.

• Local disks are attached to OpenShift Container Platform nodes with a taint.

• Tainted nodes are expected to provision local storage.

Procedure

To configure local volumes for scheduling on tainted nodes:

1. Modify the YAML file that defines the Pod and add the LocalVolume spec, as shown in the following example:

    apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "local-disks" namespace: "openshift-local-storage" spec: tolerations: - key: localstorage (1) operator: Equal (2) value: "localstorage" (3) storageClassDevices: - storageClassName: "localblock-sc" volumeMode: Block (4) devicePaths: (5) - /dev/xvdg

   1	Specify the key that you added to the node.
   2	Specify the Equal operator to require the key/value parameters to match. If operator is Exists, the system checks that the key exists and ignores the value. If operator is Equal, then the key and value must match.
   3	Specify the value local of the tainted node.
   4	The volume mode, either Filesystem or Block, defining the type of the local volumes.
   5	The path containing a list of local storage devices to choose from.

2. Optional: To create local persistent volumes on only tainted nodes, modify the YAML file and add the LocalVolume spec, as shown in the following example:

   spec: tolerations: - key: node-role.kubernetes.io/master operator: Exists

The defined tolerations will be passed to the resulting daemon sets, allowing the diskmaker and provisioner pods to be created for nodes that contain the specified taints.

Provisioning local volumes by using the Local Storage Operator

Local volumes cannot be created by dynamic provisioning. Instead, persistent volumes can be created by the Local Storage Operator. The local volume provisioner looks for any file system or block volume devices at the paths specified in the defined resource.

Prerequisites

• The Local Storage Operator is installed.

• You have a local disk that meets the following conditions:

  • It is attached to a node.

  • It is not mounted.

  • It does not contain partitions.

Procedure

1. Create the local volume resource. This resource must define the nodes and paths to the local volumes.

   Do not use different storage class names for the same device. Doing so will create multiple persistent volumes (PVs).

   Example: Filesystem

   apiVersion: "local.storage.openshift.io/v1"kind: "LocalVolume"metadata: name: "local-disks" namespace: "openshift-local-storage" (1)spec: nodeSelector: (2) nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ip-10-0-140-183 - ip-10-0-158-139 - ip-10-0-164-33 storageClassDevices: - storageClassName: "local-sc" (3) volumeMode: Filesystem (4) fsType: xfs (5) devicePaths: (6) - /path/to/device (7)

   1	The namespace where the Local Storage Operator is installed.
   2	Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.
   3	The name of the storage class to use when creating persistent volume objects. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.
   4	The volume mode, either Filesystem or Block, that defines the type of local volumes.
   5	The file system that is created when the local volume is mounted for the first time.
   6	The path containing a list of local storage devices to choose from.
   7	Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as /dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   A raw block volume (volumeMode: block) is not formatted with a file system. You should use this mode only if any application running on the pod can use raw block devices.

   Example: Block

   apiVersion: "local.storage.openshift.io/v1"kind: "LocalVolume"metadata: name: "local-disks" namespace: "openshift-local-storage" (1)spec: nodeSelector: (2) nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ip-10-0-136-143 - ip-10-0-140-255 - ip-10-0-144-180 storageClassDevices: - storageClassName: "localblock-sc" (3) volumeMode: Block (4) devicePaths: (5) - /path/to/device (6)

   1	The namespace where the Local Storage Operator is installed.
   2	Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.
   3	The name of the storage class to use when creating persistent volume objects.
   4	The volume mode, either Filesystem or Block, that defines the type of local volumes.
   5	The path containing a list of local storage devices to choose from.
   6	Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

2. Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <local-volume>.yaml

3. Verify that the provisioner was created and that the corresponding daemon sets were created:

   $ oc get all -n openshift-local-storage

   Example output

   NAME READY STATUS RESTARTS AGE
   pod/diskmaker-manager-9wzms 1/1 Running 0 5m43s
   pod/diskmaker-manager-jgvjp 1/1 Running 0 5m43s
   pod/diskmaker-manager-tbdsj 1/1 Running 0 5m43s
   pod/local-storage-operator-7db4bd9f79-t6k87 1/1 Running 0 14m
   NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEservice/local-storage-operator-metrics ClusterIP 172.30.135.36 <none> 8383/TCP,8686/TCP 14mNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEdaemonset.apps/diskmaker-manager 3 3 3 3 3 <none> 5m43sNAME READY UP-TO-DATE AVAILABLE AGE
   deployment.apps/local-storage-operator 1/1 1 1 14m
   NAME DESIRED CURRENT READY AGE
   replicaset.apps/local-storage-operator-7db4bd9f79 1 1 1 14m

   Note the desired and current number of daemon set processes. A desired count of 0 indicates that the label selectors were invalid.

4. Verify that the persistent volumes were created:

   Example output

   NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
   local-pv-1cec77cf 100Gi RWO Delete Available local-sc 88m
   local-pv-2ef7cd2a 100Gi RWO Delete Available local-sc 82m
   local-pv-3fa1c73 100Gi RWO Delete Available local-sc 48m

Installing the Local Storage Operator

The Local Storage Operator is not installed in OpenShift Container Platform by default. Use the following procedure to install and configure this Operator to enable local volumes in your cluster.

Prerequisites

• Access to the OpenShift Container Platform web console or command-line interface (CLI).

Procedure

1. Create the openshift-local-storage project:

   $ oc adm new-project openshift-local-storage

2. Optional: Allow local storage creation on infrastructure nodes.

   You might want to use the Local Storage Operator to create volumes on infrastructure nodes in support of components such as logging and monitoring.

   You must adjust the default node selector so that the Local Storage Operator includes the infrastructure nodes, and not just worker nodes.

   To block the Local Storage Operator from inheriting the cluster-wide default selector, enter the following command:

   $ oc annotate project openshift-local-storage openshift.io/node-selector=''

From the UI

To install the Local Storage Operator from the web console, follow these steps:

1. Log in to the OpenShift Container Platform web console.

2. Navigate to Operators → OperatorHub.

3. Type Local Storage into the filter box to locate the Local Storage Operator.

4. Click Install.

5. On the Install Operator page, select A specific namespace on the cluster. Select openshift-local-storage from the drop-down menu.

6. Adjust the values for Update Channel and Approval Strategy to the values that you want.

7. Click Install.

Once finished, the Local Storage Operator will be listed in the Installed Operators section of the web console.

From the CLI

1. Install the Local Storage Operator from the CLI.

   1. Run the following command to get the OpenShift Container Platform major and minor version. It is required for the channel value in the next
      step.

      $ OC_VERSION=$(oc version -o yaml | grep openshiftVersion | \ grep -o '[0-9]*[.][0-9]*' | head -1)

   2. Create an object YAML file to define an Operator group and subscription for the Local Storage Operator,
      such as openshift-local-storage.yaml:

      Example openshift-local-storage.yaml

      apiVersion: operators.coreos.com/v1kind: OperatorGroupmetadata: name: local-operator-group namespace: openshift-local-storagespec: targetNamespaces: - openshift-local-storage---apiVersion: operators.coreos.com/v1alpha1kind: Subscriptionmetadata: name: local-storage-operator namespace: openshift-local-storagespec: channel: "${OC_VERSION}" installPlanApproval: Automatic (1) name: local-storage-operator source: redhat-operators sourceNamespace: openshift-marketplace

      1	The user approval policy for an install plan.

2. Create the Local Storage Operator object by entering the following command:

   $ oc apply -f openshift-local-storage.yaml

   At this point, the Operator Lifecycle Manager (OLM) is now aware of the Local Storage Operator. A ClusterServiceVersion (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

3. Verify local storage installation by checking that all pods and the Local Storage Operator have been created:

   1. Check that all the required pods have been created:

      $ oc -n openshift-local-storage get pods

      Example output

      NAME READY STATUS RESTARTS AGE
      local-storage-operator-746bf599c9-vlt5t 1/1 Running 0 19m

   2. Check the ClusterServiceVersion (CSV) YAML manifest to see that the Local Storage Operator is available in the openshift-local-storage project:

      $ oc get csvs -n openshift-local-storage

      Example output

      NAME DISPLAY VERSION REPLACES PHASE
      local-storage-operator.4.2.26-202003230335 Local Storage 4.2.26-202003230335 Succeeded

After all checks have passed, the Local Storage Operator is installed successfully.

Deleting the Local Storage Operator resources

Removing a local volume or local volume set

Occasionally, local volumes and local volume sets must be deleted. While removing the entry in the resource and deleting the persistent volume is typically enough, if you want to reuse the same device path or have it managed by a different storage class, then additional steps are needed.

Prerequisites

• The persistent volume must be in a Released or Available state.

  Deleting a persistent volume that is still in use can result in data loss or corruption.

Procedure

1. Edit the previously created local volume to remove any unwanted disks.

   1. Edit the cluster resource:

      $ oc edit localvolume <name> -n openshift-local-storage

   2. Navigate to the lines under devicePaths, and delete any representing unwanted disks.

2. Delete any persistent volumes created.

3. Delete any symlinks on the node.

   The following step involves accessing a node as the root user. Modifying the state of the node beyond the steps in this procedure could result in cluster instability.

   1. Create a debug pod on the node:

      $ oc debug node/<node-name>

   2. Change your root directory to /host:

   3. Navigate to the directory containing the local volume symlinks.

      $ cd /mnt/openshift-local-storage/<sc-name> (1)

      1	The name of the storage class used to create the local volumes.

   4. Delete the symlink belonging to the removed device.

Uninstalling the Local Storage Operator

To uninstall the Local Storage Operator, you must remove the Operator and all created resources in the openshift-local-storage project.

Prerequisites

• Access to the OpenShift Container Platform web console.

Procedure

1. Delete any local volume resources installed in the project, such as localvolume, localvolumeset, and localvolumediscovery:

   $ oc delete localvolume --all --all-namespaces$ oc delete localvolumeset --all --all-namespaces$ oc delete localvolumediscovery --all --all-namespaces

2. Uninstall the Local Storage Operator from the web console.

   1. Log in to the OpenShift Container Platform web console.

   2. Navigate to Operators → Installed Operators.

   3. Type Local Storage into the filter box to locate the Local Storage Operator.

   4. Click the Options menu at the end of the Local Storage Operator.

   5. Click Uninstall Operator.

   6. Click Remove in the window that appears.

3. The PVs created by the Local Storage Operator will remain in the cluster until deleted. After these volumes are no longer in use, delete them by running the following command:

4. Delete the openshift-local-storage project:

   $ oc delete project openshift-local-storage

Automating discovery and provisioning for local storage devices

The Local Storage Operator automates local storage discovery and provisioning. With this feature, you can simplify installation when dynamic provisioning is not available during deployment, such as with bare metal, VMware, or AWS store instances with attached devices.

Use the following procedure to automatically discover local devices, and to automatically provision local volumes for selected devices.

Prerequisites

• You have cluster administrator permissions.

• You have installed the Local Storage Operator.

• You have attached local disks to OpenShift Container Platform nodes.

• You have access to the OpenShift Container Platform web console and the oc command-line interface (CLI).

Procedure

1. To enable automatic discovery of local devices from the web console:

   1. In the Administrator perspective, navigate to Operators → Installed Operators and click on the Local Volume Discovery tab.

   2. Click Create Local Volume Discovery.

   3. Select either All nodes or Select nodes, depending on whether you want to discover available disks on all or specific nodes.

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   4. Click Create.

A local volume discovery instance named auto-discover-devices is displayed.

1. To display a continuous list of available devices on a node:

   1. Log in to the OpenShift Container Platform web console.

   2. Navigate to Compute → Nodes.

   3. Click the node name that you want to open. The «Node Details» page is displayed.

   4. Select the Disks tab to display the list of the selected devices.

      The device list updates continuously as local disks are added or removed. You can filter the devices by name, status, type, model, capacity, and mode.

2. To automatically provision local volumes for the discovered devices from the web console:

   1. Navigate to Operators → Installed Operators and select Local Storage from the list of Operators.

   2. Select Local Volume Set → Create Local Volume Set.

   3. Enter a volume set name and a storage class name.

   4. Choose All nodes or Select nodes to apply filters accordingly.

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   5. Select the disk type, mode, size, and limit you want to apply to the local volume set, and click Create.

      A message displays after several minutes, indicating that the «Operator reconciled successfully.»

3. Alternatively, to provision local volumes for the discovered devices from the CLI:

   1. Create an object YAML file to define the local volume set, such as local-volume-set.yaml, as shown in the following example:

      apiVersion: local.storage.openshift.io/v1alpha1kind: LocalVolumeSetmetadata: name: example-autodetectspec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - worker-0 - worker-1 storageClassName: example-storageclass (1) volumeMode: Filesystem fsType: ext4 maxDeviceCount: 10 deviceInclusionSpec: deviceTypes: (2) - disk - part deviceMechanicalProperties: - NonRotational minSize: 10G maxSize: 100G models: - SAMSUNG - Crucial_CT525MX3 vendors: - ATA - ST2000LM

      1	Determines the storage class that is created for persistent volumes that are provisioned from discovered devices. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.
      2	When using the local volume set feature, the Local Storage Operator does not support the use of logical volume management (LVM) devices.

   2. Create the local volume set object:

      $ oc apply -f local-volume-set.yaml

   3. Verify that the local persistent volumes were dynamically provisioned based on the storage class:

      Example output

      NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
      local-pv-1cec77cf 100Gi RWO Delete Available example-storageclass 88m
      local-pv-2ef7cd2a 100Gi RWO Delete Available example-storageclass 82m
      local-pv-3fa1c73 100Gi RWO Delete Available example-storageclass 48m

Conclusion

Local volume provisionner is a great tool for static Kubernetes cluster and allow us to not care about node affinity.

Keep in mind that this type of setup is not a good choice for autoscaling or dynamic cluster because if a dead node is replaced by a healthy node you lose all the data.

If you don’t care about losing data for exemple because you’re using a native clustered or fault tolerant technology it can work great and allow you to not care about an external volume provisioner.