In Part 4 we got a fully operational two-node K3s cluster. But there is a critical problem. If you deploy a database or any stateful application, its data lives on one node’s local disk. If that node goes down, the data is unreachable or lost.<\/p>
This is the storage problem on bare metal Kubernetes. Cloud providers solve it with managed storage (EBS, Persistent Disk) that is independent of any node. On bare metal we need to solve it ourselves.<\/p>
Longhorn is the solution. It is a distributed block storage system that runs entirely inside Kubernetes and replicates your data volumes across multiple nodes automatically.<\/p>
Without Longhorn:<\/p>
PostgreSQL pod scheduled on thor\n data stored at \/var\/lib\/postgresql on thor's local disk\n\nthor goes down\n PostgreSQL pod cannot start on jarvis\n data is locked on thor's disk\n application is down<\/code><\/pre>With Longhorn:<\/p>
PostgreSQL pod scheduled on thor\n data stored in a Longhorn volume\n primary replica on thor's disk\n second replica on jarvis's disk (automatic, real-time)\n\nthor goes down\n Longhorn detects node failure\n rebuilds volume from jarvis replica\n PostgreSQL pod reschedules on jarvis\n application recovers<\/code><\/pre>How Longhorn Works<\/h3>
Longhorn creates volumes, which are virtual block devices that pods mount like a regular disk. Under the hood each volume consists of replicas spread across nodes.<\/p>
Longhorn Volume\n Replica 1, jarvis disk, \/var\/lib\/longhorn\/\n Replica 2, thor disk, \/var\/lib\/longhorn\/\n synced in real-time\n\nmounted as block device\n PostgreSQL pod\n \/var\/lib\/postgresql<\/code><\/pre>Key components:<\/p>
- Longhorn Manager – runs on every node, manages volume lifecycle<\/li>
- Longhorn Engine – handles I\/O for each volume<\/li>
- Longhorn UI – web dashboard at longhorn.domain.com<\/li>
- CSI Driver – integrates with Kubernetes storage API<\/li><\/ul>
Prerequisites – Why iSCSI Matters<\/h3>
In Part 1 we installed open-iscsi and enabled iscsid. Here is why:<\/p>
- name: Enable iscsid for Longhorn\n systemd:\n name: iscsid\n enabled: yes\n state: started<\/code><\/pre>Longhorn uses iSCSI (Internet Small Computer Systems Interface) to present volumes to pods as block devices. Without iscsid running, Longhorn volumes simply will not mount.<\/p>
This is one of those things that fails silently. Longhorn installs fine, but pods needing persistent storage get stuck in ContainerCreating forever.<\/p>
Role: longhorn<\/h3>Full tasks\/main.yml<\/h3>- name: Add Longhorn Helm repo\n shell: |\n helm repo add longhorn https:\/\/charts.longhorn.io\n helm repo update\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre>Adds the official Longhorn Helm chart repository. helm repo update fetches the latest chart index so we always get the right version.<\/p>
- name: Create longhorn-system namespace\n shell: |\n kubectl create namespace longhorn-system \\\n --dry-run=client -o yaml | kubectl apply -f -\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre>The –dry-run=client -o yaml | kubectl apply -f – pattern is the idempotent way to create a namespace. Normally kubectl create namespace fails if the namespace already exists. The dry-run generates the YAML without applying it, then kubectl apply creates it if missing or does nothing if it exists.<\/p>
- name: Install Longhorn\n shell: |\n helm upgrade --install longhorn longhorn\/longhorn \\\n --namespace longhorn-system \\\n --set defaultSettings.defaultReplicaCount={{ longhorn_replica_count }} \\\n --set defaultSettings.storageMinimalAvailablePercentage=10 \\\n --set service.ui.type=LoadBalancer \\\n --version {{ longhorn_version }}\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre>Helm values explained:<\/p>
- defaultSettings.defaultReplicaCount, 2, every new volume gets 2 replicas, one on each node<\/li>
- defaultSettings.storageMinimalAvailablePercentage, 10, do not use a disk if less than 10% free<\/li>
- service.ui.type, LoadBalancer, Longhorn UI gets a real IP from MetalLB<\/li>
- –version, 1.6.2, pin to a specific chart version to prevent unexpected upgrades<\/li><\/ul>
Why defaultReplicaCount=2? With 2 replicas across 2 nodes, every volume has a copy on jarvis and a copy on thor. Lose either node and your data survives. This is the minimum meaningful redundancy for a two-node cluster.<\/p>
- name: Wait for Longhorn to be ready\n shell: |\n kubectl wait --namespace longhorn-system \\\n --for=condition=ready pod \\\n --selector=app=longhorn-manager \\\n --timeout=120s\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml\n retries: 3\n delay: 20<\/code><\/pre>Longhorn deploys many components. We wait for the manager pods specifically because everything else depends on them. retries: 3 with delay: 20 gives Longhorn up to 4 minutes to initialize.<\/p>
- name: Set Longhorn as default StorageClass\n shell: |\n kubectl patch storageclass local-path \\\n -p '{\"metadata\": {\"annotations\":{\n \"storageclass.kubernetes.io\/is-default-class\":\"false\"\n }}}'\n kubectl patch storageclass longhorn \\\n -p '{\"metadata\": {\"annotations\":{\n \"storageclass.kubernetes.io\/is-default-class\":\"true\"\n }}}'\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml\n ignore_errors: yes<\/code><\/pre>K3s ships with local-path as the default StorageClass. local-path stores data on the local node disk with no replication, exactly what we want to avoid.<\/p>
This task removes the default annotation from local-path and adds it to longhorn. Now any PVC created without specifying a StorageClass automatically uses Longhorn with replication.<\/p>
ignore_errors: yes means if local-path does not exist or is already non-default, the task fails harmlessly.<\/p>
Understanding PersistentVolumes and PVCs<\/h3>PersistentVolumeClaim (PVC) what your app requests\n |\n | bound to\n v\nPersistentVolume (PV) the actual storage\n |\n | provisioned by\n v\nStorageClass (Longhorn) how storage is created\n |\n | creates\n v\nLonghorn Volume the replicated block device<\/code><\/pre>StorageClass defines how storage is provisioned. When you set Longhorn as default, every PVC automatically gets a Longhorn volume.<\/p>
PVC is what you put in your pod’s YAML:<\/p>
apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: postgres-data\n namespace: myapp\nspec:\n accessModes:\n - ReadWriteOnce\n storageClassName: longhorn\n resources:\n requests:\n storage: 10Gi<\/code><\/pre>Access Modes:<\/p>
- ReadWriteOnce – one node, read\/write – use for databases (PostgreSQL, MySQL)<\/li>
- ReadOnlyMany – multiple nodes, read-only – use for shared config files<\/li>
- ReadWriteMany – multiple nodes, read\/write – use for shared uploads and media files<\/li><\/ul>
Mounting a PVC in a Pod<\/h3>apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: postgres\n namespace: myapp\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: postgres\n template:\n metadata:\n labels:\n app: postgres\n spec:\n nodeSelector:\n workload: apps\n containers:\n - name: postgres\n image: postgres:15\n env:\n - name: POSTGRES_PASSWORD\n value: \"mypassword\"\n volumeMounts:\n - name: postgres-storage\n mountPath: \/var\/lib\/postgresql\/data\n volumes:\n - name: postgres-storage\n persistentVolumeClaim:\n claimName: postgres-data<\/code><\/pre>When this pod starts, Kubernetes sees it needs the postgres-data PVC, Longhorn mounts the volume on the node where the pod is scheduled, and the pod starts with \/var\/lib\/postgresql\/data backed by the replicated Longhorn volume.<\/p>
Verifying Longhorn is Working<\/h3>
Check Longhorn pods:<\/p>
kubectl get pods -n longhorn-system<\/code><\/pre>All pods should be Running.<\/p>
Check StorageClass:<\/p>
kubectl get storageclass\nNAME PROVISIONER DEFAULT\nlonghorn (default) driver.longhorn.io true\nlocal-path rancher.io\/local-path<\/code><\/pre>Test with a PVC:<\/p>
cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: test-pvc\nspec:\n accessModes: [ReadWriteOnce]\n resources:\n requests:\n storage: 1Gi\nEOF<\/code><\/pre>kubectl get pvc test-pvc\nNAME STATUS VOLUME CAPACITY STORAGECLASS\ntest-pvc Bound pvc-xxxxx 1Gi longhorn<\/code><\/pre>Bound means Longhorn successfully created a volume. Pending means something is wrong. Check kubectl describe pvc test-pvc for the error.<\/p>
Open http:\/\/longhorn.domain.com in your browser to see the Longhorn UI showing all volumes, replica health per node, and disk usage.<\/p>
Storage Layout<\/h3>jarvis (1TB NVMe)\n Ubuntu OS, 8GB\n K3s and containers, 10GB\n Prometheus data, 20GB (15 days retention)\n Loki logs, 10GB\n Jenkins artifacts, 20GB\n Longhorn replicas, 100GB (replica copies of all volumes)\n Free, 830GB<\/code><\/pre>thor (512GB SSD)\n Ubuntu OS, 8GB\n K3s and containers, 10GB\n Dev environment, 20GB (code, node_modules, venvs)\n Longhorn primary, 50GB (primary copies of app volumes)\n Free, 424GB<\/code><\/pre>What’s Next<\/h3>
In Part 6 I will cover the full monitoring stack, Prometheus, Grafana, Loki, Promtail, and Jaeger. I will explain the Helm values for each, what gets scraped automatically, and how to read the dashboards once everything is running.<\/p>","protected":false},"excerpt":{"rendered":"
Overview In Part 4 we got a fully operational two-node K3s cluster. But there is a critical problem. If you deploy a…<\/p>\n","protected":false},"author":2,"featured_media":164,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"5","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-05-longhorn-distributed-storage-across-two-nodes-reza-chegini-7sdlc\/","footnotes":""},"categories":[10,62,9],"tags":[8,66,3,98,99],"series":[64],"class_list":["post-162","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-home-lab","category-kubernetes","tag-devops","tag-home-lab","tag-kubernetes","tag-longhorn","tag-storage","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-05-longhorn-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=162"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/162\/revisions"}],"predecessor-version":[{"id":163,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/162\/revisions\/163"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/164"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=162"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}