In Part 3 I walked through the Ansible roles line by line. In this article I focus on what happens after the control plane is up – how thor joins the cluster as a worker node, how Kubernetes networking works between the two nodes, and how pod scheduling is controlled with labels and taints.<\/p>
By the end of this article the cluster is fully operational with two nodes communicating over WiFi.<\/p>
Before jumping into the YAML, it helps to understand how two K3s nodes communicate.<\/p>
jarvis (10.0.0.148) thor (10.0.0.149)\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 K3s API server \u2502 \u2502 K3s agent \u2502\n\u2502 :6443 \u2502\u25c4\u2500\u2500\u2500\u2500\u2500\u25ba\u2502 \u2502\n\u2502 \u2502 \u2502 \u2502\n\u2502 Pod CIDR: \u2502 \u2502 Pod CIDR: \u2502\n\u2502 10.42.0.0\/24 \u2502 \u2502 10.42.1.0\/24 \u2502\n\u2502 \u2502 \u2502 \u2502\n\u2502 Flannel VXLAN \u2502\u25c4\u2500\u2500\u2500\u2500\u2500\u25ba\u2502 Flannel VXLAN \u2502\n\u2502 :8472 (UDP) \u2502 \u2502 :8472 (UDP) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25b2 \u25b2\n \u2502 WiFi LAN \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n 10.0.0.0\/24<\/code><\/pre>Three networks exist simultaneously:<\/p>Network<\/th> Range<\/th> Purpose<\/th><\/tr><\/thead> Home LAN<\/td> 10.0.0.0\/24<\/td> Node-to-node communication<\/td><\/tr> Pod network<\/td> 10.42.0.0\/16<\/td> Pod-to-pod communication (virtual)<\/td><\/tr> Service network<\/td> 10.43.0.0\/16<\/td> ClusterIP services (virtual)<\/td><\/tr><\/tbody><\/table><\/figure>Flannel is K3s’s default CNI (Container Network Interface) plugin. It creates a virtual overlay network (VXLAN) on top of the physical WiFi network. When a pod on thor wants to talk to a pod on jarvis, Flannel wraps the packet in a UDP envelope and sends it over port 8472. The receiving node unwraps it and delivers it to the right pod.<\/p>
This is why 8472\/UDP must be open in UFW on both nodes – it is the tunnel that makes cross-node pod communication possible.<\/p>
Role: k3s_agent<\/h3>
The k3s_agent role runs on thor and joins it to the cluster as a worker node.<\/p>
Full tasks\/main.yml<\/h3>---\n- name: Check if K3s agent is already installed\n stat:\n path: \/usr\/local\/bin\/k3s\n register: k3s_binary<\/code><\/pre>Same pattern as the server role – check if K3s binary exists before trying to install. The binary is the same for both server and agent. The mode (server vs agent) is determined by how it is started.<\/p>
- name: Install K3s agent\n shell: |\n curl -sfL https:\/\/get.k3s.io | sh -s - agent \\\n --server https:\/\/{{ k3s_server_ip }}:{{ k3s_server_port }} \\\n --token {{ hostvars['k3s_token_holder']['k3s_token'] }} \\\n --node-name {{ node_name }}\n when: not k3s_binary.stat.exists<\/code><\/pre>Flags explained:<\/p>
- agent – Run as a worker node, not a server<\/li>
- –server – Address of the control plane API server (https:\/\/10.0.0.148:6443)<\/li>
- –token – Secret used to authenticate the join request<\/li>
- –node-name – This node’s name in the cluster (thor)<\/li><\/ul>
hostvars[‘k3s_token_holder’][‘k3s_token’] – this is how we access the token saved in the k3s_server role using add_host. hostvars is a special Ansible variable containing all variables for all hosts. k3s_token_holder is the dummy host we created, and k3s_token is the variable stored on it.<\/p>
This is the bridge between the two plays – the control plane play runs first, saves the token, then the worker play picks it up.<\/p>
- name: Ensure K3s agent service is enabled and running\n systemd:\n name: k3s-agent\n enabled: yes\n state: started<\/code><\/pre>Note: the service name is k3s-agent on worker nodes, not k3s. This is different from the server. Always use the correct service name when restarting or checking status.<\/p>
# On jarvis (control plane)\nsudo systemctl status k3s\n\n# On thor (worker)\nsudo systemctl status k3s-agent<\/code><\/pre>- name: Wait for worker node to join cluster\n shell: |\n kubectl get node {{ node_name }} --no-headers\n delegate_to: k3s-cp\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml\n register: node_status\n until: \"'Ready' in node_status.stdout\"\n retries: 15\n delay: 10<\/code><\/pre>Three important concepts in this task:<\/p>
- delegate_to: k3s-cp – this task is in the workers play (targeting thor), but it runs on jarvis. Why? Because kubectl is only installed on the control plane. We cannot run kubectl on thor – it does not have a kubeconfig. delegate_to redirects execution to a different host.<\/li>
- until: + retries: + delay: – this is Ansible’s retry loop. It runs the task, checks if Ready is in the output, and if not, waits 10 seconds and tries again. It retries up to 15 times (2.5 minutes total).<\/li>
- –no-headers – removes the column headers from kubectl get node output, making the string match simpler.<\/li><\/ul>
- name: Label worker node\n shell: |\n kubectl label node {{ node_name }} \\\n node-role=worker \\\n workload={{ workload_label }} \\\n --overwrite\n delegate_to: k3s-cp\n environment:\n KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre>Again using delegate_to: k3s-cp – the label command runs on jarvis against the cluster, but the label applies to thor.<\/p>
workload=apps on thor means we can pin app workloads to the worker node:<\/p>
# In any Deployment spec\nnodeSelector:\n workload: apps # runs on thor only<\/code><\/pre>Verifying the Cluster<\/h3>
After both roles complete, run from your MacBook:<\/p>
kubectl get nodes -o wide\nNAME STATUS ROLES AGE VERSION INTERNAL-IP\njarvis Ready control-plane,master 10m v1.29.4+k3s1 10.0.0.148\nthor Ready <none> 5m v1.29.4+k3s1 10.0.0.149<\/code><\/pre>Both nodes Ready means the cluster is healthy. If thor shows NotReady, check:<\/p>
# On thor\nsudo systemctl status k3s-agent\nsudo journalctl -u k3s-agent -f<\/code><\/pre>Common causes of NotReady:<\/p>
- Wrong token in the install command<\/li>
- Port 6443 blocked by UFW on jarvis<\/li>
- Port 8472 blocked on either node (Flannel cannot establish the overlay)<\/li><\/ul>
Understanding Pod Scheduling<\/h3>
Kubernetes decides which node to run a pod on based on several factors. For our home lab the two most important are nodeSelector and taints\/tolerations.<\/p>
nodeSelector – Pinning Pods to Specific Nodes<\/h3># Monitoring pod - runs on jarvis\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: prometheus\nspec:\n template:\n spec:\n nodeSelector:\n workload: monitoring # jarvis has this label\n containers:\n - name: prometheus\n image: prom\/prometheus\n# App pod - runs on thor\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: django-app\nspec:\n template:\n spec:\n nodeSelector:\n workload: apps # thor has this label\n containers:\n - name: django-app\n image: registry.rezaops.com\/django-app:latest<\/code><\/pre>Without nodeSelector, Kubernetes distributes pods across both nodes based on available resources. With it, we control exactly where things land.<\/p>
Checking Node Labels<\/h3>kubectl get nodes --show-labels\nNAME STATUS LABELS\njarvis Ready node-role=control-plane,workload=monitoring,...\nthor Ready node-role=worker,workload=apps,...<\/code><\/pre>How MetalLB Assigns IPs<\/h3>
After K3s and MetalLB are installed, every Service of type LoadBalancer gets a real IP from the pool (10.0.0.200-10.0.0.220).<\/p>
kubectl get svc -A | grep LoadBalancer\nNAMESPACE NAME TYPE EXTERNAL-IP\nmonitoring kube-prometheus-stack-grafana LoadBalancer 10.0.0.202\ncicd jenkins LoadBalancer 10.0.0.205\ningress-nginx ingress-nginx-controller LoadBalancer 10.0.0.200<\/code><\/pre>MetalLB assigns IPs sequentially from the pool. The first service gets .200, the second .201, and so on. The actual IP assigned depends on the order services are created – do not assume a specific service gets a specific IP. Always check with kubectl get svc.<\/p>
This is a lesson I learned the hard way – I assumed the registry would get 10.0.0.200 but MetalLB gave it 10.0.0.206. Always verify:<\/p>
kubectl get svc -n registry\n# Check EXTERNAL-IP column<\/code><\/pre>How NGINX Ingress Routes Traffic<\/h3>
MetalLB gives the NGINX Ingress controller a real IP (10.0.0.200). All HTTP traffic to the cluster goes through this single IP – NGINX then routes it to the right service based on the hostname.<\/p>
Browser: http:\/\/grafana.rezaops.com\n |\n v\nDNS resolves to 10.0.0.200 (ingress IP)\n |\n v\nNGINX Ingress Controller (10.0.0.200:80)\n |\n | checks Host header: grafana.rezaops.com\n |\n v\nIngress resource matches - routes to Grafana Service\n |\n v\nGrafana pod (on jarvis)<\/code><\/pre>The Ingress resource is the routing rule:<\/p>
apiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n name: monitoring-ingress\n namespace: monitoring\nspec:\n ingressClassName: nginx\n rules:\n - host: grafana.rezaops.com\n http:\n paths:\n - path: \/\n pathType: Prefix\n backend:\n service:\n name: kube-prometheus-stack-grafana\n port:\n number: 80<\/code><\/pre>Key insight: You only need one LoadBalancer IP (the ingress IP) for all your services. Every additional service just needs an Ingress resource with a different hostname – not a new LoadBalancer IP.<\/p>
Checking Cluster Health<\/h3># All nodes ready\nkubectl get nodes\n\n# All system pods running\nkubectl get pods -A\n\n# All services with their IPs\nkubectl get svc -A | grep -v ClusterIP\n\n# Events - useful for debugging scheduling issues\nkubectl get events -A --sort-by='.lastTimestamp' | tail -20\n\n# Resource usage per node\nkubectl top nodes\n\n# Which node is each pod running on\nkubectl get pods -A -o wide | grep -v Completed<\/code><\/pre>What’s Next<\/h3>
In Part 5 I will cover Longhorn – how distributed block storage works on bare metal Kubernetes, what the Helm values mean, and how to verify your volumes are being replicated correctly across jarvis and thor.<\/p>","protected":false},"excerpt":{"rendered":"
Overview In Part 3 I walked through the Ansible roles line by line. In this article I focus on what happens after…<\/p>\n","protected":false},"author":2,"featured_media":120,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"4","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-04-networking-k3s-cluster-reza-chegini-8ytuc\/","footnotes":""},"categories":[10,62,9],"tags":[8,66,68,3,69],"series":[64],"class_list":["post-118","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-home-lab","category-kubernetes","tag-devops","tag-home-lab","tag-k3s","tag-kubernetes","tag-networking","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-04-networking-k3s-cluster-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=118"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/118\/revisions"}],"predecessor-version":[{"id":119,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/118\/revisions\/119"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/120"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=118"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}