{"id":121,"date":"2026-06-21T12:00:00","date_gmt":"2026-06-21T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=121"},"modified":"2026-10-30T07:11:20","modified_gmt":"2026-10-30T07:11:20","slug":"part-09-dns-and-networking-coredns-route-53-and-nginx-ingress","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=121","title":{"rendered":"Part 09 &#8211; DNS and Networking: CoreDNS, Route 53 and NGINX Ingress"},"content":{"rendered":"<h3 class=\"wp-block-heading\">Overview<\/h3><p class=\"wp-block-paragraph\">In Part 8 we completed the GitOps stack. But all those services are useless if you cannot reach them by name. In this article I cover the full DNS and networking setup that makes http:\/\/grafana.rezaops.com work from any device on the home network.<\/p><p class=\"wp-block-paragraph\">This was the most complex part of the entire build. We went through three different approaches before landing on the right solution. I will explain what we tried, why each failed, and what finally worked.<\/p><h3 class=\"wp-block-heading\">The DNS Problem on a Home Lab<\/h3><p class=\"wp-block-paragraph\">This is the fundamental challenge. You type http:\/\/grafana.rezaops.com and your browser asks what IP is grafana.rezaops.com. Your ISP&#8217;s DNS server has no idea that grafana.rezaops.com should resolve to 10.0.0.200 on your local network.<\/p><p class=\"wp-block-paragraph\">Three approaches I tried, in order.<\/p><h3 class=\"wp-block-heading\">Approach 1 &#8211; \/etc\/hosts (Works on Mac Only)<\/h3><p class=\"wp-block-paragraph\">The simplest fix, add entries to \/etc\/hosts on your Mac:<\/p><pre class=\"wp-block-code\"><code>10.0.0.200  grafana.rezaops.com\n10.0.0.200  jenkins.rezaops.com<\/code><\/pre><p class=\"wp-block-paragraph\">Works on Mac. Does not work on phone, tablet, or other laptops.<\/p><h3 class=\"wp-block-heading\">Approach 2 &#8211; Route 53 Private Hosted Zone (Does Not Work)<\/h3><p class=\"wp-block-paragraph\">Route 53 private hosted zones seem perfect but there is a hard AWS requirement:<\/p><p class=\"wp-block-paragraph\">Error: When creating a private hosted zone you must specify VPCId and VPCRegion<\/p><p class=\"wp-block-paragraph\">Private hosted zones must be associated with an AWS VPC. Your home servers are not in AWS. Private zones are only for AWS infrastructure.<\/p><p class=\"wp-block-paragraph\">What we did instead was create a public hosted zone with private IP records:<\/p><p class=\"wp-block-paragraph\">grafana.rezaops.com, 10.0.0.200 (public record, private IP)<\/p><p class=\"wp-block-paragraph\">The record is publicly visible but 10.0.0.200 is a private RFC1918 address unreachable from the internet. This still does not solve the device problem because your home devices use your ISP&#8217;s DNS, not Route 53 directly.<\/p><h3 class=\"wp-block-heading\">Approach 3 &#8211; Pi-hole (Broken in v6)<\/h3><p class=\"wp-block-paragraph\">Pi-hole v6 completely rewrote its internals, replacing dnsmasq with its own FTL DNS engine. As a result:<\/p><ul class=\"wp-block-list\"><li>\/etc\/dnsmasq.d\/ config files are ignored, the FTL engine does not read them<\/li><li>pihole restartdns command no longer works, returns an error about a readonly variable<\/li><li>pihole reloaddns partially broken, cannot reload custom config<\/li><\/ul><p class=\"wp-block-paragraph\">After multiple attempts to work around the v6 changes, I abandoned Pi-hole entirely.<\/p><h3 class=\"wp-block-heading\">Final Solution &#8211; CoreDNS on Kubernetes<\/h3><p class=\"wp-block-paragraph\">CoreDNS is what K3s uses internally for pod DNS. It is battle-tested, runs natively in Kubernetes, and has a simple configuration format called a Corefile.<\/p><p class=\"wp-block-paragraph\">I deployed a second CoreDNS instance specifically for external home network DNS, separate from K3s&#8217;s internal CoreDNS.<\/p><h3 class=\"wp-block-heading\">The Corefile<\/h3><pre class=\"wp-block-code\"><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: coredns-external\n  namespace: kube-system\ndata:\n  Corefile: |\n    rezaops.com {\n        hosts {\n            10.0.0.200 grafana.rezaops.com\n            10.0.0.200 prometheus.rezaops.com\n            10.0.0.200 jenkins.rezaops.com\n            10.0.0.200 argocd.rezaops.com\n            10.0.0.200 vault.rezaops.com\n            10.0.0.200 longhorn.rezaops.com\n            10.0.0.200 home.rezaops.com\n            fallthrough\n        }\n    }\n    . {\n        forward . 1.1.1.1 8.8.8.8\n        cache 30\n    }<\/code><\/pre><p class=\"wp-block-paragraph\">Corefile explained block by block.<\/p><p class=\"wp-block-paragraph\">The rezaops.com block handles all queries for rezaops.com and its subdomains. The hosts plugin works like \/etc\/hosts. Each line maps a hostname to an IP. When any device asks what grafana.rezaops.com is, CoreDNS returns 10.0.0.200.<\/p><p class=\"wp-block-paragraph\">fallthrough means if a hostname is not in the hosts list, pass the query to the next block instead of returning NXDOMAIN.<\/p><p class=\"wp-block-paragraph\">The dot block matches everything. Any query that is not for rezaops.com, like google.com or github.com, gets forwarded to Cloudflare (1.1.1.1) or Google (8.8.8.8).<\/p><p class=\"wp-block-paragraph\">cache 30 caches DNS responses for 30 seconds, reducing upstream queries and making browsing faster.<\/p><h3 class=\"wp-block-heading\">Deploying CoreDNS<\/h3><pre class=\"wp-block-code\"><code>- name: Deploy CoreDNS external\n  shell: |\n    cat &lt;&lt;EOF | kubectl apply -f -\n    apiVersion: apps\/v1\n    kind: Deployment\n    metadata:\n      name: coredns-external\n      namespace: kube-system\n    spec:\n      replicas: 1\n      selector:\n        matchLabels:\n          app: coredns-external\n      template:\n        metadata:\n          labels:\n            app: coredns-external\n        spec:\n          nodeSelector:\n            workload: monitoring\n          containers:\n          - name: coredns\n            image: coredns\/coredns:1.11.1\n            args: [\"-conf\", \"\/etc\/coredns\/Corefile\"]\n            ports:\n            - containerPort: 53\n              protocol: UDP\n            - containerPort: 53\n              protocol: TCP\n            livenessProbe:\n              httpGet:\n                path: \/health\n                port: 8080\n              initialDelaySeconds: 10\n            volumeMounts:\n            - name: config\n              mountPath: \/etc\/coredns\n          volumes:\n          - name: config\n            configMap:\n              name: coredns-external\n    EOF<\/code><\/pre><p class=\"wp-block-paragraph\">args: [&#8220;-conf&#8221;, &#8220;\/etc\/coredns\/Corefile&#8221;] tells CoreDNS to use our ConfigMap-mounted Corefile.<\/p><p class=\"wp-block-paragraph\">The volumes section mounts the ConfigMap as a file at \/etc\/coredns\/Corefile. When we update the ConfigMap, CoreDNS picks up the change automatically without restarting.<\/p><p class=\"wp-block-paragraph\">The livenessProbe checks \/health on port 8080. If it fails, Kubernetes restarts the pod automatically.<\/p><h3 class=\"wp-block-heading\">The TCP\/UDP Split &#8211; A MetalLB Limitation<\/h3><p class=\"wp-block-paragraph\">DNS uses both TCP and UDP on port 53. A typical service with mixed protocols stays pending forever with MetalLB. This is a known MetalLB limitation in L2 mode.<\/p><p class=\"wp-block-paragraph\">The fix is to split into two services with the same allow-shared-ip annotation:<\/p><pre class=\"wp-block-code\"><code># TCP service\nmetadata:\n  annotations:\n    metallb.universe.tf\/allow-shared-ip: \"coredns-dns\"\nspec:\n  ports:\n  - port: 53\n    protocol: TCP\n\n# UDP service (same annotation value means same IP)\nmetadata:\n  annotations:\n    metallb.universe.tf\/allow-shared-ip: \"coredns-dns\"\nspec:\n  ports:\n  - port: 53\n    protocol: UDP<\/code><\/pre><p class=\"wp-block-paragraph\">Both services get the same MetalLB IP (10.0.0.209 in our setup).<\/p><h3 class=\"wp-block-heading\">Testing CoreDNS<\/h3><pre class=\"wp-block-code\"><code># Query CoreDNS directly from your Mac\ndig @10.0.0.209 grafana.rezaops.com\n\n# Expected output\n;; ANSWER SECTION:\ngrafana.rezaops.com.   3600   IN   A   10.0.0.200\n\n# Test forwarding works for external domains\ndig @10.0.0.209 google.com\n# Should resolve to Google's IPs via 1.1.1.1 forwarding<\/code><\/pre><h3 class=\"wp-block-heading\">Pointing Devices at CoreDNS<\/h3><p class=\"wp-block-paragraph\">Mac, System Settings, WiFi, your network, Details, DNS tab:<\/p><pre class=\"wp-block-code\"><code>10.0.0.209    CoreDNS (home lab)\n1.1.1.1       Cloudflare fallback\n8.8.8.8       Google fallback<\/code><\/pre><p class=\"wp-block-paragraph\">iPhone, Settings, WiFi, your network, Configure DNS, Manual:<\/p><pre class=\"wp-block-code\"><code>10.0.0.209\n1.1.1.1<\/code><\/pre><p class=\"wp-block-paragraph\">Docker Desktop on Mac, Settings, Docker Engine, add to JSON:<\/p><pre class=\"wp-block-code\"><code>{\n  \"dns\": [\"10.0.0.209\", \"1.1.1.1\"]\n}<\/code><\/pre><p class=\"wp-block-paragraph\">Docker Desktop runs in a VM with its own DNS and does not inherit your Mac&#8217;s system DNS settings. Without this, docker push registry.rezaops.com\/myapp:latest fails with no such host.<\/p><h3 class=\"wp-block-heading\">How NGINX Ingress Routes Traffic<\/h3><p class=\"wp-block-paragraph\">With DNS working, every request to *.rezaops.com resolves to 10.0.0.200, the NGINX Ingress controller&#8217;s MetalLB IP. NGINX then routes the request to the right service based on the Host header.<\/p><pre class=\"wp-block-code\"><code>Request: GET http:\/\/grafana.rezaops.com\/dashboards\n        |\n        | DNS: grafana.rezaops.com, 10.0.0.200\n        v\nNGINX Ingress Controller (10.0.0.200:80)\n  reads Host header: grafana.rezaops.com\n  checks Ingress rules\n        |\n        | matches rule in monitoring namespace\n        v\nService: kube-prometheus-stack-grafana\n        |\n        v\nGrafana Pod (on jarvis)<\/code><\/pre><p class=\"wp-block-paragraph\">The Ingress resource is the routing rule:<\/p><pre class=\"wp-block-code\"><code>apiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: monitoring-ingress\n  namespace: monitoring\nspec:\n  ingressClassName: nginx\n  rules:\n  - host: grafana.rezaops.com\n    http:\n      paths:\n      - path: \/\n        pathType: Prefix\n        backend:\n          service:\n            name: kube-prometheus-stack-grafana\n            port:\n              number: 80\n  - host: prometheus.rezaops.com\n    http:\n      paths:\n      - path: \/\n        pathType: Prefix\n        backend:\n          service:\n            name: kube-prometheus-stack-prometheus\n            port:\n              number: 9090<\/code><\/pre><p class=\"wp-block-paragraph\">Key insight: you only need one LoadBalancer IP (the ingress IP) for all your services. Every additional service just needs an Ingress resource with a different hostname, not a new LoadBalancer IP.<\/p><h3 class=\"wp-block-heading\">Adding a New App Domain<\/h3><p class=\"wp-block-paragraph\">For every new app you deploy, two steps:<\/p><p class=\"wp-block-paragraph\">Step 1, update the CoreDNS ConfigMap:<\/p><pre class=\"wp-block-code\"><code>kubectl edit configmap coredns-external -n kube-system\n# add: 10.0.0.200 myapp.rezaops.com<\/code><\/pre><p class=\"wp-block-paragraph\">CoreDNS picks up ConfigMap changes automatically.<\/p><p class=\"wp-block-paragraph\">Step 2, create an Ingress resource:<\/p><pre class=\"wp-block-code\"><code>apiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: myapp-ingress\n  namespace: myapp\nspec:\n  ingressClassName: nginx\n  rules:\n  - host: myapp.rezaops.com\n    http:\n      paths:\n      - path: \/\n        pathType: Prefix\n        backend:\n          service:\n            name: myapp-service\n            port:\n              number: 8000<\/code><\/pre><p class=\"wp-block-paragraph\">Two steps, under 2 minutes, and your new app is accessible at http:\/\/myapp.rezaops.com from every device on the network.<\/p><h3 class=\"wp-block-heading\">DNS Resolution Full Picture<\/h3><pre class=\"wp-block-code\"><code>Device: \"What is grafana.rezaops.com?\"\n        |\n        | DNS query sent to 10.0.0.209 (CoreDNS)\n        v\nCoreDNS checks hosts block for rezaops.com\n  finds: 10.0.0.200 grafana.rezaops.com\n        |\n        | returns: 10.0.0.200\n        v\nDevice connects to 10.0.0.200:80\n        |\n        v\nNGINX Ingress Controller\n  reads Host: grafana.rezaops.com\n  routes to Grafana service\n        |\n        v\nGrafana pod responds<\/code><\/pre><p class=\"wp-block-paragraph\">For external domains (google.com), CoreDNS forwards to 1.1.1.1 automatically.<\/p><h3 class=\"wp-block-heading\">What&#8217;s Next<\/h3><p class=\"wp-block-paragraph\">In Part 10 I will cover the homepage and container registry setup, how I deployed a static HTML dashboard as a Kubernetes pod and configured the Docker registry so every machine on the network can push and pull images with a proper domain name.<\/p>","protected":false},"excerpt":{"rendered":"<p>Overview In Part 8 we completed the GitOps stack. But all those services are useless if you cannot reach them by name.&hellip;<\/p>\n","protected":false},"author":2,"featured_media":124,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"9","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-09-dns-networking-coredns-route-53-nginx-ingress-reza-chegini-s6kwc\/","footnotes":""},"categories":[10,62,9],"tags":[71,8,70,66,3,73,72],"series":[64],"class_list":["post-121","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-home-lab","category-kubernetes","tag-coredns","tag-devops","tag-dns","tag-home-lab","tag-kubernetes","tag-nginx-ingress","tag-route-53","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-09-dns-networking-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/121","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=121"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/121\/revisions"}],"predecessor-version":[{"id":122,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/121\/revisions\/122"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/124"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=121"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=121"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=121"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=121"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}