{"id":115,"date":"2026-06-14T12:00:00","date_gmt":"2026-06-14T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=115"},"modified":"2026-10-30T07:06:20","modified_gmt":"2026-10-30T07:06:20","slug":"part-02-ansible-project-structure","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=115","title":{"rendered":"Part 02-Ansible Project Structure"},"content":{"rendered":"<h3 class=\"wp-block-heading\">Overview<\/h3><p class=\"wp-block-paragraph\">In Part 1, I walked through every OS configuration step needed before installing Kubernetes. In this article I&#8217;ll explain how I organized all of that into a clean, reusable Ansible project, and why the structure matters.<\/p><p class=\"wp-block-paragraph\">A lot of engineers use Ansible as a collection of ad-hoc scripts. That works for small tasks, but it doesn&#8217;t scale. When you have two nodes, 10 different tools to install, and 50+ tasks per role, you need a structure that is:<\/p><ul class=\"wp-block-list\"><li>Readable: anyone can understand what each file does<\/li><li>Idempotent: safe to run multiple times without breaking things<\/li><li>Modular: you can run one phase without running everything<\/li><li>Maintainable: adding a new tool means adding one role, not editing a 500-line playbook<\/li><\/ul><p class=\"wp-block-paragraph\">This is the structure I landed on after several iterations.<\/p><h3 class=\"wp-block-heading\">Project Layout<\/h3><pre class=\"wp-block-code\"><code>homelab-ansible\/\n\u2502\n\u251c\u2500\u2500 ansible.cfg                  \u2190 Ansible configuration\n\u251c\u2500\u2500 site.yml                     \u2190 Master playbook (runs everything)\n\u2502\n\u251c\u2500\u2500 inventory\/\n\u2502   \u2514\u2500\u2500 hosts.ini                \u2190 Node IPs, SSH config, all variables\n\u2502\n\u251c\u2500\u2500 playbooks\/                   \u2190 Individual phase playbooks\n\u2502   \u251c\u2500\u2500 01-common.yml\n\u2502   \u251c\u2500\u2500 02-k3s.yml\n\u2502   \u251c\u2500\u2500 03-longhorn.yml\n\u2502   \u251c\u2500\u2500 04-monitoring.yml\n\u2502   \u251c\u2500\u2500 05-cicd.yml\n\u2502   \u251c\u2500\u2500 06-gitops.yml\n\u2502   \u251c\u2500\u2500 07-dns-tls.yml\n\u2502   \u251c\u2500\u2500 08-pihole.yml            \u2190 (CoreDNS)\n\u2502   \u251c\u2500\u2500 09-homepage.yml\n\u2502   \u2514\u2500\u2500 10-registry.yml\n\u2502\n\u251c\u2500\u2500 roles\/                       \u2190 One folder per tool\/concern\n\u2502   \u251c\u2500\u2500 common\/\n\u2502   \u2502   \u251c\u2500\u2500 tasks\/\n\u2502   \u2502   \u2502   \u2514\u2500\u2500 main.yml\n\u2502   \u2502   \u251c\u2500\u2500 handlers\/\n\u2502   \u2502   \u2502   \u2514\u2500\u2500 main.yml\n\u2502   \u2502   \u2514\u2500\u2500 files\/\n\u2502   \u251c\u2500\u2500 networking\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 k3s_server\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 k3s_agent\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 longhorn\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 monitoring\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 cicd\/\n\u2502   \u2502   \u251c\u2500\u2500 tasks\/main.yml\n\u2502   \u2502   \u2514\u2500\u2500 handlers\/main.yml\n\u2502   \u251c\u2500\u2500 gitops\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 dns_tls\/\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 pihole\/                  \u2190 (CoreDNS role)\n\u2502   \u2502   \u2514\u2500\u2500 tasks\/main.yml\n\u2502   \u251c\u2500\u2500 homepage\/\n\u2502   \u2502   \u251c\u2500\u2500 tasks\/main.yml\n\u2502   \u2502   \u2514\u2500\u2500 files\/\n\u2502   \u2502       \u2514\u2500\u2500 index.html\n\u2502   \u2514\u2500\u2500 registry_dns\/\n\u2502       \u2514\u2500\u2500 tasks\/main.yml\n\u2502\n\u2514\u2500\u2500 templates\/\n    \u2514\u2500\u2500 new-app-ingress-template.yml<\/code><\/pre><h3 class=\"wp-block-heading\">ansible.cfg: The Configuration File<\/h3><pre class=\"wp-block-code\"><code>[defaults]\ninventory         = inventory\/hosts.ini\nroles_path        = roles\nremote_user       = reza\nhost_key_checking = False\n\n[ssh_connection]\npipelining = True<\/code><\/pre><p class=\"wp-block-paragraph\">Line by line:<\/p><p class=\"wp-block-paragraph\">inventory = inventory\/hosts.ini Tells Ansible where to find the inventory file. Without this, you&#8217;d have to pass -i inventory\/hosts.ini every time you run a playbook.<\/p><p class=\"wp-block-paragraph\">roles_path = roles This is the most important line. It tells Ansible to look for roles in the roles\/ directory relative to the project root. Without this, Ansible looks for roles relative to the playbook file, so playbooks\/01-common.yml would look for roles in playbooks\/roles\/ which doesn&#8217;t exist. This caused our first error:<\/p><pre class=\"wp-block-code\"><code>[ERROR]: the role 'common' was not found in ...\/playbooks\/roles<\/code><\/pre><p class=\"wp-block-paragraph\">remote_user = reza The SSH user Ansible connects as. Without this you&#8217;d need ansible_user=reza in every inventory entry.<\/p><p class=\"wp-block-paragraph\">host_key_checking = False Disables SSH host key verification. Normally SSH asks &#8220;are you sure you want to connect to this new host?&#8221; and waits for input. In automation, there&#8217;s nobody to type yes, this skips that check.<\/p><p class=\"wp-block-paragraph\">pipelining = True An SSH optimization. By default Ansible uploads a temporary Python script to the remote machine, runs it, then deletes it, for every single task. With pipelining enabled, Ansible sends the script directly over the SSH connection without writing to disk. This makes playbooks run significantly faster (2\u20133x on slow connections).<\/p><h3 class=\"wp-block-heading\">inventory\/hosts.ini: The Inventory File<\/h3><p class=\"wp-block-paragraph\">The inventory file is where I define every node and every variable. This is the most important file in the project.<\/p><pre class=\"wp-block-code\"><code>[control_plane]\njarvis ansible_host=10.0.0.148 ansible_user=reza node_ip=10.0.0.148 node_name=jarvis node_role=control-plane workload_label=monitoring\n\n[workers]\nthor ansible_host=10.0.0.149 ansible_user=reza node_ip=10.0.0.149 node_name=thor node_role=worker workload_label=apps\n\n[all:vars]\nansible_python_interpreter=\/usr\/bin\/python3\nansible_ssh_private_key_file=~\/.ssh\/id_ed25519\ntimezone=America\/Vancouver\nlab_user=reza\nlab_network=10.0.0.0\/24\nk3s_server_ip=10.0.0.148\nk3s_server_port=6443\nmetallb_ip_range=10.0.0.200-10.0.0.220\nlonghorn_replica_count=2\nlonghorn_version=1.6.2\ngrafana_admin_password=homelab123\nprometheus_retention=15d\njenkins_admin_password=homelab123\nregistry_ip=10.0.0.206\nregistry_port=5000\nbase_domain=rezaops.com\ncp_ip=10.0.0.148\nworker_ip=10.0.0.149\ningress_ip=10.0.0.200\naws_region=us-east-1\naws_access_key_id=REPLACE_WITH_YOUR_KEY\naws_secret_access_key=REPLACE_WITH_YOUR_SECRET\npihole_ip=10.0.0.209\npihole_password=homelab123<\/code><\/pre><p class=\"wp-block-paragraph\">Breaking this down:<\/p><h3 class=\"wp-block-heading\">Host Groups<\/h3><pre class=\"wp-block-code\"><code>[control_plane]\njarvis ...\n\n[workers]\nthor ...<\/code><\/pre><p class=\"wp-block-paragraph\">Ansible uses groups to target specific machines. When a playbook says hosts: control_plane, it only runs on jarvis. When it says hosts: all, it runs on both.<\/p><p class=\"wp-block-paragraph\">This matters for roles like k3s_server which only makes sense on the control plane, vs common which runs everywhere.<\/p><h3 class=\"wp-block-heading\">Per-host Variables (inline)<\/h3><pre class=\"wp-block-code\"><code>jarvis ansible_host=10.0.0.148 node_name=jarvis node_role=control-plane workload_label=monitoring\nthor   ansible_host=10.0.0.149 node_name=thor   node_role=worker        workload_label=apps<\/code><\/pre><p class=\"wp-block-paragraph\">Each host has variables defined inline on the same line. These are host-specific:<\/p><ul class=\"wp-block-list\"><li>ansible_host: the actual IP Ansible SSH&#8217;s into<\/li><li>node_name: used to set the hostname and K3s node name<\/li><li>node_role: used in when: conditions to run tasks only on the right node<\/li><li>workload_label: applied as a Kubernetes label for pod scheduling<\/li><\/ul><p class=\"wp-block-paragraph\">Example of node_role in use:<\/p><pre class=\"wp-block-code\"><code># In common\/tasks\/main.yml\n- name: Install dev toolchains on worker node\n  block:\n    - name: Install Python\n      apt:\n        name: python3\n        state: present\n    # ... more tasks\n  when: node_role == \"worker\"   # \u2190 only runs on thor<\/code><\/pre><p class=\"wp-block-paragraph\">This single condition means the same role can run on both nodes but behave differently based on the node&#8217;s role. No separate playbooks needed.<\/p><h3 class=\"wp-block-heading\">[all:vars]: Shared Variables<\/h3><pre class=\"wp-block-code\"><code>[all:vars]\ntimezone=America\/Vancouver\nk3s_server_ip=10.0.0.148\nmetallb_ip_range=10.0.0.200-10.0.0.220\n...<\/code><\/pre><p class=\"wp-block-paragraph\">Variables under [all:vars] apply to every host. This is where I put:<\/p><ul class=\"wp-block-list\"><li>Configuration shared across both nodes (timezone, lab_user)<\/li><li>K3s cluster settings (server IP, port)<\/li><li>Tool-specific settings (Grafana password, Prometheus retention)<\/li><li>Domain and DNS settings<\/li><\/ul><p class=\"wp-block-paragraph\">Lesson learned: I originally put variables in group_vars\/all.yml (the standard Ansible approach). This works fine when you run playbooks from the project root. But when you run from a subdirectory (playbooks\/), Ansible can&#8217;t find group_vars\/. Moving everything into [all:vars] in the inventory file solves this permanently, the inventory file is always explicitly specified with -i.<\/p><h3 class=\"wp-block-heading\">site.yml :The Master Playbook<\/h3><pre class=\"wp-block-code\"><code>---\n# Run everything: ansible-playbook -i inventory\/hosts.ini site.yml\n\n- name: Common setup on all nodes\n  hosts: all\n  become: yes\n  roles:\n    - common\n    - networking\n\n- name: K3s control plane\n  hosts: control_plane\n  become: yes\n  roles:\n    - k3s_server\n\n- name: K3s worker\n  hosts: workers\n  become: yes\n  roles:\n    - k3s_agent\n\n- name: Longhorn storage\n  hosts: control_plane\n  become: yes\n  roles:\n    - longhorn\n\n- name: Monitoring stack\n  hosts: control_plane\n  become: yes\n  roles:\n    - monitoring\n\n- name: CI\/CD stack\n  hosts: control_plane\n  become: yes\n  roles:\n    - cicd\n\n- name: GitOps stack\n  hosts: control_plane\n  become: yes\n  roles:\n    - gitops\n\n- name: DNS + Ingress\n  hosts: control_plane\n  become: yes\n  roles:\n    - dns_tls\n\n- name: CoreDNS external resolver\n  hosts: control_plane\n  become: yes\n  roles:\n    - pihole\n\n- name: Homepage\n  hosts: control_plane\n  become: yes\n  roles:\n    - homepage\n\n- name: Registry DNS + Docker config\n  hosts: all\n  become: yes\n  roles:\n    - registry_dns<\/code><\/pre><p class=\"wp-block-paragraph\">Key design decisions:<\/p><p class=\"wp-block-paragraph\">Why separate plays for control plane vs workers? K3s must be installed on the server (jarvis) before the agent (thor) can join. Running them in separate plays ensures the ordering is always correct, k3s_server play completes fully before k3s_agent starts.<\/p><p class=\"wp-block-paragraph\">Why does longhorn, monitoring, cicd, gitops target control_plane? These roles use kubectl and helm to deploy workloads. kubectl is only installed on the control plane node (jarvis has the kubeconfig). The actual pods may run on tho, but the commands that create them run from jarvis.<\/p><p class=\"wp-block-paragraph\">Why does registry_dns target all? The registry DNS role configures Docker daemon trust settings on both nodes. Both jarvis and thor need to be able to pull images from the local registry.<\/p><h3 class=\"wp-block-heading\">Individual Playbooks: Running One Phase at a Time<\/h3><pre class=\"wp-block-code\"><code># playbooks\/01-common.yml\n- name: Common OS setup\n  hosts: all\n  become: yes\n  roles:\n    - common\n    - networking<\/code><\/pre><p class=\"wp-block-paragraph\">Every phase has its own playbook. This is critical for debugging.<\/p><p class=\"wp-block-paragraph\">When you run the full site.yml and something fails in phase 5 (monitoring), you fix the issue and rerun just:<\/p><pre class=\"wp-block-code\"><code>ansible-playbook -i inventory\/hosts.ini playbooks\/04-monitoring.yml<\/code><\/pre><p class=\"wp-block-paragraph\">You don&#8217;t have to rerun the previous 4 phases. Ansible is idempotent, tasks that already completed show ok and are skipped. Only tasks that haven&#8217;t run yet (or failed) actually execute.<\/p><p class=\"wp-block-paragraph\">Running order:<\/p><pre class=\"wp-block-code\"><code>ansible-playbook -i inventory\/hosts.ini playbooks\/01-common.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/02-k3s.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/03-longhorn.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/04-monitoring.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/05-cicd.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/06-gitops.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/07-dns-tls.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/08-pihole.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/09-homepage.yml\nansible-playbook -i inventory\/hosts.ini playbooks\/10-registry.yml<\/code><\/pre><h3 class=\"wp-block-heading\">Role Structure, Inside a Role<\/h3><p class=\"wp-block-paragraph\">Every role follows the same pattern:<\/p><pre class=\"wp-block-code\"><code>roles\/common\/\n\u251c\u2500\u2500 tasks\/\n\u2502   \u2514\u2500\u2500 main.yml     \u2190 the actual tasks\n\u251c\u2500\u2500 handlers\/\n\u2502   \u2514\u2500\u2500 main.yml     \u2190 tasks triggered by notify:\n\u2514\u2500\u2500 files\/\n    \u2514\u2500\u2500 ...          \u2190 static files to copy to remote<\/code><\/pre><p class=\"wp-block-paragraph\">tasks\/main.yml is where the work happens. Ansible automatically runs tasks\/main.yml when a role is included.<\/p><p class=\"wp-block-paragraph\">handlers\/main.yml contains tasks that only run when triggered by a notify: directive. The classic example:<\/p><pre class=\"wp-block-code\"><code># In tasks\/main.yml\n- name: Write Docker daemon config\n  copy:\n    dest: \/etc\/docker\/daemon.json\n    content: |\n      {\"insecure-registries\": [\"registry.rezaops.com\"]}\n  notify: Restart Docker     # \u2190 triggers the handler\n\n# In handlers\/main.yml\n- name: Restart Docker\n  systemd:\n    name: docker\n    state: restarted<\/code><\/pre><p class=\"wp-block-paragraph\">The handler only runs if the copy task actually changed something. If Docker&#8217;s config already has the right content, the handler doesn&#8217;t run, no unnecessary restarts.<\/p><p class=\"wp-block-paragraph\">files\/ holds static files that get copied to remote machines with the copy module. The homepage index.html lives here.<\/p><h3 class=\"wp-block-heading\">Ansible Key Concepts Recap<\/h3><figure class=\"wp-block-table\"><table><thead><tr><th>Concept<\/th><th>What it is<\/th><th>Example<\/th><\/tr><\/thead><tbody><tr><td>Inventory<\/td><td>List of machines + variables<\/td><td>hosts.ini<\/td><\/tr><tr><td>Play<\/td><td>A set of tasks targeting specific hosts<\/td><td>hosts: control_plane<\/td><\/tr><tr><td>Role<\/td><td>A reusable collection of tasks<\/td><td>common, monitoring<\/td><\/tr><tr><td>Task<\/td><td>A single action<\/td><td>Install a package, copy a file<\/td><\/tr><tr><td>Module<\/td><td>The tool a task uses<\/td><td>apt, copy, shell, systemd<\/td><\/tr><tr><td>Variable<\/td><td>A named value<\/td><td>{{ node_name }}, {{ base_domain }}<\/td><\/tr><tr><td>Handler<\/td><td>A task triggered by notify:<\/td><td>Restart a service after config change<\/td><\/tr><tr><td>Idempotent<\/td><td>Safe to run multiple times<\/td><td>Most built-in modules are idempotent<\/td><\/tr><tr><td>become: yes<\/td><td>Use sudo<\/td><td>Required for system-level changes<\/td><\/tr><tr><td>when:<\/td><td>Conditional execution<\/td><td>when: node_role == &#8220;worker&#8221;<\/td><\/tr><\/tbody><\/table><\/figure><h3 class=\"wp-block-heading\">Useful Ansible Commands<\/h3><pre class=\"wp-block-code\"><code># Test connectivity to all nodes\nansible all -i inventory\/hosts.ini -m ping\n\n# Dry run \u2014 shows what would change without doing it\nansible-playbook -i inventory\/hosts.ini site.yml --check\n\n# Verbose output for debugging\nansible-playbook -i inventory\/hosts.ini playbooks\/01-common.yml -vvv\n\n# Run from a specific task (after a failure)\nansible-playbook -i inventory\/hosts.ini playbooks\/02-k3s.yml \\\n  --start-at-task \"Install K3s server\"\n\n# Run only tasks with a specific tag\nansible-playbook -i inventory\/hosts.ini site.yml --tags monitoring\n\n# List all tasks without running them\nansible-playbook -i inventory\/hosts.ini site.yml --list-tasks<\/code><\/pre><h3 class=\"wp-block-heading\">What&#8217;s Next<\/h3><p class=\"wp-block-paragraph\">In Part 3 I&#8217;ll do a deep dive into the Ansible roles themselves , walking through the common, networking, and k3s_server role YAML files line by line, explaining what each task does and why it&#8217;s needed.<\/p><h3 class=\"wp-block-heading\">Deep Dive, group_vars, host_vars, and Why We Moved Everything to hosts.ini<\/h3><p class=\"wp-block-paragraph\">This section explains something that tripped me up badly during this build, and understanding it will save you hours of debugging.<\/p><h3 class=\"wp-block-heading\">The Standard Ansible Way<\/h3><p class=\"wp-block-paragraph\">Ansible has a clean, organized way to manage variables:<\/p><pre class=\"wp-block-code\"><code>inventory\/\n  hosts.ini          \u2190 just the hostnames and IPs\ngroup_vars\/\n  all.yml            \u2190 variables for ALL hosts\n  control_plane.yml  \u2190 variables only for [control_plane] group\nhost_vars\/\n  jarvis.yml         \u2190 variables only for jarvis\n  thor.yml           \u2190 variables only for thor<\/code><\/pre><p class=\"wp-block-paragraph\">group_vars\/all.yml: shared variables every host gets:<\/p><pre class=\"wp-block-code\"><code># group_vars\/all.yml\ntimezone: America\/Vancouver\nlab_user: reza\nk3s_server_ip: \"10.0.0.148\"\nmetallb_ip_range: \"10.0.0.200-10.0.0.220\"\ngrafana_admin_password: \"homelab123\"<\/code><\/pre><p class=\"wp-block-paragraph\">host_vars\/jarvis.yml: variables specific to jarvis only:<\/p><pre class=\"wp-block-code\"><code># host_vars\/jarvis.yml\nnode_ip: \"10.0.0.148\"\nnode_name: \"jarvis\"\nnode_role: \"control-plane\"\nworkload_label: \"monitoring\"<\/code><\/pre><p class=\"wp-block-paragraph\">host_vars\/thor.yml: variables specific to thor only:<\/p><pre class=\"wp-block-code\"><code># host_vars\/thor.yml\nnode_ip: \"10.0.0.149\"\nnode_name: \"thor\"\nnode_role: \"worker\"\nworkload_label: \"apps\"<\/code><\/pre><p class=\"wp-block-paragraph\">This structure is clean, readable, and is the recommended approach in the official Ansible documentation. But it stopped working for me, and here&#8217;s exactly why.<\/p><h3 class=\"wp-block-heading\">Why group_vars and host_vars Failed<\/h3><p class=\"wp-block-paragraph\">When I ran:<\/p><pre class=\"wp-block-code\"><code>ansible-playbook -i inventory\/hosts.ini playbooks\/01-common.yml<\/code><\/pre><p class=\"wp-block-paragraph\">I got this error:<\/p><pre class=\"wp-block-code\"><code>[ERROR]: 'node_name' is undefined\n[ERROR]: 'timezone' is undefined<\/code><\/pre><p class=\"wp-block-paragraph\">Both variables were clearly defined in host_vars\/jarvis.yml and group_vars\/all.yml. So why were they undefined?<\/p><p class=\"wp-block-paragraph\">The root cause: Ansible resolves group_vars\/ and host_vars\/ relative to the inventory file location, not relative to where you run the command from.<\/p><p class=\"wp-block-paragraph\">My inventory was at inventory\/hosts.ini. So Ansible looked for:<\/p><pre class=\"wp-block-code\"><code>inventory\/group_vars\/all.yml       \u2190 doesn't exist\ninventory\/host_vars\/jarvis.yml     \u2190 doesn't exist<\/code><\/pre><p class=\"wp-block-paragraph\">My actual files were at:<\/p><pre class=\"wp-block-code\"><code>group_vars\/all.yml                 \u2190 exists but not found\nhost_vars\/jarvis.yml               \u2190 exists but not found<\/code><\/pre><p class=\"wp-block-paragraph\">Ansible never found them. No error about missing files, variables were just silently undefined.<\/p><p class=\"wp-block-paragraph\">There are two fixes for this:<\/p><p class=\"wp-block-paragraph\">Fix 1: Move group_vars and host_vars next to the inventory file:<\/p><pre class=\"wp-block-code\"><code>inventory\/\n  hosts.ini\n  group_vars\/\n    all.yml              \u2190 now found\n  host_vars\/\n    jarvis.yml           \u2190 now found\n    thor.yml             \u2190 now found<\/code><\/pre><p class=\"wp-block-paragraph\">This works and keeps the clean structure. If you prefer this layout, use it.<\/p><p class=\"wp-block-paragraph\">Fix 2: Move everything into hosts.ini directly (what I chose):<\/p><pre class=\"wp-block-code\"><code>[control_plane]\njarvis ansible_host=10.0.0.148 ansible_user=reza node_ip=10.0.0.148 node_name=jarvis node_role=control-plane workload_label=monitoring\n\n[workers]\nthor ansible_host=10.0.0.149 ansible_user=reza node_ip=10.0.0.149 node_name=thor node_role=worker workload_label=apps\n\n[all:vars]\nansible_python_interpreter=\/usr\/bin\/python3\nansible_ssh_private_key_file=~\/.ssh\/id_ed25519\ntimezone=America\/Vancouver\nlab_user=reza\nlab_network=10.0.0.0\/24\nk3s_server_ip=10.0.0.148\nk3s_server_port=6443\nmetallb_ip_range=10.0.0.200-10.0.0.220\nlonghorn_replica_count=2\nlonghorn_version=1.6.2\ngrafana_admin_password=homelab123\nprometheus_retention=15d\njenkins_admin_password=homelab123\nregistry_ip=10.0.0.206\nregistry_port=5000\nbase_domain=rezaops.com\ncp_ip=10.0.0.148\nworker_ip=10.0.0.149\ningress_ip=10.0.0.200\naws_region=us-east-1\naws_access_key_id=REPLACE_WITH_YOUR_KEY\naws_secret_access_key=REPLACE_WITH_YOUR_SECRET<\/code><\/pre><p class=\"wp-block-paragraph\">I chose this because it puts everything in one file, no path resolution issues possible. The inventory file is always explicitly referenced with -i inventory\/hosts.ini so Ansible always finds it.<\/p><h3 class=\"wp-block-heading\">The .ini Format Limitation: Lists and Dicts<\/h3><p class=\"wp-block-paragraph\">There&#8217;s one more catch with hosts.ini: it only supports strings. You can&#8217;t define lists or dictionaries.<\/p><p class=\"wp-block-paragraph\">For example, my internal_dns_records variable is a list of objects. In YAML format it looks like this:<\/p><pre class=\"wp-block-code\"><code>internal_dns_records: [{\"name\":\"grafana\",\"ip\":\"10.0.0.200\"},{\"name\":\"jenkins\",\"ip\":\"10.0.0.200\"}]<\/code><\/pre><p class=\"wp-block-paragraph\">When you try to loop over it:<\/p><pre class=\"wp-block-code\"><code>- name: Add DNS records\n  shell: \"nsupdate -k {{ dns_key }} &lt;&lt;EOF\\nserver {{ dns_server }}\\nupdate add {{ item.name }}.{{ base_domain }} 3600 A {{ item.ip }}\\nsend\\nEOF\"\n  loop: \"{{ internal_dns_records }}\"<\/code><\/pre><p class=\"wp-block-paragraph\">Ansible fails with:<\/p><p class=\"wp-block-paragraph\">The loop value must resolve to a &#8216;list&#8217;, not &#8216;str&#8217;.<\/p><p class=\"wp-block-paragraph\">Even though the value looks like a JSON list, .ini format treats everything as a string. Ansible doesn&#8217;t automatically parse it into a list.<\/p><p class=\"wp-block-paragraph\">Solutions:<\/p><p class=\"wp-block-paragraph\">Option A, Use YAML inventory instead of .ini:<\/p><pre class=\"wp-block-code\"><code># inventory.yml\nall:\n  hosts:\n    jarvis:\n      ansible_host: 10.0.0.148\n      node_name: jarvis\n      node_role: control-plane\n  vars:\n    internal_dns_records:\n      - name: grafana\n        ip: 10.0.0.200\n      - name: jenkins\n        ip: 10.0.0.200<\/code><\/pre><p class=\"wp-block-paragraph\">YAML inventory supports native lists and dicts. But it&#8217;s more verbose than .ini format for simple setups.<\/p><p class=\"wp-block-paragraph\">Option B: Avoid loops, use direct values:<\/p><p class=\"wp-block-paragraph\">In my Route 53 automation I wrote individual commands instead of loops:<\/p><pre class=\"wp-block-code\"><code>aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID \\\n  --change-batch '{\n    \"Changes\": [{\n      \"Action\": \"UPSERT\",\n      \"ResourceRecordSet\": {\n        \"Name\": \"grafana.rezaops.com\",\n        \"Type\": \"A\",\n        \"TTL\": 300,\n        \"ResourceRecords\": [{\"Value\": \"10.0.0.200\"}]\n      }\n    }]\n  }'<\/code><\/pre><p class=\"wp-block-paragraph\">For a home lab with a handful of services, this is perfectly fine. You only need loops when you&#8217;re managing dozens of records.<\/p><h3 class=\"wp-block-heading\">Variable Precedence in Ansible<\/h3><p class=\"wp-block-paragraph\">When the same variable is defined in multiple places, Ansible uses the highest-precedence value. The full order from lowest to highest:<\/p><ol class=\"wp-block-list\"><li>command line values (ansible-playbook &#8211;extra-vars)<\/li><li>role defaults (roles\/common\/defaults\/main.yml)<\/li><li>inventory file vars ([all:vars] in hosts.ini)<\/li><li>host_vars files (host_vars\/jarvis.yml)<\/li><li>group_vars files (group_vars\/all.yml)<\/li><li>group_vars\/all (group_vars\/all.yml)<\/li><li>role vars (roles\/common\/vars\/main.yml)<\/li><li>task vars (vars: in a task)<\/li><li>extra vars (ansible-playbook -e &#8220;key=value&#8221;)<\/li><\/ol><p class=\"wp-block-paragraph\">In practice for a home lab, you only need two:<\/p><ul class=\"wp-block-list\"><li>[all:vars] in hosts.ini for shared variables<\/li><li>Inline host variables for per-node settings<\/li><\/ul><p class=\"wp-block-paragraph\">This covers 95% of real-world cases. The full precedence chain only matters when you&#8217;re debugging why a variable isn&#8217;t taking effect, check if it&#8217;s being overridden somewhere higher in the chain.<\/p><h3 class=\"wp-block-heading\">Summary: Variable Loading Rules<\/h3><ol class=\"wp-block-list\"><li>[all:vars] in .ini Works: always, file is explicitly referenced Limit: strings only, no lists or dicts<\/li><li>group_vars\/all.yml Works: when placed in the same directory as the inventory file Limit: path must be relative to inventory location<\/li><li>host_vars\/jarvis.yml Works: when placed in the same directory as the inventory file Limit: same path issue as group_vars<\/li><li>YAML inventory Works: always, supports complex types Limit: more verbose than .ini format<\/li><li>&#8211;extra-vars Works: always Limit: command-line only, not repeatable across runs<\/li><\/ol><p class=\"wp-block-paragraph\">Recommendation: For a home lab with simple string variables, put everything in [all:vars]. For projects with complex data structures (lists, nested dicts), switch to YAML inventory format.<\/p>","protected":false},"excerpt":{"rendered":"<p>Overview In Part 1, I walked through every OS configuration step needed before installing Kubernetes. In this article I&#8217;ll explain how I&hellip;<\/p>\n","protected":false},"author":2,"featured_media":117,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"2","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-02-ansible-project-structure-reza-chegini-29k8c\/","footnotes":""},"categories":[63,10,62],"tags":[65,67,8,66,6],"series":[64],"class_list":["post-115","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ansible","category-devops","category-home-lab","tag-ansible","tag-automation","tag-devops","tag-home-lab","tag-yaml","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-02-ansible-project-structure-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=115"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/115\/revisions"}],"predecessor-version":[{"id":116,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/115\/revisions\/116"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/117"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=115"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}