{"id":168,"date":"2026-06-19T12:00:00","date_gmt":"2026-06-19T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=168"},"modified":"2026-10-30T07:09:51","modified_gmt":"2026-10-30T07:09:51","slug":"part-07-ci-cd-pipeline-jenkins-registry-and-jenkinsfile","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=168","title":{"rendered":"Part 07 &#8211; CI\/CD Pipeline: Jenkins, Registry and Jenkinsfile"},"content":{"rendered":"<h3 class=\"wp-block-heading\">Overview<\/h3><p class=\"wp-block-paragraph\">In Part 6 we got full observability across the cluster. In this article I build the CI\/CD pipeline that takes code from a developer&#8217;s machine all the way to a running pod in Kubernetes.<\/p><p class=\"wp-block-paragraph\">The stack:<\/p><ul class=\"wp-block-list\"><li>Jenkins, runs pipelines and builds Docker images<\/li><li>Local container registry, stores built images inside the home lab<\/li><li>Jenkinsfile, the pipeline definition that connects everything<\/li><\/ul><p class=\"wp-block-paragraph\">By the end of this article, pushing code to GitHub automatically triggers a build, pushes an image to the local registry, and prepares it for deployment via ArgoCD.<\/p><h3 class=\"wp-block-heading\">CI\/CD Flow<\/h3><pre class=\"wp-block-code\"><code>Developer pushes code to GitHub\n        |\n        | webhook triggers\n        v\nJenkins (on jarvis)\n  pulls code from GitHub\n  runs tests\n  builds Docker image\n  tags image with git commit SHA\n  pushes to local registry\n        |\n        | image available at:\n        | registry.rezaops.com\/myapp:abc1234\n        v\nArgoCD detects image tag change\n  deploys new pod on thor<\/code><\/pre><h3 class=\"wp-block-heading\">Role: cicd<\/h3><h3 class=\"wp-block-heading\">Full tasks\/main.yml<\/h3><pre class=\"wp-block-code\"><code>- name: Add Jenkins Helm repo\n  shell: |\n    helm repo add jenkins https:\/\/charts.jenkins.io\n    helm repo update\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Adds the official Jenkins Helm chart repository. helm repo update ensures we get the latest chart version, which matters because Jenkins chart APIs change between versions. The adminPassword to admin.password rename we hit in Part 12 is a good example of this.<\/p><pre class=\"wp-block-code\"><code>- name: Create cicd namespace\n  shell: |\n    kubectl create namespace cicd \\\n      --dry-run=client -o yaml | kubectl apply -f -\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Separate namespace for all CI\/CD tools. Keeps them isolated from monitoring and app namespaces.<\/p><h3 class=\"wp-block-heading\">Installing Jenkins<\/h3><pre class=\"wp-block-code\"><code>- name: Install Jenkins\n  shell: |\n    helm upgrade --install jenkins jenkins\/jenkins \\\n      --namespace cicd \\\n      --set controller.serviceType=LoadBalancer \\\n      --set controller.admin.password={{ jenkins_admin_password }} \\\n      --set controller.nodeSelector.\"workload\"=monitoring \\\n      --set agent.nodeSelector.\"workload\"=apps \\\n      --set persistence.storageClass=longhorn \\\n      --set persistence.size=20Gi\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Helm values explained:<\/p><ul class=\"wp-block-list\"><li>controller.serviceType, LoadBalancer, Jenkins UI gets a real IP from MetalLB<\/li><li>controller.admin.password, homelab123, admin login password<\/li><li>controller.nodeSelector.&#8221;workload&#8221;=monitoring, Jenkins controller runs on jarvis<\/li><li>agent.nodeSelector.&#8221;workload&#8221;=apps, Jenkins build agents run on thor<\/li><li>persistence.storageClass, longhorn, Jenkins data stored in a replicated Longhorn volume<\/li><li>persistence.size, 20Gi, 20GB for Jenkins home directory<\/li><\/ul><p class=\"wp-block-paragraph\">Why controller on jarvis, agents on thor? Jenkins has two components. The controller is the main Jenkins server that manages jobs, stores config, and serves the UI. It is lightweight in terms of CPU but needs stable storage. The agents are the workers that actually run pipeline steps including build, test, and push. They are CPU-intensive during builds.<\/p><p class=\"wp-block-paragraph\">Separating them means heavy build workloads run on thor without competing with the Jenkins UI and other monitoring tools on jarvis.<\/p><p class=\"wp-block-paragraph\">Why Longhorn for persistence? Jenkins stores job history, build logs, plugin data, and credentials in \/var\/jenkins_home. Without persistent storage this is lost every time the pod restarts. With Longhorn it is replicated across both nodes.<\/p><pre class=\"wp-block-code\"><code>- name: Wait for Jenkins to be ready\n  shell: |\n    kubectl wait --namespace cicd \\\n      --for=condition=ready pod \\\n      --selector=app.kubernetes.io\/name=jenkins \\\n      --timeout=180s\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml\n  retries: 3\n  delay: 30<\/code><\/pre><p class=\"wp-block-paragraph\">Jenkins takes longer to initialize than most pods because it downloads plugins on first start. timeout=180s with retries: 3 gives it up to 9 minutes total.<\/p><h3 class=\"wp-block-heading\">Local Container Registry<\/h3><pre class=\"wp-block-code\"><code>- name: Deploy local container registry\n  shell: |\n    cat &lt;&lt;EOF | kubectl apply -f -\n    apiVersion: apps\/v1\n    kind: Deployment\n    metadata:\n      name: registry\n      namespace: registry\n    spec:\n      replicas: 1\n      selector:\n        matchLabels:\n          app: registry\n      template:\n        metadata:\n          labels:\n            app: registry\n        spec:\n          nodeSelector:\n            workload: monitoring\n          containers:\n          - name: registry\n            image: registry:2\n            ports:\n            - containerPort: 5000\n            env:\n            - name: REGISTRY_STORAGE_DELETE_ENABLED\n              value: \"true\"\n            volumeMounts:\n            - name: registry-data\n              mountPath: \/var\/lib\/registry\n          volumes:\n          - name: registry-data\n            persistentVolumeClaim:\n              claimName: registry-pvc\n    ---\n    apiVersion: v1\n    kind: PersistentVolumeClaim\n    metadata:\n      name: registry-pvc\n      namespace: registry\n    spec:\n      accessModes: [ReadWriteOnce]\n      storageClassName: longhorn\n      resources:\n        requests:\n          storage: 50Gi\n    ---\n    apiVersion: v1\n    kind: Service\n    metadata:\n      name: registry\n      namespace: registry\n    spec:\n      type: LoadBalancer\n      selector:\n        app: registry\n      ports:\n      - port: 5000\n        targetPort: 5000\n    EOF\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Three Kubernetes resources in one apply, Deployment, PVC, and Service.<\/p><p class=\"wp-block-paragraph\">REGISTRY_STORAGE_DELETE_ENABLED=true enables image deletion. By default the registry does not allow deleting images. This is useful for cleaning up old images when disk space fills up.<\/p><p class=\"wp-block-paragraph\">The registry stores all image layers at \/var\/lib\/registry inside the container. Mounting a 50GB Longhorn PVC here means images persist across pod restarts and are replicated across both nodes.<\/p><p class=\"wp-block-paragraph\">The registry Service is type LoadBalancer so MetalLB assigns it a real IP. Always verify the actual assigned IP:<\/p><pre class=\"wp-block-code\"><code>kubectl get svc -n registry\n# use whatever EXTERNAL-IP shows<\/code><\/pre><p class=\"wp-block-paragraph\">In our setup MetalLB assigned 10.0.0.206, not 10.0.0.200 as I expected. MetalLB assigns IPs sequentially so you can never assume which IP a service gets.<\/p><h3 class=\"wp-block-heading\">Configuring K3s to Trust the Registry<\/h3><pre class=\"wp-block-code\"><code>- name: Create K3s registries config\n  copy:\n    dest: \/etc\/rancher\/k3s\/registries.yaml\n    content: |\n      mirrors:\n        \"registry.{{ base_domain }}\":\n          endpoint:\n            - \"http:\/\/registry.{{ base_domain }}\"\n        \"{{ registry_ip }}:{{ registry_port }}\":\n          endpoint:\n            - \"http:\/\/{{ registry_ip }}:{{ registry_port }}\"<\/code><\/pre><p class=\"wp-block-paragraph\">By default K3s&#8217;s containerd runtime only trusts HTTPS registries. Since our registry uses HTTP, we need to explicitly tell containerd to trust it. registries.yaml maps the registry hostname to its actual endpoint and tells containerd to use HTTP.<\/p><p class=\"wp-block-paragraph\">Without this file, when Kubernetes tries to pull an image from registry.rezaops.com, containerd rejects it with:<\/p><pre class=\"wp-block-code\"><code>Failed to pull image: http: server gave HTTP response to HTTPS client<\/code><\/pre><h3 class=\"wp-block-heading\">The Jenkinsfile Pipeline<\/h3><pre class=\"wp-block-code\"><code>pipeline {\n  agent { label 'k3s-worker' }\n\n  environment {\n    REGISTRY  = 'registry.rezaops.com'\n    APP_NAME  = 'myapp'\n    IMAGE     = \"${REGISTRY}\/${APP_NAME}\"\n    TAG       = \"${GIT_COMMIT[0..7]}\"\n  }\n\n  stages {\n\n    stage('Checkout') {\n      steps {\n        checkout scm\n        echo \"Building commit: ${TAG}\"\n      }\n    }\n\n    stage('Test') {\n      steps {\n        sh '''\n          pip install -r requirements.txt\n          python manage.py test\n        '''\n      }\n    }\n\n    stage('Build') {\n      steps {\n        sh \"docker build -t ${IMAGE}:${TAG} .\"\n        sh \"docker tag  ${IMAGE}:${TAG} ${IMAGE}:latest\"\n      }\n    }\n\n    stage('Push') {\n      steps {\n        sh \"docker push ${IMAGE}:${TAG}\"\n        sh \"docker push ${IMAGE}:latest\"\n      }\n    }\n\n    stage('Update Manifest') {\n      steps {\n        sh \"\"\"\n          git clone https:\/\/github.com\/mohammadrezachegini\/homelab-gitops.git\n          cd homelab-gitops\n          sed -i 's|image: ${IMAGE}:.*|image: ${IMAGE}:${TAG}|' \\\n            apps\/overlays\/prod\/deployment.yaml\n          git config user.email \"ci@rezaops.com\"\n          git config user.name \"Jenkins\"\n          git add .\n          git commit -m \"ci: update ${APP_NAME} to ${TAG} [skip ci]\"\n          git push\n        \"\"\"\n      }\n    }\n  }\n\n  post {\n    always { cleanWs() }\n    success { echo \"Build ${TAG} pushed to ${IMAGE}\" }\n    failure { echo \"Build failed\" }\n  }\n}<\/code><\/pre><p class=\"wp-block-paragraph\">Pipeline stages explained:<\/p><p class=\"wp-block-paragraph\">agent, label k3s-worker routes the pipeline to run on thor, keeping heavy build workloads off the control plane.<\/p><p class=\"wp-block-paragraph\">TAG = GIT_COMMIT[0..7] uses the first 8 characters of the git commit SHA as the image tag. Every build produces a unique traceable image. You can always trace a running pod back to the exact code that built it.<\/p><p class=\"wp-block-paragraph\">The Build stage uses Docker layer caching. COPY requirements.txt comes before COPY . . in the Dockerfile so if only code changes and not dependencies, Docker reuses the cached pip install layer. Builds go from 3 minutes to 10 seconds.<\/p><p class=\"wp-block-paragraph\">The Update Manifest stage is the GitOps bridge. Instead of Jenkins deploying directly to Kubernetes, it updates the deployment manifest in the GitOps repository. ArgoCD watches that repository and applies the change automatically.<\/p><p class=\"wp-block-paragraph\">[skip ci] in the commit message prevents GitHub from triggering Jenkins again for this commit. Without it you get an infinite loop.<\/p><p class=\"wp-block-paragraph\">cleanWs() in the post block deletes all build files from the workspace. Prevents disk from filling up with old build artifacts over time.<\/p><h3 class=\"wp-block-heading\">Verifying Jenkins is Working<\/h3><p class=\"wp-block-paragraph\">Get Jenkins URL:<\/p><pre class=\"wp-block-code\"><code>kubectl get svc -n cicd jenkins<\/code><\/pre><p class=\"wp-block-paragraph\">Access Jenkins at http:\/\/jenkins.rezaops.com and login with admin and your password.<\/p><p class=\"wp-block-paragraph\">Check the Longhorn volume is bound:<\/p><pre class=\"wp-block-code\"><code>kubectl get pvc -n cicd\nNAME      STATUS   VOLUME      CAPACITY   STORAGECLASS\njenkins   Bound    pvc-xxxxx   20Gi       longhorn<\/code><\/pre><p class=\"wp-block-paragraph\">Test the registry:<\/p><pre class=\"wp-block-code\"><code>docker tag nginx:alpine registry.rezaops.com\/nginx:alpine\ndocker push registry.rezaops.com\/nginx:alpine<\/code><\/pre><pre class=\"wp-block-code\"><code>curl http:\/\/registry.rezaops.com\/v2\/_catalog\n# {\"repositories\":[\"nginx\"]}<\/code><\/pre><h3 class=\"wp-block-heading\">What&#8217;s Next<\/h3><p class=\"wp-block-paragraph\">In Part 8 I will cover the GitOps stack, ArgoCD, Vault, and External Secrets Operator. I will explain how ArgoCD watches a GitHub repository and automatically keeps the cluster in sync, and how Vault stores secrets that get injected into pods at runtime.<\/p>","protected":false},"excerpt":{"rendered":"<p>Overview In Part 6 we got full observability across the cluster. In this article I build the CI\/CD pipeline that takes code&hellip;<\/p>\n","protected":false},"author":2,"featured_media":170,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"7","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-07-cicd-pipeline-jenkins-registry-jenkinsfile-reza-chegini-6iepc\/","footnotes":""},"categories":[27,10,62],"tags":[30,103,8,66,41],"series":[64],"class_list":["post-168","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci-cd","category-devops","category-home-lab","tag-ci-cd","tag-container-registry","tag-devops","tag-home-lab","tag-jenkins","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-07-cicd-pipeline-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=168"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":169,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/168\/revisions\/169"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/170"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=168"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}