Step-by-Step Guide to Setting Up a CI/CD Pipeline with GitLab and AWS

In this post, I’m sharing a detailed breakdown of a CI/CD pipeline I designed and implemented using GitLab CI/CD, Docker, and AWS. This pipeline automates the build, test, packaging, and deployment processes to ensure efficient and reliable application delivery.

Here’s the full pipeline code:

stages:
- build
- test
- sonarqube
- package
- deploy

build-job:
  stage: build
  tags:
  - ec2
  - shell
  script:
  - pip install -r requirements.txt

sast:
  stage: test
include:
- template: Jobs/SAST.gitlab-ci.yml

sonarqube-check:
  stage: sonarqube
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [ "" ]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Analysis task cache
    GIT_DEPTH: "0" # Fetch all branches
  cache:
    key: "${CI_JOB_NAME}"
    paths:
    - .sonar/cache
  script:
  - sonar-scanner
  allow_failure: true
  rules:
  - if: $CI_COMMIT_BRANCH == 'main'

package-job:
  stage: package
  tags:
  - ec2
  - shell
  script:
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD registry.gitlab.com
  - docker build -t registry.gitlab.com/lowyiiii/python-project .
  - docker push registry.gitlab.com/lowyiiii/python-project

deploy-job:
  stage: deploy
  tags:
  - ec2
  - shell
  script:
  - aws configure set aws_access_key $AWS_ACCESS_KEY_ID
  - aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
  - aws configure set default_region $AWS_DEFAULT_REGION
  - kubectl apply -f Application.yaml 

Setting up a CI/CD pipeline can seem complex, but breaking it down into manageable steps reveals how straightforward and efficient it can be. Below, I’ll guide you through a GitLab pipeline I created to automate the development lifecycle, from code integration to deployment in a Kubernetes environment on AWS.

The Big Picture

The pipeline is divided into five stages:

stages:
- build
- test
- sonarqube
- package
- deploy 

Each stage plays a critical role in automating the process of building, testing, and deploying an application.

Step 1: Build Stage

This stage prepares the application by installing dependencies and setting up the environment.

build-job:
  stage: build
  tags:
  - ec2
  - shell
  script:
  - pip install -r requirements.txt 

What Happens Here?

1. Tags: The tags (ec2, shell) ensure the job runs on a specific GitLab runner (in this case, hosted on an EC2 instance).
2. Script: The command pip install -r requirements.txt installs all the dependencies defined in the requirements.txt file, ensuring the application environment is ready for testing and packaging.

Step 2: Static Application Security Testing (SAST)

This stage integrates GitLab’s built-in security scanning.

sast:
  stage: test
include:
- template: Jobs/SAST.gitlab-ci.yml 

What Happens Here?

1. Security Check: The Jobs/SAST.gitlab-ci.yml template runs static application security tests on the codebase to identify vulnerabilities.
2. Automation: These scans ensure that security is embedded early in the development cycle.

💡 Why SAST? Security vulnerabilities caught during development are cheaper and easier to fix than those caught in production.

Step 3: SonarQube Code Quality Check

This stage uses SonarQube to evaluate the codebase for quality and maintainability.

sonarqube-check:
  stage: sonarqube
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [ "" ]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
    GIT_DEPTH: "0"
  cache:
    key: "${CI_JOB_NAME}"
    paths:
    - .sonar/cache
  script:
  - sonar-scanner
  allow_failure: true
  rules:
  - if: $CI_COMMIT_BRANCH == 'main' 

What Happens Here?

1. Docker Image: The SonarQube scanner runs in a Docker container to ensure a consistent runtime environment.
2. Variables:SONAR_USER_HOME ensures cache files are stored locally.GIT_DEPTH fetches the full Git history, which SonarQube needs for detailed analysis.
3. Caching: Improves performance by reusing previously downloaded analysis data.
4. Rules: The stage runs only on the main branch to focus on production-ready code.

💡 What’s Special?

• The allow_failure: true option ensures the pipeline continues even if this stage fails, allowing flexibility in addressing non-critical code quality issues.

Step 4: Package Stage

This is where the application is containerized using Docker.

package-job:
  stage: package
  tags:
  - ec2
  - shell
  script:
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD registry.gitlab.com
  - docker build -t registry.gitlab.com/lowyiiii/python-project .
  - docker push registry.gitlab.com/lowyiiii/python-project 

What Happens Here?

1. Docker Login: Authenticates to GitLab’s container registry using environment variables for secure access.
2. Build: Creates a Docker image of the application using the Dockerfile in the project.
3. Push: Uploads the Docker image to GitLab’s container registry.

💡 Why Docker? Docker ensures that the application runs consistently across different environments, from development to production.

Step 5: Deploy Stage

The final stage deploys the Dockerized application to a Kubernetes cluster using AWS.

deploy-job:
  stage: deploy
  tags:
  - ec2
  - shell
  script:
  - aws configure set aws_access_key $AWS_ACCESS_KEY_ID
  - aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
  - aws configure set default_region $AWS_DEFAULT_REGION
  - kubectl apply -f Application.yaml 

What Happens Here?

1. AWS Credentials: Configures AWS CLI with environment variables for secure authentication.
2. Kubernetes Deployment: Uses kubectl to apply the deployment configuration (Application.yaml) to the Kubernetes cluster.

💡 Scalable Deployment: This stage ensures the application can scale dynamically based on traffic by leveraging Kubernetes’ orchestration capabilities.

Setting Up Environment Variables in GitLab

Environment variables are essential for securely managing secrets like AWS keys and Docker credentials in CI/CD pipelines. GitLab provides a simple way to define these variables, which are stored securely and injected into your pipeline jobs at runtime. This guide walks you through the process of setting up and using these variables effectively.

How to Set Up Environment Variables

Step 1: Navigate to Your Project

1. Open your GitLab project.
2. Go to Settings > CI/CD.

Step 2: Add Variables

1. Expand the Variables section and click Add Variable.
2. Fill in the following fields:

Step 3: Set Permissions

1. Environment scope: Leave it as * to apply the variable to all environments unless specific scoping is required.
2. Protect Variable: Check this box if the variable should only be available in protected branches or tags.
3. Mask Variable: Check this box if the variable contains sensitive information (e.g., passwords) to ensure it is not exposed in job logs.

Step 4: Save Changes

Click Save Variable to store it securely.

Variables Used in This Pipeline

Docker Registry Variables

• CI_REGISTRY_USER: Your GitLab username.
• CI_REGISTRY_PASSWORD: A personal access token or password for your GitLab account.

Example:

• Key: CI_REGISTRY_USER Value: your-gitlab-username
• Key: CI_REGISTRY_PASSWORD Value: your-personal-access-token

AWS Credentials

• AWS_ACCESS_KEY_ID: Your AWS IAM user’s access key ID.
• AWS_SECRET_ACCESS_KEY: Your AWS IAM user’s secret access key.
• AWS_DEFAULT_REGION: The AWS region where your resources are located (e.g., us-west-2).

Example:

• Key: AWS_ACCESS_KEY_ID Value: AKIAIOSFODNN7EXAMPLE
• Key: AWS_SECRET_ACCESS_KEY Value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
• Key: AWS_DEFAULT_REGION Value: us-west-2

SonarQube Variables (if applicable)

• SONAR_TOKEN: Your authentication token for SonarQube.

Other Variables

• GIT_DEPTH and SONAR_USER_HOME: These are set directly in the pipeline configuration and do not require manual input.

How These Variables Are Used

Docker Login

The pipeline uses CI_REGISTRY_USER and CI_REGISTRY_PASSWORD to authenticate with GitLab’s container registry, ensuring secure access to private Docker images.

AWS Configuration

• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION allow the pipeline to securely authenticate and interact with AWS services.

SonarQube Integration

• The SonarQube scanner uses SONAR_TOKEN for authentication (if required) to perform detailed code quality and security analysis.

Benefits of Using Environment Variables in GitLab

1. Security: Sensitive information is not hardcoded into the pipeline or repository, reducing the risk of leaks.
2. Reusability: Variables can be reused across multiple pipelines or projects, simplifying management.
3. Flexibility: Changing the value of a variable in the settings automatically updates all jobs that use it.

Key Benefits of This Pipeline

1. End-to-End Automation: From building to deploying, the entire workflow is automated.
2. Security First: Static code analysis (SAST) and SonarQube integration ensure code quality and security.
3. Portability: Docker containers enable consistent application behavior across environments.
4. Cloud-Native Deployment: The use of Kubernetes on AWS ensures high availability and scalability.

Final Thoughts

CI/CD pipelines like this one are essential for modern DevOps practices, enabling teams to deliver high-quality software efficiently. By integrating tools like GitLab CI/CD, Docker, and AWS, you can:

• Automate repetitive tasks,
• Improve code quality and security,
• Deploy applications reliably.