After building CI\/CD pipelines for four different programming languages, I discovered something critical: the DevOps pattern is universal, but the technical implementation details are completely different.<\/p>\n
Every pipeline follows the same stages:<\/p>\n
But what happens INSIDE each stage? That’s where the real learning happens.<\/p>\n
This isn’t a tutorial. This is a technical deep-dive showing:<\/p>\n
All code is from my production home lab running on AWS EKS.<\/p>\n
pipeline {\n agent any\n \n options {\n buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')\n timeout(time: 30, unit: 'MINUTES')\n }\n \n tools {\n \/\/ Language-specific tool (maven\/nodejs\/python)\n }\n \n environment {\n dockercred = credentials('docker-hub')\n \/\/ AWS credentials for kubectl\n }\n \n stages {\n stage('Checkout') { \/* git operations *\/ }\n stage('Build') { \/* language-specific commands *\/ }\n stage('Test') { \/* language-specific tests *\/ }\n stage('Docker') { \/* docker build & push *\/ }\n stage('Deploy-Dev') { \/* kubectl apply -n dev *\/ }\n stage('Deploy-Staging') { \/* kubectl apply -n staging *\/ }\n stage('Deploy-Prod') { \/* manual approval + kubectl *\/ }\n }\n}<\/code><\/pre>\nWhat this structure provides:<\/p>\n
1. Build History Management:<\/p>\n
buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')<\/code><\/pre>\nTechnical behavior:<\/p>\n
Jenkins stores build artifacts in $JENKINS_HOME\/jobs\/{job-name}\/builds\/{build-number}\/<\/p>\n
Each build directory contains: build.xml – Build metadata log – Console output archive\/ – Archived artifacts workflow\/ – Pipeline state<\/p>\n
Without buildDiscarder:<\/p>\n
After 1 year:<\/p>\n
- 365 builds \u00d7 500MB average = 182GB disk space!<\/li>
- Jenkins slows down reading build history<\/li>
- Backup time increases<\/li><\/ul>\n
With logRotator:<\/p>\n
Keeps only:<\/p>\n
- Last 2 builds (regardless of age)<\/li>
- Any builds from last 7 days<\/li>
- Deletes older builds automatically<\/li>
- Typical disk usage: ~2GB for active job<\/li><\/ul>\n
2. Timeout Protection:<\/p>\n
timeout(time: 30, unit: 'MINUTES')<\/code><\/pre>\nWhy this matters:<\/p>\n
Scenario: npm install hangs (network issue)<\/p>\n
Without timeout:<\/p>\n
- Build runs forever<\/li>
- Executor blocked<\/li>
- Other builds queue indefinitely<\/li>
- Manual intervention required<\/li><\/ul>\n
With timeout:<\/p>\n
- After 30 minutes \u2192 Pipeline ABORTED<\/li>
- Executor released<\/li>
- Next build can run<\/li>
- Alert sent to team<\/li><\/ul>\n
3. Tools Configuration:<\/p>\n
The tools {} block tells Jenkins where to find build tools:<\/p>\n
tools {\n maven 'Maven' \/\/ References \"Maven\" in Global Tool Configuration\n}<\/code><\/pre>\nWhat happens behind the scenes:<\/p>\n
When pipeline starts:<\/p>\n
Jenkins reads Global Tool Configuration:<\/p>\n
Jenkins sets environment variables:<\/p>\n
Now mvn command is available:<\/p>\n
Multi-version support:<\/p>\n
If you have multiple tool versions:<\/p>\n
pipeline {\n agent any\n tools {\n maven 'Maven-3.9.0' \/\/ For new projects\n \/\/ OR\n maven 'Maven-3.6.3' \/\/ For legacy projects\n }\n}<\/code><\/pre>\nJenkins automatically uses the correct version!<\/p>\n
Language 1: Java\/Maven – Dependency Hell and Resolution<\/h3>\nComplete Pipeline Code<\/h3>\npipeline {\n agent any\n \n options {\n buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')\n }\n \n tools {\n maven 'Maven'\n }\n \n environment {\n dockercred = credentials('docker-hub')\n }\n \n stages {\n stage('CheckOut') {\n steps {\n checkout scmGit(\n branches: [[name: '*\/main']], \n userRemoteConfigs: [[url: 'https:\/\/github.com\/mohammadrezachegini\/AddressBook\/']]\n )\n }\n }\n \n stage('mvn test') {\n steps {\n sh 'mvn test'\n }\n }\n \n stage('build stage') {\n steps {\n sh 'mvn package'\n }\n }\n \n stage('Docker build and push') {\n steps {\n sh 'docker build -t lowyiiii\/jenkins-java:${BUILD_NUMBER} .'\n sh 'docker build -t lowyiiii\/jenkins-java:latest .'\n sh 'echo $dockercred_PSW | docker login -u $dockercred_USR --password-stdin'\n sh 'docker push lowyiiii\/jenkins-java:${BUILD_NUMBER}'\n sh 'docker push lowyiiii\/jenkins-java:latest'\n }\n }\n \n stage('Kubectl Deploy') {\n steps {\n sh 'kubectl apply -f jenkins-java-application.yaml'\n }\n }\n }\n}<\/code><\/pre>\nDeep Dive: Maven Dependency Resolution<\/h3>\n
What happens during mvn test and mvn package?<\/p>\n
Stage 1: Parse pom.xml<\/h3>\n<project>\n <dependencies>\n <dependency>\n <groupId>org.springframework.boot<\/groupId>\n <artifactId>spring-boot-starter-web<\/artifactId>\n <version>3.1.5<\/version>\n <\/dependency>\n <dependency>\n <groupId>junit<\/groupId>\n <artifactId>junit<\/artifactId>\n <version>4.13.2<\/version>\n <scope>test<\/scope>\n <\/dependency>\n <\/dependencies>\n<\/project><\/code><\/pre>\nMaven reads this and creates a dependency graph.<\/p>\n
Stage 2: Dependency Graph Construction<\/h3>\n
Maven builds a tree:<\/p>\n
Your Application\n\u251c\u2500\u2500 spring-boot-starter-web:3.1.5\n\u2502 \u251c\u2500\u2500 spring-boot-starter:3.1.5\n\u2502 \u2502 \u251c\u2500\u2500 spring-boot:3.1.5\n\u2502 \u2502 \u251c\u2500\u2500 spring-boot-autoconfigure:3.1.5\n\u2502 \u2502 \u2514\u2500\u2500 logback-classic:1.4.11\n\u2502 \u2502 \u251c\u2500\u2500 logback-core:1.4.11\n\u2502 \u2502 \u2514\u2500\u2500 slf4j-api:2.0.9\n\u2502 \u251c\u2500\u2500 spring-webmvc:6.0.13\n\u2502 \u2514\u2500\u2500 tomcat-embed-core:10.1.15\n\u2514\u2500\u2500 junit:4.13.2 (scope: test)\n \u2514\u2500\u2500 hamcrest-core:1.3<\/code><\/pre>\nTotal dependencies for this simple app: 47 JAR files!<\/p>\n
Stage 3: Version Conflict Resolution<\/h3>\n
Common conflict scenario:<\/p>\n
Your app needs:<\/p>\n
\u251c\u2500\u2500 Library A \u2192 requires slf4j-api:1.7.30\n\u2514\u2500\u2500 Library B \u2192 requires slf4j-api:2.0.9<\/code><\/pre>\nWhich version does Maven choose?<\/p>\n
Maven uses nearest definition strategy:<\/p>\n
Dependency Tree Depth:<\/p>\n
Your-App \u2192 Library-A \u2192 slf4j-api:1.7.30 (depth = 2)<\/p>\n
Your-App \u2192 Library-B \u2192 slf4j-api:2.0.9 (depth = 2)<\/p>\n
Same depth? Use first declaration in pom.xml!<\/p>\n
If Library A declared first \u2192 slf4j-api:1.7.30 wins If Library B declared first \u2192 slf4j-api:2.0.9 wins<\/p>\n
This can cause bugs!<\/p>\n
Library B expects 2.0.9 features \u2192 crashes at runtime with 1.7.30<\/p>\n
Solution: Dependency Management<\/p>\n
<dependencyManagement>\n <dependencies>\n <dependency>\n <groupId>org.slf4j<\/groupId>\n <artifactId>slf4j-api<\/artifactId>\n <version>2.0.9<\/version> <!-- Force this version -->\n <\/dependency>\n <\/dependencies>\n<\/dependencyManagement><\/code><\/pre>\nNow ALL transitive dependencies use 2.0.9!<\/p>\n
Stage 4: Download from Maven Central<\/h3>\n
For each dependency, Maven:<\/p>\n
1. Checks local repository:<\/p>\n
~\/.m2\/repository\/org\/springframework\/boot\/spring-boot-starter-web\/3.1.5\/<\/code><\/pre>\nIf exists \u2192 Use it \u2705 If not \u2192 Download from remote \u2b07\ufe0f<\/p>\n
2. Queries Maven Central:<\/p>\n
GET https:\/\/repo.maven.apache.org\/maven2\/org\/springframework\/boot\/spring-boot-starter-web\/3.1.5\/spring-boot-starter-web-3.1.5.pom<\/code><\/pre>\nResponse:<\/p>\n
- POM file (metadata)<\/li>
- JAR file (actual code)<\/li>
- Sources JAR (source code)<\/li>
- Javadoc JAR (documentation)<\/li>
- SHA-1 checksum<\/li>
- MD5 checksum<\/li><\/ul>\n
3. Validates checksums:<\/p>\n
# Maven downloads:\nspring-boot-starter-web-3.1.5.jar\nspring-boot-starter-web-3.1.5.jar.sha1\n\n# Calculates SHA-1 of downloaded JAR\ncalculated_hash = sha1sum spring-boot-starter-web-3.1.5.jar\n\n# Compares with downloaded .sha1 file\nexpected_hash = cat spring-boot-starter-web-3.1.5.jar.sha1\n\nif calculated_hash == expected_hash:\n Installation successful \u2705\nelse:\n ERROR: Checksum mismatch! Corrupted download! \u274c<\/code><\/pre>\n4. Stores in local repo:<\/p>\n
~\/.m2\/repository\/\n\u2514\u2500\u2500 org\/\n \u2514\u2500\u2500 springframework\/\n \u2514\u2500\u2500 boot\/\n \u2514\u2500\u2500 spring-boot-starter-web\/\n \u2514\u2500\u2500 3.1.5\/\n \u251c\u2500\u2500 spring-boot-starter-web-3.1.5.jar\n \u251c\u2500\u2500 spring-boot-starter-web-3.1.5.pom\n \u251c\u2500\u2500 spring-boot-starter-web-3.1.5-sources.jar\n \u2514\u2500\u2500 spring-boot-starter-web-3.1.5-javadoc.jar<\/code><\/pre>\nNext build? Uses cached version! (MUCH faster)<\/p>\n
Stage 5: mvn test – Test Execution<\/h3>\nmvn test<\/code><\/pre>\nWhat happens:<\/p>\n
1. Compile test code:<\/p>\n
[INFO] --- maven-compiler-plugin:3.11.0:testCompile (default-testCompile) @ addressbook ---\n[INFO] Compiling 5 source files to \/home\/jenkins\/workspace\/java-project\/target\/test-classes<\/code><\/pre>\nCreates .class files from .java test files.<\/p>\n
2. Run Surefire plugin:<\/p>\n
[INFO] --- maven-surefire-plugin:3.0.0:test (default-test) @ addressbook ---\n[INFO] \n[INFO] -------------------------------------------------------\n[INFO] T E S T S\n[INFO] -------------------------------------------------------\n[INFO] Running com.example.UserServiceTest\n[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.532 s\n[INFO] Running com.example.AddressBookTest\n[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.421 s<\/code><\/pre>\nTechnical details:<\/p>\n
Surefire creates a forked JVM:<\/p>\n
Main Maven Process (PID: 1234)\n\u2514\u2500 Forked Test JVM (PID: 5678)\n \u2514\u2500 Runs all tests in isolation<\/code><\/pre>\nWhy fork?<\/p>\n
- Tests can’t affect Maven process<\/li>
- Test failures don’t crash Maven<\/li>
- Separate memory space<\/li>
- Can set test-specific JVM args<\/li><\/ul>\n
3. Generate test reports:<\/p>\n
target\/\n\u2514\u2500\u2500 surefire-reports\/\n \u251c\u2500\u2500 TEST-com.example.UserServiceTest.xml\n \u251c\u2500\u2500 TEST-com.example.AddressBookTest.xml\n \u2514\u2500\u2500 com.example.UserServiceTest.txt<\/code><\/pre>\nJenkins parses these XML files for test results!<\/p>\n
Stage 6: mvn package – Build Application<\/h3>\nmvn package<\/code><\/pre>\nPhase execution order:<\/p>\n
1. validate \u2192 Validate project is correct\n2. compile \u2192 Compile source code\n3. test \u2192 Run unit tests\n4. package \u2192 Create JAR\/WAR file\n5. verify \u2192 Run integration tests (if configured)\n6. install \u2192 Install to local repository (if mvn install)\n7. deploy \u2192 Deploy to remote repo (if mvn deploy)<\/code><\/pre>\nmvn package executes phases 1-4.<\/p>\n
Packaging process:<\/p>\n
[INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ addressbook ---\n[INFO] Building jar: \/workspace\/target\/addressbook-1.0-SNAPSHOT.jar<\/code><\/pre>\nWhat goes into the JAR:<\/p>\n
addressbook-1.0-SNAPSHOT.jar\n\u251c\u2500\u2500 META-INF\/\n\u2502 \u251c\u2500\u2500 MANIFEST.MF (metadata: Main-Class, Class-Path, etc.)\n\u2502 \u2514\u2500\u2500 maven\/\n\u2502 \u2514\u2500\u2500 com.example\/\n\u2502 \u2514\u2500\u2500 addressbook\/\n\u2502 \u251c\u2500\u2500 pom.xml\n\u2502 \u2514\u2500\u2500 pom.properties\n\u251c\u2500\u2500 com\/\n\u2502 \u2514\u2500\u2500 example\/\n\u2502 \u251c\u2500\u2500 AddressBook.class\n\u2502 \u251c\u2500\u2500 User.class\n\u2502 \u2514\u2500\u2500 UserService.class\n\u2514\u2500\u2500 application.properties<\/code><\/pre>\nMANIFEST.MF example:<\/p>\n
Manifest-Version: 1.0\nMain-Class: com.example.AddressBook\nClass-Path: spring-boot-starter-web-3.1.5.jar logback-classic-1.4.11.jar\nBuild-Jdk: 11.0.20\nCreated-By: Apache Maven 3.9.0<\/code><\/pre>\nBuild time breakdown:<\/p>\n
Total time: 2 minutes 15 seconds<\/p>\n
Breakdown:<\/p>\n
- Download dependencies: 45s (first time only)<\/li>
- Compile main code: 12s<\/li>
- Run tests: 28s<\/li>
- Package JAR: 8s<\/li>
- Docker build: 42s<\/li><\/ul>\n
Memory usage during build:<\/p>\n
Maven process:<\/p>\n
- Heap: -Xmx512m (default)<\/li>
- Metaspace: ~150MB<\/li>
- Total: ~700MB<\/li><\/ul>\n
Test fork JVM:<\/p>\n
- Heap: -Xmx256m (default)<\/li>
- Total: ~350MB<\/li><\/ul>\n
Peak memory: ~1GB<\/p>\n
Deep Dive: Docker Multi-Tag Strategy<\/h3>\nsh 'docker build -t lowyiiii\/jenkins-java:${BUILD_NUMBER} .'\nsh 'docker build -t lowyiiii\/jenkins-java:latest .'<\/code><\/pre>\nWhy build twice? That’s inefficient!<\/p>\n
Better approach – multi-tag in single build:<\/p>\n
sh 'docker build -t lowyiiii\/jenkins-java:${BUILD_NUMBER} -t lowyiiii\/jenkins-java:latest .'<\/code><\/pre>\nWhat this does:<\/p>\n
Single build creates TWO tags pointing to SAME image:<\/p>\n
Image ID: sha256:abc123...\n\u251c\u2500\u2500 lowyiiii\/jenkins-java:47\n\u2514\u2500\u2500 lowyiiii\/jenkins-java:latest<\/code><\/pre>\nDisk usage: 450MB (not 900MB!)<\/p>\n
Build time: 42s (not 84s!)<\/p>\n
Docker push behavior:<\/p>\n
sh 'docker push lowyiiii\/jenkins-java:${BUILD_NUMBER}'\nsh 'docker push lowyiiii\/jenkins-java:latest'<\/code><\/pre>\nWhat actually transfers:<\/p>\n
Push #1 (lowyiiii\/jenkins-java:47):<\/p>\n
- Uploads all layers (450MB)<\/li>
- Time: ~3 minutes on 100Mbps<\/li><\/ul>\n
Push #2 (lowyiiii\/jenkins-java:latest):<\/p>\n
- Detects layers already exist in registry<\/li>
- Only updates tag pointer<\/li>
- Transfers: ~2KB (metadata only!)<\/li>
- Time: ~2 seconds<\/li><\/ul>\n
Registry storage:<\/p>\n
Docker Hub storage:<\/p>\n
Docker Hub storage:\nlowyiiii\/jenkins-java\n\u251c\u2500\u2500 47: sha256:abc123... (450MB)\n\u251c\u2500\u2500 48: sha256:def456... (450MB)\n\u2514\u2500\u2500 latest \u2192 48 (0MB, just a pointer)<\/code><\/pre>\nTotal: 900MB (not 1.35GB)<\/p>\n
Why two tags?<\/p>\n
:${BUILD_NUMBER} (47, 48, 49…)<\/p>\n
- Immutable<\/li>
- Allows rollback<\/li>
- Audit trail<\/li>
- “What version is in production?”<\/li><\/ul>\n
:latest<\/p>\n
- Always points to newest<\/li>
- Convenient for dev environment<\/li>
- “Give me the latest build”<\/li><\/ul>\n
Real-world usage:<\/p>\n
# Development\napiVersion: apps\/v1\nkind: Deployment\nspec:\n template:\n spec:\n containers:\n - image: lowyiiii\/jenkins-java:latest # Auto-updates\n\n# Production\napiVersion: apps\/v1\nkind: Deployment\nspec:\n template:\n spec:\n containers:\n - image: lowyiiii\/jenkins-java:47 # Pinned version<\/code><\/pre>\nLanguage 2: Node.js – npm Dependency Algorithm<\/h3>\nComplete Pipeline Code<\/h3>\npipeline {\n agent any\n \n options {\n buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')\n }\n \n tools {\n nodejs 'NodeJs'\n }\n \n environment {\n dockercred = credentials('docker-hub')\n }\n \n stages {\n stage('CheckOut') {\n steps {\n checkout scmGit(\n branches: [[name: '*\/main']], \n userRemoteConfigs: [[url: 'https:\/\/github.com\/mohammadrezachegini\/Node-JS-Jenkins\/']]\n )\n }\n }\n \n stage('node stage') {\n steps {\n sh 'node -v'\n sh 'npm -v'\n sh 'npm ci'\n }\n }\n \n stage('Docker build') {\n steps {\n sh 'docker build -t lowyiiii\/jenkins-nodejs:latest .'\n sh 'echo $dockercred_PSW | docker login -u $dockercred_USR --password-stdin'\n sh 'docker push lowyiiii\/jenkins-nodejs:latest'\n }\n }\n \n stage('kubectl stage') {\n steps {\n sh 'kubectl apply -f Application.yaml'\n }\n }\n }\n}<\/code><\/pre>\nDeep Dive: npm ci vs npm install<\/h3>\n
Why use npm ci instead of npm install?<\/p>\n
Let me show you the technical differences:<\/p>\n
npm install – The Developer Command<\/h3>\nnpm install<\/code><\/pre>\nAlgorithm:<\/p>\n
- Read package.json<\/li>
- Read package-lock.json (if exists)<\/li>
- Compare versions:<\/li>
- Download packages<\/li>
- Update package-lock.json if needed<\/li>
- Install to node_modules\/<\/li><\/ol>\n
- If lock file has compatible version \u2192 use it<\/li>
- If lock file outdated \u2192 resolve new version<\/li>
- If lock file missing \u2192 create it<\/li><\/ul>\n
Example scenario:<\/p>\n
\/\/ package.json\n{\n \"dependencies\": {\n \"express\": \"^4.18.0\"\n }\n}\n\n\/\/ package-lock.json (3 months old)\n{\n \"dependencies\": {\n \"express\": {\n \"version\": \"4.18.2\"\n }\n }\n}<\/code><\/pre>\nToday, express@4.19.5 is available:<\/p>\n
npm install<\/code><\/pre>\nResult:<\/p>\n
- Sees “^4.18.0” allows 4.19.5<\/li>
- Updates to express@4.19.5<\/li>
- Modifies package-lock.json<\/li>
- \u26a0\ufe0f Different version than last build!<\/li><\/ul>\n
npm ci – The CI\/CD Command<\/h3>\nnpm ci<\/code><\/pre>\nAlgorithm:<\/p>\n
- Check package-lock.json exists<\/li>
- If missing \u2192 ERROR and exit \u274c<\/li>
- Delete node_modules\/ directory<\/li>
- Read package-lock.json ONLY (ignores package.json versions)<\/li>
- Download EXACT versions from lock file<\/li>
- Verify integrity checksums<\/li>
- Install to node_modules\/<\/li>
- NEVER modify package-lock.json<\/li><\/ol>\n
Same scenario:<\/p>\n
npm ci<\/code><\/pre>\nResult:<\/p>\n
- Reads package-lock.json<\/li>
- Installs express@4.18.2 (EXACT version)<\/li>
- Doesn’t check npm registry for newer versions<\/li>
- \u2705 Same version every build!<\/li><\/ul>\n
Performance comparison:<\/p>\n
Test environment: 500 packages, SSD, 100Mbps internet<\/p>\n
npm install (with node_modules\/ present):<\/p>\n
- Checks each package<\/li>
- Compares versions<\/li>
- Updates if needed<\/li>
- Time: 45 seconds<\/li><\/ul>\n
npm install (cold, no node_modules\/):<\/p>\n
- Resolves dependencies<\/li>
- Downloads packages<\/li>
- Creates node_modules\/<\/li>
- Time: 62 seconds<\/li><\/ul>\n
npm ci (always cold):<\/p>\n
- Deletes node_modules\/<\/li>
- No version resolution<\/li>
- Direct download<\/li>
- Time: 38 seconds<\/li><\/ul>\n
Winner: npm ci (40% faster!) \u2705<\/p>\n
Why npm ci is faster:<\/p>\n
npm install workflow:<\/p>\n
package.json \u2192 version resolution \u2192 package-lock.json \u2192 download \u2192 install\n(many network round-trips for version checking)<\/code><\/pre>\nnpm ci workflow:<\/p>\n
package-lock.json \u2192 download \u2192 install\n(direct downloads, no version queries)<\/code><\/pre>\nDeep Dive: npm Dependency Tree Algorithm<\/h3>\n
Example package.json:<\/p>\n
{\n \"dependencies\": {\n \"express\": \"^4.18.2\",\n \"lodash\": \"^4.17.21\"\n }\n}<\/code><\/pre>\nWhat npm actually installs:<\/p>\n
npm ci<\/code><\/pre>\nDownloads:<\/p>\n
- express@4.18.2<\/li>
- lodash@4.17.21<\/li><\/ol>\n
But express@4.18.2 depends on:<\/p>\n
- accepts@1.3.8<\/li>
- array-flatten@1.1.1<\/li>
- body-parser@1.20.1<\/li>
- content-disposition@0.5.4<\/li>
- cookie@0.5.0<\/li>
- cookie-signature@1.0.6<\/li>
- debug@2.6.9<\/li>
- depd@2.0.0<\/li><\/ol>\n
… (35 total dependencies)<\/p>\n
Total packages installed: 502!<\/p>\n
Tree Construction Algorithm<\/h3>\n
npm uses hoisting to flatten the dependency tree:<\/p>\n
Ideal scenario (no conflicts):<\/p>\n
node_modules\/\n\u251c\u2500\u2500 express\/\n\u251c\u2500\u2500 lodash\/\n\u251c\u2500\u2500 accepts\/ (hoisted from express)\n\u251c\u2500\u2500 body-parser\/ (hoisted from express)\n\u2514\u2500\u2500 debug\/ (hoisted from express and body-parser)<\/code><\/pre>\nAll shared at top level!<\/p>\n
Conflict scenario:<\/p>\n
Your app needs:<\/p>\n
\u251c\u2500\u2500 express \u2192 requires debug@2.6.9\n\u2514\u2500\u2500 some-package \u2192 requires debug@4.3.4<\/code><\/pre>\nnpm’s resolution:<\/p>\n
node_modules\/\n\u251c\u2500\u2500 debug@4.3.4 (newer version hoisted to top)\n\u251c\u2500\u2500 express\/\n\u2502 \u2514\u2500\u2500 node_modules\/\n\u2502 \u2514\u2500\u2500 debug@2.6.9 (nested due to conflict)\n\u2514\u2500\u2500 some-package\/ (uses top-level debug@4.3.4)<\/code><\/pre>\nWhy this matters:<\/p>\n
Disk usage:<\/p>\n
- Hoisted (no conflicts): 150MB<\/li>
- Nested (many conflicts): 220MB<\/li><\/ul>\n
Require time:<\/p>\n
- Hoisted: 0.5ms to find debug<\/li>
- Nested: 2.3ms (searches up directory tree)<\/li><\/ul>\n
Integrity Verification<\/h3>\n
package-lock.json includes SHA-512 checksums:<\/p>\n
{\n \"dependencies\": {\n \"express\": {\n \"version\": \"4.18.2\",\n \"resolved\": \"https:\/\/registry.npmjs.org\/express\/-\/express-4.18.2.tgz\",\n \"integrity\": \"sha512-5\/PsL6iGPdfQ\/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ==\"\n }\n }\n}<\/code><\/pre>\nInstallation process:<\/p>\n
- Download express-4.18.2.tgz from npm registry<\/li>
- Calculate SHA-512 hash:<\/li><\/ol>\n
calculated = sha512(express-4.18.2.tgz)\nexpected = \"sha512-5\/PsL6iGPdfQ\/lKM1UuielYgv3BUoJfz1aUwU9vHZ...\"<\/code><\/pre>\nCompare:<\/p>\n
if calculated == expected:\n Extract to node_modules\/express\/\nelse:\n ERROR: Integrity check failed!\n Possible reasons:\n - Corrupted download\n - Man-in-the-middle attack\n - Compromised npm registry<\/code><\/pre>\nThis prevents supply chain attacks!<\/p>\n
Real incident (2018):<\/p>\n
Attacker compromised “event-stream” package<\/p>\n
Injected malicious code to steal Bitcoin wallet keys<\/p>\n
Downloaded 8 million times before detection<\/p>\n
With integrity checks:<\/p>\n
- Hash wouldn’t match<\/li>
- npm ci would fail<\/li>
- Attack prevented \u2705<\/li><\/ul>\n
Deep Dive: Semantic Versioning in npm<\/h3>\n
Version specifiers in package.json:<\/p>\n
{\n \"dependencies\": {\n \"express\": \"4.18.2\", \/\/ Exact\n \"lodash\": \"^4.17.21\", \/\/ Caret\n \"async\": \"~3.2.4\", \/\/ Tilde\n \"react\": \"*\", \/\/ Wildcard (dangerous!)\n \"vue\": \">=3.0.0 <4.0.0\", \/\/ Range\n \"axios\": \"latest\" \/\/ Latest (very dangerous!)\n }\n}<\/code><\/pre>\nTechnical meaning:<\/p>\n
1. Exact version: 4.18.2<\/p>\n
Allows: ONLY 4.18.2<\/p>\n
Blocks: 4.18.3, 4.19.0, 5.0.0<\/p>\n
2. Caret: ^4.18.2<\/p>\n
Version format: MAJOR.MINOR.PATCH\n ^ ^ ^\nAllows: 4 any any<\/code><\/pre>\nExamples:<\/p>\n
- \u2705 4.18.2 (exact)<\/li>
- \u2705 4.18.3 (patch update – bug fixes)<\/li>
- \u2705 4.19.0 (minor update – new features, backward compatible)<\/li>
- \u2705 4.20.5 (minor + patch)<\/li>
- \u274c 5.0.0 (major update – breaking changes)<\/li><\/ul>\n
3. Tilde: ~3.2.4<\/p>\n
Allows: 3.2.x only<\/p>\n
Examples:<\/p>\n
- \u2705 3.2.4 (exact)<\/li>
- \u2705 3.2.5 (patch update)<\/li>
- \u2705 3.2.99 (many patch updates)<\/li>
- \u274c 3.3.0 (minor update blocked)<\/li>
- \u274c 4.0.0 (major update blocked)<\/li><\/ul>\n
4. Range: >=3.0.0 <4.0.0<\/p>\n
Allows: 3.0.0 up to (but not including) 4.0.0<\/p>\n
Examples:<\/p>\n
- \u2705 3.0.0<\/li>
- \u2705 3.5.2<\/li>
- \u2705 3.99.99<\/li>
- \u274c 4.0.0<\/li><\/ul>\n
Real-world impact:<\/p>\n
# Scenario: You develop with express@4.18.2\n# package.json has: \"express\": \"^4.18.2\"\n\nDeveloper A (Jan 2024):\nnpm install \u2192 Gets express@4.18.2\n\nDeveloper B (Jun 2024):\nnpm install \u2192 Gets express@4.19.8\n\nProduction (Aug 2024):\nnpm install \u2192 Gets express@4.20.1\n\n# Three different versions in three environments!\n# Potential bugs only appear in some environments!<\/code><\/pre>\nSolution: package-lock.json locks the version!<\/p>\n
Language 3: Python – pip and Virtual Environments<\/h3>\nComplete Pipeline Code<\/h3>\npipeline {\n agent any\n \n options {\n buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')\n }\n \n environment {\n dockercred = credentials('docker-hub')\n }\n \n stages {\n stage('CheckOut') {\n steps {\n checkout scmGit(\n branches: [[name: '*\/main']], \n userRemoteConfigs: [[url: 'https:\/\/github.com\/mohammadrezachegini\/python-jenkins\/']]\n )\n }\n }\n \n stage('build stage') {\n steps {\n sh 'python3 --version'\n sh 'pip3 install -r requirements.txt'\n }\n }\n \n stage('Docker build and push') {\n steps {\n sh 'docker build -t lowyiiii\/jenkins-python:latest .'\n sh 'echo $dockercred_PSW | docker login -u $dockercred_USR --password-stdin'\n sh 'docker push lowyiiii\/jenkins-python:latest'\n }\n }\n \n stage('Kubectl deployment') {\n steps {\n sh 'kubectl apply -f python-deployment.yaml'\n }\n }\n }\n}<\/code><\/pre>\nDeep Dive: pip install Technical Details<\/h3>\n
Example requirements.txt:<\/p>\n
flask==2.3.2\ngunicorn>=21.0.0,<22.0.0\npsycopg2-binary~=2.9.6\nrequests\nredis[hiredis]>=4.5.0<\/code><\/pre>\nVersion Specifier Semantics<\/h3>\n
1. Exact version: flask==2.3.2<\/p>\n
Allows: ONLY 2.3.2<\/p>\n
Install command:<\/p>\n
pip install flask==2.3.2<\/code><\/pre>\n2. Compatible release: psycopg2-binary~=2.9.6<\/p>\n
Equivalent to: >=2.9.6, ==2.9.*<\/p>\n
Allows:<\/p>\n
- \u2705 2.9.6<\/li>
- \u2705 2.9.7<\/li>
- \u2705 2.9.99<\/li>
- \u274c 2.10.0 (different minor version)<\/li><\/ul>\n
3. Version range: gunicorn>=21.0.0,<22.0.0<\/p>\n
Allows: 21.x.x only<\/p>\n
- \u2705 21.0.0<\/li>
- \u2705 21.2.5<\/li>
- \u274c 22.0.0<\/li><\/ul>\n
4. No version specified: requests<\/p>\n
Installs: Latest stable version<\/p>\n
\u26a0\ufe0f Dangerous! Version changes between builds!<\/p>\n
Better practice:<\/p>\n
requests==2.31.0<\/code><\/pre>\n5. Extras syntax: redis[hiredis]>=4.5.0<\/p>\n
Installs redis AND optional hiredis extension<\/p>\n
redis[hiredis] is equivalent to:<\/p>\n
- redis>=4.5.0<\/li>
- hiredis>=2.0.0<\/li><\/ul>\n
hiredis is a C extension for faster Redis parsing<\/p>\n
Installation Process – Step by Step<\/h3>\npip3 install -r requirements.txt<\/code><\/pre>\nStep 1: Parse requirements.txt<\/p>\n
# pip reads line by line:\npackages = [\n ('flask', '==2.3.2'),\n ('gunicorn', '>=21.0.0,<22.0.0'),\n ('psycopg2-binary', '~=2.9.6'),\n ('requests', None),\n ('redis[hiredis]', '>=4.5.0')\n]<\/code><\/pre>\nStep 2: Resolve dependencies<\/p>\n
For each package, pip queries PyPI API:<\/p>\n
GET https:\/\/pypi.org\/pypi\/flask\/json\n\nResponse:\n{\n \"info\": {\n \"version\": \"2.3.2\",\n \"requires_dist\": [\n \"Werkzeug>=2.3.3\",\n \"Jinja2>=3.1.2\",\n \"itsdangerous>=2.1.2\",\n \"click>=8.1.3\"\n ]\n },\n \"releases\": {\n \"2.3.2\": [...]\n }\n}<\/code><\/pre>\nDependency tree construction:<\/p>\n
flask==2.3.2\n\u251c\u2500\u2500 Werkzeug>=2.3.3 \u2192 2.3.6 (latest matching)\n\u2502 \u2514\u2500\u2500 MarkupSafe>=2.1.1 \u2192 2.1.3\n\u251c\u2500\u2500 Jinja2>=3.1.2 \u2192 3.1.2\n\u2502 \u2514\u2500\u2500 MarkupSafe>=2.0 \u2192 2.1.3 (already resolved)\n\u251c\u2500\u2500 itsdangerous>=2.1.2 \u2192 2.1.2\n\u2514\u2500\u2500 click>=8.1.3 \u2192 8.1.3<\/code><\/pre>\nMarkupSafe appears twice but installed once!<\/p>\n
Step 3: Download packages<\/p>\n
pip downloads either:<\/p>\n
Wheel (preferred):<\/p>\n
flask-2.3.2-py3-none-any.whl\n\u2502\n\u251c\u2500\u2500 Filename format:\n\u2502 {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl\n\u2502 \n\u251c\u2500\u2500 flask-2.3.2-py3-none-any.whl breakdown:\n\u2502 flask: package name\n\u2502 2.3.2: version\n\u2502 py3: Python 3.x\n\u2502 none: No specific ABI\n\u2502 any: Any platform\n\u2502\n\u2514\u2500\u2500 Advantages:\n - Pre-built binary\n - Fast installation (~1 second)\n - No compilation required<\/code><\/pre>\nSource distribution (fallback):<\/p>\n
flask-2.3.2.tar.gz\n\u2502\n\u251c\u2500\u2500 Contains:\n\u2502 - setup.py\n\u2502 - Source code\n\u2502 - README, LICENSE\n\u2502\n\u2514\u2500\u2500 Disadvantages:\n - Must be built\n - Slower (~15 seconds)\n - Requires build tools (gcc, etc.)<\/code><\/pre>\nStep 4: Installation<\/p>\n
# For wheel:\nunzip flask-2.3.2-py3-none-any.whl -d \/usr\/local\/lib\/python3.11\/site-packages\/\n\n# For source:\npython setup.py build\npython setup.py install<\/code><\/pre>\nInstalled location:<\/p>\n
\/usr\/local\/lib\/python3.11\/site-packages\/\n\u251c\u2500\u2500 flask\/\n\u2502 \u251c\u2500\u2500 __init__.py\n\u2502 \u251c\u2500\u2500 app.py\n\u2502 \u251c\u2500\u2500 cli.py\n\u2502 \u2514\u2500\u2500 ...\n\u251c\u2500\u2500 flask-2.3.2.dist-info\/\n\u2502 \u251c\u2500\u2500 METADATA\n\u2502 \u251c\u2500\u2500 RECORD\n\u2502 \u251c\u2500\u2500 WHEEL\n\u2502 \u2514\u2500\u2500 top_level.txt\n\u2514\u2500\u2500 bin\/\n \u2514\u2500\u2500 flask (executable)<\/code><\/pre>\nStep 5: Record installation<\/p>\n
pip creates a RECORD file:<\/p>\n
flask-2.3.2.dist-info\/RECORD:\n\nflask\/__init__.py,sha256=xyz123...,5123\nflask\/app.py,sha256=abc456...,12456\nflask\/cli.py,sha256=def789...,3421\n...<\/code><\/pre>\nThis allows:<\/p>\n
- pip uninstall flask (knows what to delete)<\/li>
- pip check (verify file integrity)<\/li><\/ul>\n
Binary Extensions: The Compilation Problem<\/h3>\n
Some packages contain C\/C++ code:<\/p>\n
# requirements.txt\npsycopg2==2.9.6 # PostgreSQL adapter (has C code)<\/code><\/pre>\nInstallation without wheel:<\/p>\n
pip install psycopg2==2.9.6\n\nCollecting psycopg2==2.9.6\n Downloading psycopg2-2.9.6.tar.gz (384 kB)\n\nBuilding wheels for collected packages: psycopg2\n Building wheel for psycopg2 (setup.py) ...\n\n# Compilation starts:\ngcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall ...\n \n# Compiling C source:\npsycopgmodule.c\nconnection_type.c\ncursor_type.c\n...\n\n# Linking:\ncreating build\/lib.linux-x86_64-cpython-311\/psycopg2\ngcc -pthread -shared ... -lpq -o build\/lib.linux-x86_64-cpython-311\/psycopg2\/_psycopg.cpython-311-x86_64-linux-gnu.so\n\n# Takes 30-60 seconds!<\/code><\/pre>\nRequirements for compilation:<\/p>\n
# Need these installed:\n- gcc (C compiler)\n- python3-dev (Python headers)\n- libpq-dev (PostgreSQL development files)\n\n# If missing:\nerror: command 'gcc' failed: No such file or directory<\/code><\/pre>\nSolution: Use binary packages<\/p>\n
# requirements.txt\npsycopg2-binary==2.9.6 # Pre-compiled!<\/code><\/pre>\npip install psycopg2-binary==2.9.6\n\nCollecting psycopg2-binary==2.9.6\n Downloading psycopg2_binary-2.9.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n\nInstalling collected packages: psycopg2-binary\nSuccessfully installed psycopg2-binary-2.9.6\n\n# Installation: 2 seconds! \u2705<\/code><\/pre>\nBinary wheel contents:<\/p>\n
psycopg2_binary-2.9.6-cp311-cp311-manylinux_2_17_x86_64.whl\n\u2502\n\u2514\u2500\u2500 Contains:\n - _psycopg.cpython-311-x86_64-linux-gnu.so (compiled C extension)\n - libpq.so.5 (bundled PostgreSQL library)\n - All Python files\n \n Ready to use immediately!<\/code><\/pre>\nCaching for Speed<\/h3>\n
pip cache directory:<\/p>\n
~\/.cache\/pip\/\n\u251c\u2500\u2500 http\/ (cached downloads)\n\u2514\u2500\u2500 wheels\/ (built wheels)<\/code><\/pre>\nFirst install:<\/p>\n
pip install requests==2.31.0\n\nCollecting requests==2.31.0\n Downloading requests-2.31.0-py3-none-any.whl (62 kB)\nInstalling...\nDone. (3.2 seconds)\n\nCache saved:\n~\/.cache\/pip\/http\/xyz\/requests-2.31.0-py3-none-any.whl<\/code><\/pre>\nSecond install (same version):<\/p>\n
pip install requests==2.31.0\n\nCollecting requests==2.31.0\n Using cached requests-2.31.0-py3-none-any.whl (62 kB)\nInstalling...\nDone. (0.4 seconds)\n\n# 8x faster! \u2705<\/code><\/pre>\nIn Jenkins, cache persists between builds if workspace preserved!<\/p>\n
Why Virtual Environments Matter<\/h3>\n
Without virtual environment:<\/p>\n
# System Python 3.11.4\n\npip install flask==2.3.2\n# Installs to \/usr\/local\/lib\/python3.11\/site-packages\/\n\npip install flask==3.0.0 # Newer project needs different version\n# OVERWRITES flask 2.3.2!\n\n# Old project breaks! \u274c<\/code><\/pre>\nWith virtual environment:<\/p>\n
# Project 1\npython3 -m venv project1-env\nsource project1-env\/bin\/activate\npip install flask==2.3.2\n# Installs to project1-env\/lib\/python3.11\/site-packages\/\n\n# Project 2\npython3 -m venv project2-env\nsource project2-env\/bin\/activate\npip install flask==3.0.0\n# Installs to project2-env\/lib\/python3.11\/site-packages\/\n\n# Both projects work! \u2705<\/code><\/pre>\nFor Jenkins (Docker handles isolation):<\/p>\n
In my pipeline, I use Docker so virtual environment is optional:<\/p>\n
FROM python:3.11-slim\n\nWORKDIR \/app\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt<\/code><\/pre>\nEach Docker container is already isolated!<\/p>\n
Language 4: PHP\/Composer – Autoloading Magic<\/h3>\nComplete Pipeline Code<\/h3>\npipeline {\n agent any\n \n options {\n buildDiscarder logRotator(daysToKeepStr: '7', numToKeepStr: '2')\n }\n \n environment {\n dockercred = credentials('docker-hub')\n }\n \n stages {\n stage('CheckOut') {\n steps {\n checkout scmGit(\n branches: [[name: '*\/main']], \n userRemoteConfigs: [[url: 'https:\/\/github.com\/mohammadrezachegini\/PHP-jenkins\/']]\n )\n }\n }\n \n stage('build stage') {\n steps {\n sh 'php -v'\n sh 'composer install'\n }\n }\n \n stage('Docker build and push') {\n steps {\n sh 'docker build -t lowyiiii\/jenkins-php:latest .'\n sh 'echo $dockercred_PSW | docker login -u $dockercred_USR --password-stdin'\n sh 'docker push lowyiiii\/jenkins-php:latest'\n }\n }\n \n stage('kubectl deployment') {\n steps {\n sh 'kubectl apply -f deploymen.yaml'\n }\n }\n }\n}<\/code><\/pre>\nDeep Dive: Composer Dependency Resolution<\/h3>\n
Example composer.json:<\/p>\n
{\n \"require\": {\n \"symfony\/http-foundation\": \"^6.0\",\n \"guzzlehttp\/guzzle\": \"^7.5\",\n \"monolog\/monolog\": \"~3.0\"\n },\n \"require-dev\": {\n \"phpunit\/phpunit\": \"^10.0\"\n }\n}<\/code><\/pre>\ncomposer install Process<\/h3>\ncomposer install<\/code><\/pre>\nStep 1: Parse composer.json and composer.lock<\/p>\n
If composer.lock exists:<\/p>\n
Use exact versions from lock file \u2705<\/p>\n
Else:<\/p>\n
Resolve versions from composer.json<\/p>\n
Create composer.lock<\/p>\n
Step 2: SAT Solver Dependency Resolution<\/p>\n
Composer uses Boolean satisfiability (SAT) solver:<\/p>\n
Problem: Find package versions satisfying ALL constraints<\/p>\n
Constraints:<\/p>\n
- symfony\/http-foundation: ^6.0<\/li>
- symfony\/http-foundation requires symfony\/polyfill-mbstring: ~1.1<\/li>
- symfony\/polyfill-mbstring requires php: >=7.1<\/li>
- guzzlehttp\/guzzle: ^7.5<\/li>
- guzzlehttp\/guzzle requires guzzlehttp\/psr7: ^2.0<\/li>
- guzzlehttp\/psr7 requires psr\/http-message: ^1.0 || ^2.0<\/li>
- …<\/li><\/ol>\n
SAT solver tries combinations until ALL constraints satisfied:<\/p>\n
Solution found:<\/p>\n
- symfony\/http-foundation: 6.0.8<\/li>
- symfony\/polyfill-mbstring: 1.27.0<\/li>
- guzzlehttp\/guzzle: 7.5.0<\/li>
- guzzlehttp\/psr7: 2.4.5<\/li>
- psr\/http-message: 2.0<\/li><\/ul>\n
This is NP-complete problem! Can take 30+ seconds for complex apps.<\/p>\n
Step 3: Download packages<\/p>\n
For each package, Composer:<\/p>\n
1. Queries Packagist.org:<\/p>\n
GET https:\/\/repo.packagist.org\/p2\/symfony\/http-foundation.json\n\nResponse:\n{\n \"packages\": {\n \"symfony\/http-foundation\": [\n {\n \"version\": \"v6.0.8\",\n \"dist\": {\n \"type\": \"zip\",\n \"url\": \"https:\/\/api.github.com\/repos\/symfony\/http-foundation\/zipball\/c9c86b...\",\n \"reference\": \"c9c86b7b9313ee7d8dd9...\"\n },\n \"source\": {\n \"type\": \"git\",\n \"url\": \"https:\/\/github.com\/symfony\/http-foundation.git\",\n \"reference\": \"c9c86b7b9313ee7d8dd9...\"\n }\n }\n ]\n }\n}<\/code><\/pre>\n2. Downloads ZIP or clones Git:<\/p>\n
# Composer prefers dist (ZIP) for speed:\nDownloading symfony\/http-foundation (v6.0.8)\n - Installing symfony\/http-foundation (v6.0.8): Extracting archive\n\n# ZIP downloaded from GitHub API\n# Extracted to vendor\/symfony\/http-foundation\/<\/code><\/pre>\n3. Verifies Git reference:<\/p>\n
# composer.lock stores Git commit hash:\n{\n \"packages\": [\n {\n \"name\": \"symfony\/http-foundation\",\n \"version\": \"v6.0.8\",\n \"source\": {\n \"reference\": \"c9c86b7b9313ee7d8dd9...\" \u2190 Git commit SHA\n }\n }\n ]\n}\n\n# After download, Composer verifies:\nextracted_git_hash = git rev-parse HEAD\nexpected_hash = \"c9c86b7b9313ee7d8dd9...\"\n\nif extracted_git_hash == expected_hash:\n \u2705 Installation valid\nelse:\n \u274c ERROR: Package corrupted or tampered!<\/code><\/pre>\n