The first CI/CD pipeline I ever inherited was a single Jenkins job called “build_final_v2.” It had 41 shell steps, three hardcoded passwords, and a comment at the top that said “do not touch, ask Ray.” Unfortunately, Ray had left the company eight months earlier.
At first, that job worked fine, because the team was four people pushing code twice a week. However, by the time we hit eighteen engineers, it was the most hated thing in the building. Builds took 50 minutes. As a result, nobody trusted the test results, and deploys happened on Thursday afternoons with a Slack channel full of nervous emoji.
Since then, I’ve spent most of my career building and rebuilding delivery pipelines for teams in that awkward middle stage. In other words, they’re too big for “just SSH in and pull,” yet too small to have a dedicated platform team. This guide is what I wish someone had handed me back then. It isn’t a tour of every tool on the market. Instead, it’s a working approach to setting up a CI/CD pipeline that keeps up with your team instead of falling behind it.
Why a CI/CD Pipeline Breaks When the Team Grows
Most pipelines don’t fail because of bad tooling. Rather, they fail because they were built for a team that no longer exists.
For example, at five developers, merge conflicts are rare, everyone knows what everyone else is working on, and a 20-minute build is only mildly annoying. At twenty developers, however, that same 20-minute build gets queued behind six others. Consequently, people start batching changes to avoid the wait. Batched changes then produce bigger merges, and bigger merges break more things. Before long, the pipeline is red half the day.
Here are the symptoms I look for when I walk into a new team:
- Developers say “let me just rerun it” more than once a day
- The main branch is broken for hours at a time and nobody is sure whose change did it
- Releases need a specific person to be online
- Staging looks nothing like production
- Nobody can say how long a change takes to get from commit to customers
If you recognize three or more, then the pipeline has turned into a bottleneck. Fortunately, the fixes are well understood. The hard part, though, is doing them in the right order.
Pick a Branching Model Before Your CI/CD Pipeline Tool
People always want to talk about GitHub Actions versus GitLab CI versus Jenkins first. Even so, I push that conversation back every time, because your branching strategy decides how your CI/CD pipeline behaves far more than the tool does.
For instance, if your team keeps feature branches alive for two or three weeks, no pipeline will save you. Since you’re integrating late, you’ll also find integration problems late. After all, continuous integration means integrating continuously. Martin Fowler’s long-standing article on continuous integration is still the clearest explanation of why that matters, so I still send it to new hires.
For growing teams, therefore, I recommend trunk-based development with short-lived branches and pull requests. Similarly, AWS gives the same advice in its DevOps guidance: it recommends trunk-based development paired with a pull request workflow utilizing short-lived feature branches as the most effective branching strategy when practicing DevOps. In addition, the Trunk Based Development site makes a point about scale: with 16 or more developers, a team is more productive with short-lived feature branches and CI verifying them before commits land in the trunk.
The Branching Rules I Use
In practice, my rules look like this:
- A branch should live no longer than two days. If it has to live longer, then the work is too big and needs splitting.
- Every pull request triggers the full commit stage of the pipeline.
- Main is always deployable. So if main goes red, fixing it beats every other task.
- Unfinished features ship behind feature flags, not in long-running branches.
Admittedly, that fourth rule is where most teams hesitate. Still, it’s the one that makes everything else possible. Once flags decouple deployment from release, you can merge small pieces daily without exposing half-built features to customers.
The Stages of a CI/CD Pipeline That Actually Holds Up
Every CI/CD pipeline I build today follows roughly the same shape. Of course, the tools change from client to client, but the stages don’t.
Stage 1: The Commit Stage (Target: Under 10 Minutes)
This stage runs on every push and every pull request, and its whole job is fast feedback. Specifically, it should include:
- Compilation or build
- Linting and formatting checks
- Unit tests
- A quick static analysis or dependency scan
Ten minutes is my hard ceiling. Beyond that, developers switch context, open Slack, and forget about the build until it’s been red for an hour. So if your commit stage is slower, look at parallelizing tests, caching dependencies, and moving slow integration tests to a later stage.
Stage 2: Build Once, Promote Through the CI/CD Pipeline
This is the single most important principle in pipeline design, and yet I still see it broken constantly. First, you build your artifact one time, whether that’s a container image, a JAR, or a zipped bundle. Then you promote that exact artifact through every environment.
On the other hand, if you rebuild for staging and rebuild again for production, you aren’t testing what you ship. For example, a dependency can quietly resolve to a newer version between builds, and suddenly production runs code that never touched a test. Instead, tag each artifact with the commit SHA, push it to a registry, and pull that same tag at every later stage.
Stage 3: The Acceptance Stage
Integration tests, API contract tests, and a thin layer of end-to-end tests run here, against a deployed copy of the artifact in a test environment. This stage can take longer, maybe 15 to 30 minutes, because it only runs on changes that already passed the commit stage.
Meanwhile, I keep end-to-end browser tests to a short list of critical user journeys: signup, login, checkout, whatever makes money. Teams that try to cover everything with browser tests usually end up with suites that take an hour and fail at random.
Stage 4: Staging Deployment
Next, the artifact gets deployed to an environment that mirrors production as closely as your budget allows. Here, “mirrors” means the same infrastructure code, the same configuration structure, and the same deployment mechanism. Ideally, the only differences should be scale and data.
Infrastructure as code makes this possible. When staging and production come from the same Terraform modules or Helm charts with different variable files, environment drift mostly goes away.
Stage 5: Production Deployment
Depending on your maturity and risk tolerance, this is either a manual approval gate or fully automatic. Personally, I’m comfortable starting growing teams with a one-click approval, as long as the click is the only manual part. In other words, no runbooks, no SSH, no “remember to restart the worker.” The pipeline does the deploying.
Stage 6: Post-Deployment Verification
The pipeline isn’t finished when the deploy command succeeds. Rather, it’s finished when you know the new version is healthy. To that end, smoke tests, health check endpoints, and a short watch on error rates and latency should all run automatically after a deploy. If they fail, then roll back automatically.
Choosing CI/CD Pipeline Tooling Without Regret
Now we can talk tools. My honest advice for most growing teams is to use the CI system built into your source control platform. So if you’re on GitHub, use GitHub Actions. Likewise, if you’re on GitLab, use GitLab CI. The integration benefits (status checks on pull requests, built-in secrets handling, permissions that match your repo permissions) are worth more than whatever edge a standalone tool gives you.
Granted, Jenkins still has its place, mostly in organizations with heavy compliance needs or unusual build environments. Nevertheless, I wouldn’t start a new team on it in 2026 unless something specific forced my hand. After all, the maintenance cost of plugins and controller upgrades is real, and growing teams rarely have the people to absorb it. If you’re stuck on an aging Jenkins setup, treat the move like any other legacy system modernization project.
Whatever you choose, keep these things true:
- Pipeline definitions live in the repo. YAML in version control, reviewed like any other code. No clicking around a web UI to change build steps.
- Reusable templates. Once you have more than three or four services, pull shared steps into reusable workflows or templates. Otherwise, every repo drifts into its own snowflake.
- Ephemeral runners. Build agents should be disposable. Otherwise, a runner that carries state between jobs will eventually give you a build that passes on one machine and fails on another.
Keeping CI/CD Pipeline Tests Fast and Honest
Flaky tests kill trust in a CI/CD pipeline faster than anything else. Once developers learn that a red build might mean nothing, they stop reading results and start hitting rerun. At that point, your pipeline is basically theater.
In fact, Martin Fowler said it bluntly in his piece on eradicating non-determinism in tests: left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite. I’ve watched it happen. For example, one team I worked with had a 30 percent rerun rate, and when we dug in, about a dozen tests were responsible for nearly all of it.
A Flaky Test Policy That Works
Here’s the policy I put in place now:
- Track flakiness. Most CI platforms can report tests that pass and fail on the same commit. If yours can’t, then write a small script that parses test reports.
- Quarantine fast. A test that flakes twice in a week gets moved to a quarantine suite that runs but doesn’t block merges. It also gets a ticket with an owner.
- Fix or delete within two weeks. After all, a quarantined test that nobody fixes is a test nobody needed.
- Never add automatic retries to hide the problem. Retries are a painkiller, not a cure.
Speeding Up the Test Suite
As for speed, the biggest wins usually come from three places: running tests in parallel shards, caching dependency downloads and build layers, and only running the tests affected by a change in larger monorepos. Admittedly, test impact analysis takes effort to set up. On a big codebase, however, it can cut commit stage time in half.
Securing Your CI/CD Pipeline: Secrets and Permissions
I wish more teams treated their CI/CD pipeline as production infrastructure, because attackers already do. After all, your pipeline has credentials to your cloud accounts, your container registry, and often your production databases. So if it gets compromised, everything does. That’s also why the pipeline belongs in scope the next time you commission a penetration test.
Lock Down Flow Control First
The OWASP Top 10 CI/CD Security Risks project is the best starting point I know. Notably, the first risk on that list is CICD-SEC-1: Insufficient Flow Control Mechanisms, which in plain terms means someone can push code to production without the reviews and checks you think are in place. Fortunately, branch protection, required reviews, and required status checks on main deal with most of it.
Replace Static Keys With OIDC
Second, I fix long-lived cloud credentials sitting in CI secrets on almost every engagement. For example, if your pipeline deploys to AWS, Azure, or Google Cloud with an access key that was created two years ago and never rotated, fix that first. Thankfully, GitHub Actions and most other modern CI platforms support OpenID Connect. GitHub’s security hardening documentation for OIDC explains the benefit well: the cloud provider issues a short-lived access token that is only valid for a single job, and then automatically expires. As a result, there are no static keys to leak and no rotation calendar to forget about.
Harden Every Job
In addition, a few more rules I hold to:
- Least privilege per job. The job that runs unit tests has no business holding production deploy credentials. Therefore, scope permissions to the job, not the whole workflow.
- Pin third-party actions and plugins. Reference them by commit SHA, not by a floating tag, because a compromised upstream action can steal your secrets on its next run.
- Never run untrusted code with secrets. Pull requests from forks shouldn’t get access to deployment credentials.
- Sign and verify artifacts. The SLSA framework gives you a graded path toward supply chain integrity, and even reaching the first level puts you ahead of most teams.
CI/CD Pipeline Deployment Strategies That Let You Sleep
How you deploy matters as much as how often. For growing teams, I usually recommend one of these patterns.
Choosing a Release Pattern
Rolling deployments swap instances gradually. They’re the default in Kubernetes and fine for most stateless services. The catch, however, is that old and new versions run side by side for a while, so your changes have to be backward compatible.
Blue-green deployments run two full environments and flip traffic between them. Because the old environment is still running, rollback is instant. On the downside, you pay for double infrastructure during the switch.
Canary releases send a small slice of traffic, say 5 percent, to the new version while you watch error rates. If the numbers hold, you widen it. Otherwise, you pull back. This is my favorite once a team has decent observability, since it catches the problems tests never will.
Feature flags sit on top of whichever pattern you pick. Essentially, they let you deploy code dark and release it to users separately. That’s how teams ship daily without shipping half-finished work.
Practice Rollbacks and Safe Migrations
Whatever you choose, practice rollbacks before you need one. For this reason, I schedule a deliberate rollback drill once a quarter with every team I support. The first drill always uncovers something, like a database migration that can’t be reversed or a cache that holds stale data. Clearly, it’s much better to find that on a quiet Tuesday than during an outage.
As for database migrations specifically, use the expand and contract pattern. First, add new columns or tables. Then deploy code that writes to both old and new, backfill, and switch reads. Finally, drop the old structure in a later release. It feels slower; nonetheless, it’s the only reliable way to keep deploys reversible.
Measuring CI/CD Pipeline Performance With DORA Metrics
You can’t improve a CI/CD pipeline you don’t measure, and luckily the industry has a well-tested set of measures. The DORA research program has tracked software delivery performance for over a decade. Its classic four metrics are deployment frequency, lead time for changes, change failure rate, and failed deployment recovery time. Moreover, DORA introduced a fifth metric, rework rate, in its 2024 report and published the first benchmarks for it in 2025.
What I like about these metrics is that they balance each other. For instance, speed metrics without stability metrics push teams to ship recklessly. Conversely, stability metrics without speed metrics push them to ship nothing. Watching both, then, keeps you honest.
For a sense of where the industry sits, Axify’s breakdown of the 2025 DORA report shows 16.2% of teams deploying on demand with multiple deploys per day. Also, the 2025 report moves away from rigid performance tiers like Elite, High, Medium, and Low. I think that change is healthy, because a fintech compliance platform and a marketing site shouldn’t be judged by the same yardstick. Instead, compare your team to its own numbers from last quarter.
My practical advice is to start with lead time and deployment frequency, since your CI system already holds most of that data. Simply pull commit timestamps and deploy timestamps, and chart them weekly. Later, add change failure rate once you have a consistent way of tagging incidents back to deploys. In short, you don’t need an expensive platform for the first six months. A spreadsheet and an honest weekly look are enough to show you where work piles up.
Making the CI/CD Pipeline a Team Asset
The final piece is ownership, and it has nothing to do with technology. In a growing team, the pipeline can’t depend on one person. I’ve been that one person, and frankly, it’s bad for the team and worse for whoever’s carrying it.
Here’s what I put in place:
- A named owner rotation. One engineer per sprint is responsible for pipeline health: watching flaky test reports, reviewing build times, and triaging failures. As a result, knowledge spreads and resentment doesn’t build up.
- A short README for the pipeline. What each stage does, how to run it locally, and who to ask. One page, kept current.
- Build time budgets. Agree as a team on the maximum acceptable time for each stage. Then, when a stage creeps past its budget, someone opens a ticket, just like for any other regression.
- Cost visibility. CI minutes and runner costs add up quickly as teams grow, so review spend monthly as part of your wider cloud cost optimization work. In fact, I’ve cut CI bills by a third just by fixing caching and canceling outdated runs when a newer commit lands on the same branch.
A 90-Day CI/CD Pipeline Rollout Plan for Growing Teams
If you’re starting from a messy pipeline, or from nothing, here’s the order I’d tackle things in. Note that it assumes a team of roughly 10 to 30 engineers. If you don’t have the people to drive it internally, outside DevOps services can take the first phase off your plate.
Days 1 to 30: Stabilize
First, move pipeline definitions into version control. Next, turn on branch protection and required status checks for main. Then get the commit stage under ten minutes. Also set up flaky test tracking and quarantine the worst offenders. Finally, replace static cloud credentials with OIDC.
Days 31 to 60: Standardize
Adopt build once, promote everywhere. After that, create reusable pipeline templates and move your services onto them. Make staging come from the same infrastructure code as production. Lastly, automate the production deploy so the only manual step is approval.
Days 61 to 90: Optimize and Measure
Add post-deployment verification and automatic rollback. Meanwhile, introduce feature flags for one team as a pilot. Start tracking lead time and deployment frequency weekly. To close out the quarter, run your first rollback drill.
By day 90, you won’t have a perfect CI/CD pipeline, because nobody does. Even so, you’ll have one the team trusts, one that doesn’t depend on any single person, and one you can measure and improve.
Final Thoughts
The best CI/CD pipeline I ever worked on wasn’t the fanciest. In fact, it ran on plain GitHub Actions, deployed with a short Helm chart, and had maybe a dozen end-to-end tests. What made it great, however, was that every engineer on that team understood it, trusted it, and felt responsible for it. Consequently, deploys happened twenty times a day and nobody noticed, which is exactly how it should feel.
Ultimately, that’s the goal for a growing team. It isn’t maximum automation for its own sake. Rather, it’s a delivery process so dependable it fades into the background, so your engineers can spend their energy on the product instead of the plumbing.
CI/CD Pipeline FAQs
What is a CI/CD pipeline in simple terms?
A CI/CD pipeline is an automated sequence that takes code from a developer’s commit, builds it, tests it, and deploys it to production with as little manual work as possible. Specifically, continuous integration covers the merging and testing side, while continuous delivery or deployment covers the release side. For background, Martin Fowler’s Continuous Integration article is a good foundational read.
How long should a CI/CD pipeline take to run?
Aim to keep the commit stage under ten minutes so developers get feedback before they switch tasks. Later stages, such as acceptance testing, can take longer because they only run on changes that already passed the early checks. In addition, Harness’s guide to CI/CD best practices covers more ways to speed up builds.
Which CI/CD tool is best for a growing team?
For most teams, the CI system built into your source control platform (GitHub Actions or GitLab CI) is the most practical choice, because it plugs straight into pull requests and permissions. If you’re still weighing options, GitLab’s Ultimate Guide to CI/CD is a useful comparison point.
How do I secure secrets in a CI/CD pipeline?
Avoid long-lived credentials wherever possible. Instead, use OpenID Connect to get short-lived cloud tokens, scope permissions to individual jobs, and pin third-party actions to specific commits. GitHub’s OIDC security hardening guide walks through the setup.
What metrics should I track for my CI/CD pipeline?
Start with the DORA metrics: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time, and the newer rework rate. For details, the DORA research site explains each one and how to benchmark them.
Do I need trunk-based development to do CI/CD well?
You don’t strictly need it. However, long-lived branches make continuous integration close to impossible. Therefore, short-lived branches merged within a day or two, plus feature flags for unfinished work, give you most of the benefit. The Trunk Based Development site explains how this works for teams of different sizes.
References
- Fowler, M. Continuous Integration. martinfowler.com.
- Fowler, M. Eradicating Non-Determinism in Tests. martinfowler.com.
- Amazon Web Services. DevOps Guidance: Keep Feature Branches Short-Lived. AWS Well-Architected.
- Trunk Based Development. Short-Lived Feature Branches.
- OWASP Foundation. OWASP Top 10 CI/CD Security Risks.
- GitHub Docs. Security Hardening Your Deployments With OpenID Connect.
- DORA. DevOps Research and Assessment.
- Axify. DORA Metrics: The Complete Guide.
- Harness. CI/CD Best Practices.
- GitLab. Ultimate Guide to CI/CD.
- Open Source Security Foundation. SLSA: Supply Chain Levels for Software Artifacts.
