CI/CD Pipeline Optimization: 14 Techniques That Cut Build and Deploy Time (2026)

A slow CI/CD pipeline is a tax on every change. Twenty minutes per run, thirty runs a day, ten engineers: that is ten engineer-hours a day waiting, plus the context switching that costs more than the wait. Pipeline optimization is one of the few engineering investments with a return you can measure in the same week you make it.

These are the fourteen techniques we apply when we take over a pipeline as part of DevOps consulting, ordered roughly by payoff per hour of effort. Examples are for GitHub Actions and GitLab CI, with Jenkins notes where the approach differs.

First, measure

Before changing anything, get three numbers per pipeline: median wall-clock time from trigger to done, the slowest stage, and the queue time before a runner picks up the job. GitHub Actions exposes these in the workflow run summary and the API; GitLab has pipeline analytics; Jenkins has the Pipeline Stage View and the Build Time Trend plugin. Put them on a dashboard. Optimizations that do not move the median are not optimizations.

Also record the failure rate and the median time to a red result. A pipeline that fails fast on a bad commit is doing its job; one that runs 18 minutes before telling you a lint check failed is not.

1. Fail fast: order stages by cost and probability of failure

Run the cheapest, most likely to fail checks first: formatting, linting, type checks, dependency audit, unit tests. Integration tests, end-to-end tests and builds come after. In GitHub Actions use needs: so expensive jobs do not start until the cheap gate passes; in GitLab use stages. Most pipelines gain minutes here by simply reordering.

2. Cache dependencies properly

Every run that re-downloads node_modules, a Maven repository, pip wheels or Go modules is wasting one to five minutes. Use the platform cache keyed on the lockfile hash:

# GitHub Actions
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: npm-${{ runner.os }}-

GitLab uses cache: with key: files: [package-lock.json]. Jenkins has no native cache; use a persistent workspace on the agent or an artifact repository such as Nexus or Artifactory as a proxy. Cache the package manager's store, not node_modules itself, so partial hits still help.

3. Cache Docker layers and order the Dockerfile for it

Copy the dependency manifest and install dependencies before copying the source, so source changes do not invalidate the dependency layer:

COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

Use BuildKit with a registry cache (--cache-to type=registry,ref=...,mode=max and --cache-from) so layers survive across runners. Multi-stage builds keep the final image small and the build cache large. This alone routinely takes container builds from eight minutes to under two.

4. Parallelise test suites

Split tests across runners by timing data, not file count, so each shard finishes at about the same time. GitHub Actions strategy.matrix with a shard index, GitLab parallel: with CI_NODE_INDEX, and Jenkins parallel stages all support this. Tools such as Jest --shard, pytest-split, Knapsack and Playwright's built-in sharding do the timing-based split. Four shards on a 12-minute suite is a three-minute suite plus overhead.

5. Run only the tests the change can affect

Monorepos and large services benefit from affected-only test selection: Nx, Turborepo and Bazel compute what changed and run only dependent targets; Jest --changedSince, pytest-testmon and Gradle's incremental test support do it per language. Keep a nightly full run so nothing rots.

6. Right-size runners and use the fast ones

Hosted runners default to small machines. Moving a build to a larger hosted runner or a self-hosted runner with fast NVMe and more cores often halves build time for the cost of a few cents more per minute. Self-hosted runners on your own cloud (an auto-scaling group of spot instances, or Actions Runner Controller on Kubernetes) are cheaper at volume and keep caches and images local. Measure CPU and memory during the job before choosing; many builds are I/O bound and gain more from disk than cores.

7. Keep the pipeline's own tooling warm

A runner that installs Node, Terraform, kubectl, Helm and the AWS CLI on every run spends a minute before doing anything. Bake them into a runner image, or use a job container built from a base image you maintain. Pin versions so an upstream tool release does not change your pipeline overnight.

8. Build once, promote the artifact

Do not rebuild for each environment. Build once, tag the image or artifact with the commit SHA, run tests against that artifact, and promote the same artifact through staging and production. This removes duplicate builds, guarantees what you tested is what you shipped, and makes rollback a retag.

9. Trim what triggers a run

Use path filters so documentation changes do not run the full build, and so a change in one service does not test every service. Cancel superseded runs on the same branch (concurrency with cancel-in-progress in GitHub Actions, interruptible: true in GitLab) so a fixed-up commit does not queue behind its predecessor.

10. Make dependency and security scanning incremental

SBOM generation, container scanning and dependency audit are necessary and slow when run from scratch. Cache the scanner databases, scan the built image rather than the whole filesystem, and run the exhaustive scan on a schedule rather than on every push while keeping a fast policy check in the merge gate. Sign artifacts with cosign in the same job that builds them so provenance is free.

11. Speed up the deploy, not just the build

Deploy time is often the forgotten half. For Kubernetes, use rolling updates with sensible maxSurge and readiness probes that reflect real readiness; a probe that waits 60 seconds for no reason is a minute per pod per rollout. For serverless and container services, pre-warm where the platform allows. For infrastructure, run terraform plan in the merge request and apply on merge so the deploy is not blocked on a review that already happened.

12. Choose the deployment strategy that matches the risk

Blue-green and canary deployments cost more infrastructure and buy faster, safer rollouts. Feature flags decouple deploy from release entirely, which is the biggest pipeline speed-up of all: ship small changes continuously behind a flag, and the pipeline no longer has to be a gate for product decisions. Argo Rollouts, Flagger and the major flag services make this routine on Kubernetes.

13. Keep secrets and environments out of the critical path

Fetching secrets from a vault, assuming cloud roles and waiting for environment approvals add fixed seconds or minutes. Use OIDC federation (GitHub and GitLab both support it) so jobs get short-lived cloud credentials with no stored secrets and no rotation job. Put manual approvals only where a human decision is genuinely required, and make them notify the approver rather than wait to be found.

14. Treat the pipeline as code with an owner

The pipeline is a product. Version it, review changes to it, give it an owner, and put its metrics next to the application's. Delete steps nobody can explain. Most pipelines we inherit have at least one stage that has been red-and-ignored for months and one that duplicates another; removing both is a free win.

Metrics that prove the work

Track before and after:

• Median pipeline duration and p95, per pipeline.
• Queue time before a runner is assigned.
• Cache hit rate for dependency and layer caches.
• Change lead time (commit to production) and deployment frequency, the two DORA metrics most directly affected.
• Pipeline cost per run and per month, because faster is not the goal if it doubles the bill.

A realistic outcome for a typical service pipeline after techniques 1 to 8: from 15 to 25 minutes down to 4 to 7, with cost roughly flat because the minutes saved offset the larger runners. Techniques 9 to 14 improve reliability and deploy safety more than raw speed, and they are what turn a fast pipeline into continuous delivery.

Frequently asked questions

What is the fastest way to speed up a CI/CD pipeline?

Cache dependencies and Docker layers, then parallelise tests. Those three steps deliver most of the gain in most pipelines and take a day to implement.

Should we use self-hosted runners?

At low volume, hosted runners are simpler. Once monthly minutes are significant, or builds need caches, private network access or specific hardware, self-hosted runners on spot capacity are cheaper and faster. Manage them with Actions Runner Controller or GitLab's Kubernetes executor rather than hand-built VMs.

How do we stop flaky tests slowing the pipeline?

Quarantine them: tag flaky tests, run them in a non-blocking job, and fix or delete them on a schedule. Retrying the whole suite hides the problem and doubles the cost.

Does pipeline optimization help with compliance?

Yes, when done properly. Build-once-promote, artifact signing and SBOM generation in the pipeline are exactly the evidence SOC 2, ISO 27001 and supply-chain frameworks ask for. Certification work itself is handled by our sister firm PraxisQ Consulting.

Can Techtweek take over our pipelines?

Yes. Pipeline optimization and CI/CD platform engineering are part of our DevOps consulting services, alongside the cloud infrastructure the pipelines deploy to and the NOC monitoring that watches the result. Where the platform is Adobe Experience Manager, the pipeline is Cloud Manager and the rules are different — see AEM Cloud Manager CI/CD pipelines and quality gates.