#continuous delivery

[nofatclips]

The Engineering Room Ep. 9: Allen Holub on the pile of utterly worthless crap that Agile has become.

It's refreshing to see that not everybody has lost their freaking mind about this topic. Probably time to update my resume...

[inercia]

Smooth Sailing with Kubernetes

A comic strip about Kubernetes.

[agileanddevops-blog]

Implementing CICD with L&T Team Jan 2018 | scmGalaxy

Implementing CICD with L&T Team Jan 2018

[scmgalaxy]

Implementing CICD with L&T Team Jan 2018 | scmGalaxy

Implementing CICD with L&T Team Jan 2018

[screwdriver-cd]

Monorepo Support Using Source Paths

For those using monorepos or that only want Screwdriver jobs to run on changes in certain subdirectories or files, you can use the new source paths feature.

Source paths can be used to specify source code paths that will trigger a job upon modification.

You can either specify subdirectories and/or specific files as source paths. To denote a subdirectory, leave a trailing slash (/) at the end. The path is relative to the root of the repository.

Example

Given a repository with the file structure depicted below:

┌── README.md ├── screwdriver.yaml ├── test/ │ └── ... ├── src/ │ ├── app/ │ │ ├── main.js │ │ ├── ... │ │ └── package.json │ └── other/ │ └── ... │ ...

And the screwdriver.yaml:

jobs: main: image: node:6 requires: [~pr, ~commit] sourcePaths: ["src/app/", "screwdriver.yaml"] steps: - echo: echo hi

In this example, the job main will be triggered if there are any changes to files under src/app/ or the screwdriver.yaml file (like on src/app/main.js, src/app/package.json, etc.). The main job will not, however, be triggered on changes to README.md, test/, or src/other/.

Caveats

This feature is only available for the Github SCM right now.

sourcePaths will be ignored if you manually start a pipeline or restart a job.

The screwdriver.yaml must still be located at root.

Compatibility List

Source paths requires the following minimum versions of Screwdriver:

API: v0.5.278

UI: v1.0.252

Store: v2.4.3

Launcher: v4.0.80

Contributors

Thanks to the following people who made this feature possible:

d2lam

tkyi

Screwdriver is an open-source build automation platform designed for Continuous Delivery. It is built (and used) by Yahoo. Don’t hesitate to reach out if you have questions or would like to contribute: http://docs.screwdriver.cd/about/support.

[yahooeng]

Speed and Stability: Yahoo Mail’s Forward-Thinking Continuous Integration and Delivery Pipeline

By Mohit Goenka, Senior Engineering Manager

Building the technology powering the best consumer email inbox in the world is no easy task. When you start on such a journey, it is important to consider how to deliver such an experience to the users. After all, any consumer feature we build can only make a difference after it is delivered to everyone via the tech pipeline. 

As we began building out the new version of Yahoo Mail, we wanted to ensure that our internal developer productivity would not be hindered by how our pipelines work. Keeping this in mind, we identified the following principles as most important while designing the delivery pipeline for the new Yahoo Mail experience: 

Product updates are pushed at regular intervals

Releases are stable

Builds are not blocked by irrational test failures

Developers are notified of code pushes

Hotfixes

Rollbacks

Heartbeat pushes 

Product updates are pushed at regular intervals 

We ensure that our engineers can push any code changes to all Mail users everyday, with the ability to push multiple times a day, if necessary or desired. This is possible because of the time we spent building a solid testing infrastructure, which continues to evolve as we scale to new users and add new features to the product. Every one of our builds runs 10,000+ unit tests and 5,000+ integration tests on various combinations of operating systems and browsers. It is important to push product updates regularly as it allows all our users to get the best Mail experience possible. 

Releases are stable 

Every code release starts with the company’s internal audience first, where all our employees get to try out the latest changes before they go out to production. This begins with our alpha and beta environments that our Mail engineers use by default. Our build then goes out to the canary environment, which is a small subset of production users, before making it to all users. This gives us the ability to analyze quality metrics on internal and canary servers before rolling the build out to 100% of users in production. Once we go through this process, the code pushed to all our users is thoroughly baked and tested. 

Builds are not blocked by irrational test failures 

Running tests using web drivers on multiple browsers, as is standard when testing frontend code, comes with the problem of tests irrationally failing. As part the Yahoo Mail continuous delivery pipeline, we employ various novel strategies to recover from such failures. One such strategy is recording the data related to failed tests in the first pass of a build, and then rerunning only the failed tests in the subsequent passes. This is achieved by creating a metadata file that stores all our build-related information. As part of this process, a new bundle is created with a new set of code changes. Once a bundle is created with build metadata information, the same build job can be rerun multiple times such that subsequent reruns would only run the failing tests. This significantly improves rerun times and eliminates the chances of build detentions introduced by irrational test failures. The recorded test information is analyzed independently to understand the pattern of failing tests. This helps us in improving the stability of those intermittently failing tests. 

Developers are notified of code pushes 

Our build and deployment pipelines collect data related to all the authors contributing to any release through code commits or by merging various pull requests. This enables the build pipeline to send out email notifications to all our Mail developers as their code flows through each environment in our build pipeline (alpha, beta, canary, and production). With this ability, developers are well aware of where their code is in the pipeline and can test their changes as needed. 

Hotfixes 

We have also created a pipeline to deploy major code fixes directly to production. This is needed even after the existence of tens of thousands of tests and multitudes of checks. Every now and then, a bug may make its way into production. For such instances, we have hotfixes that are very useful. These are code patches that we quickly deploy on top of production code to address critical issues impacting large sets of users. 

Rollbacks 

If we find any issues in production, we do our best to minimize the impact on users by swiftly utilizing rollbacks, ensuring there is zero to minimal impact time. In order to do rollbacks, we maintain lists of all the versions pushed to production along with their release bundles and change logs. If needed, we pick the stable version that was previously pushed to production and deploy it directly on all the machines running our production instance. 

Heartbeat pushes

As part of our continuous delivery efforts, we have also developed a concept we call heartbeat pushes. Heartbeat pushes are notifications we send users to refresh their browsers when we issue important builds that they should immediately adopt. These can include bug fixes, product updates, or new features. Heartbeat allows us to dynamically update the latest version of Yahoo Mail when we see that a user’s current version needs to be updated.

Yahoo Mail Continuous Delivery Flow

In building the new Yahoo Mail experience, we knew that we needed to revamp from the ground up, starting with our continuous integration and delivery pipeline. The guiding principles of our new, forward-thinking infrastructure allow us to deliver new features and code fixes at a very high launch velocity and ensure that our users are always getting the latest and greatest Yahoo Mail experience.

[yahooeng]

Open Sourcing Screwdriver, Yahoo’s Continuous Delivery Build System for Dynamic Infrastructure

By James Collins, Sr. Director, Developer Platforms and Services, and St. John Johnson, Principal Engineer

Continuous Delivery enables software development teams to move faster and adapt to users’ needs quicker by reducing the inherent friction associated with releasing software changes. Yahoo's engineering has modernized as it has embraced Continuous Delivery as a strategy for improving product quality and engineering agility. All our active products deliver from commit to production with full automation and this has greatly improved Yahoo’s ability to deliver products.  

Part of what enabled Yahoo to make Continuous Delivery at scale a reality was our improved build and release tooling. Now, we are open sourcing an adaptation of our code as Screwdriver.cd, a new streamlined build system designed to enable Continuous Delivery to production at scale for dynamic infrastructure.

Some of the key design features of Screwdriver have helped Yahoo achieve Continuous Delivery at scale. At a high level these are:

Making deployment pipelines easy

Optimizing for trunk development

Making rolling back easy

Easy deployment pipelines: Deployment pipelines that continuously test, integrate, and deploy code to production greatly reduce the risk of errors and reduce the time to get feedback to developers. The challenge for many groups had been that pipelines were cumbersome to setup and maintain. We designed a solution that made pipelines easy to configure and completely self-service for any developer. By managing the pipeline configuration in the code repository Screwdriver allows developers to configure pipelines in a manner familiar to them, and as a bonus, to easily code review pipeline changes too.

Trunk development: Internally, we encourage workflows where the trunk is always shippable. Our teams use a modified GitHub flow for their workflows. Pull Requests (PRs) are the entry point for running tests and ensuring code that entered the repository has been sufficiently tested. Insisting on formal PRs also improves the quality of our code reviews.

To ensure trunks are shippable, we enable functional testing of code in the PRs. Internally, this is a configuration baked into pipelines that dynamically allocates compute resources, deploys the code, and runs tests. These tests include web testing using tools like Selenium. These dynamically-allocated resources are also available for a period after the PR build, allowing engineers to interact with the system and review visual aspects of their changes.

Easy rollbacks: To allow for easy code rollbacks, we allow phases of the pipeline to be re-run at a previously-saved state. We leverage features in our PaaS to handle the deployment, but we store and pass metadata to enable us to re-run from a specific git SHA with the same deployment data. This allows us to roll back to a previous state in production. This design makes rolling back as easy as selecting a version from a dropdown menu and clicking “deploy.” Anyone with write access to the project can make this change. This helped us move teams to a DevOps model where developers were responsible for the production state.

The successful growth of Screwdriver over the past 5 years at Yahoo has today led to Screwdriver being synonymous with Continuous Delivery within the company. Screwdriver handles over 25,000+ builds per day and 12,000+ daily git commits as a single shared entrypoint for Yahoo. It supports multiple languages and handles both virtual machine and container-based builds and deployment.

Screwdriver.cd’s architecture is comprised of four main components: a frontend for serving content to the user, a stateless API that orchestrates between user interactions and build operations, the execution engines (Docker Swarm, Kubernetes, etc.) that checkout source code and execute in containers, and the launcher that executes and monitors commands inside the container.

The diagram below shows this architecture overlaid with a typical developer flow.

To give some context around our execution engines, internal Screwdriver started as an abstraction layer on top of Jenkins and used Docker to provide isolation, common build containers, etc. We used features provided by Jenkins plugins to leverage existing work around coverage and test reports. However, as Screwdriver usage continued to climb, it outgrew a single Jenkins cluster. So in order to grow to our needs, we added capabilities in Screwdriver that allowed us to scale horizontally while also adding capabilities to schedule pipelines across a number of Jenkins clusters. As we scaled Screwdriver, we used less from Jenkins and built more supporting services utilizing our cloud infrastructure. The open-source version is focused on Kubernetes and Docker Swarm as our primary supported execution engines.

In the coming months we will expand our offering to match many of the features we have internally, including:

Mechanism to store structured build data for later use (last deployed version, test coverage, etc.)

Built-in metric collecting

System-wide templates to enable getting started quickly

Log analysis to provide insights to developers

Please join us on the path to making Continuous Delivery easy. Visit http://screwdriver.cd to get started.

[maxvalid]

7 Essential Steps for Effective CI/CD Pipeline Setup in 2026

⏱️ 5 minutes read · 856 words In the fast-paced world of software development, maintaining efficiency while ensuring high-quality output is a top priority. This is where a Continuous Integration and Continuous Deployment (CI/CD) pipeline setup becomes invaluable. As we move into 2026, integrating a robust CI/CD pipeline setup is crucial for any company looking to stay ahead of the competition.…