Oblivious Delivery

A journey from null to Continuous Delivery... and random cat pics

[obliviousdelivery-blog]

Zero Downtime Deploys Using Blue/Green Environments

Zero downtime is a new deploy requirement we have been trying to absorb at work. In the past this was accomplished with about 25 people on a 12 hour deploy call and lots and lots of manual operations work.

My goal was to automate the process using a BlueGreen strategy: http://martinfowler.com/bliki/BlueGreenDeployment.html

First I broke the environment into two. Luckily we have enough capacity (for the most part) without impacting performance.

Then I created new VIPs and internal dns for the new separate blue and green internal environments. 

I found I could swing the F5 load balancers most efficiently using irules directly so I went that route as a first pass. There would be advantages to doing it with members directly but that adds complexity I was trying to avoid.

The database replication is just using postgres streaming replication and I just had to automate the promotion and switch-back to hot standby after a deploy.

Here is a diagram:

This is what the orchestration looks like this:

Overall I think it is a good first step. The deploy does not impact customers as long as the sessions are maintained. During step 3 you don’t want to reenable replication until the deploy is successful so there is a short period after the new env db is promoted and until the external irule is switched when writes are stopped to prevent data loss. That is the biggest area where improvements can be made...

[obliviousdelivery-blog]

Evolutionary Database Design Retrospective

The team I work with has been using Liquibase to manage database refactoring for just over a year now, this is a summary of my impressions on that experience. Everything I write is just as applicable to Flyway  or any other migration tool.

So some background... The idea of our PostgreSQL database migrations living along with the code became a big topic around the office about four years ago. Through some staff changes, working with our DBAs to apply migrations was becoming a deep slog through red tape. So right off the bat our target was to simplify deploys and work around DBA policies, not necessary implement any other tenants of continuous delivery, or necessarily adopt the requisite “best practices” around Evolutionary DB Design. Kick ass right.

So the tooling we implemented was simple. Allow a developer to spin-up a local Postgres instance with appropriate customization's using Docker and apply schema and test data through a DB project on the trunk. The developer can build migrations, apply them locally and write simple unit tests to verify them. The migrations and tests are checked in together. A CI job builds the DB project (just applies migrations and runs tests) and is a prerequisite to the code being promoted to an environment just like any other unit test. Deployments apply migrations through Liquibase and version information is available through the Liquibase table in the public schema.

I believe the implementation is good. Some pluses are:

+ Agile teams write and test DB migrations as the team see fit, without any DBA’s on the team. (some might see this as a minus, in my experience it is a plus)

+ Migrations are applied though the deployment pipeline the same way. Any issues typically show up before anything gets close to production.

+ DB’s have a definite version which can be quantified and labeled/branched/released.

+ Databases roll forward! 

Some Minuses:

- Evolutionary DB Design works as a part of a continuous delivery strategy. To adopt one part of continuous delivery without working toward the goal of constantly pushing software complicates some of the design decisions of the tooling. For example:

- No Rollbacks! We don’t do rollbacks. They are impossibly hard to test and unpredictable, we only roll forward. This is an inconceivable concept to to someone who has not embraced agile development methodology. “You are giving the database a version, then why cant I go back to an older version? I don’t get it.” Its been a year plus and I still have to explain this to managers and other new engineers on a monthly basis.

- Long release cycles mean large numbers of migrations, this means longer test cycles and greater risk during deploys. Again, you should be pushing often.

- Developers don’t like to test. Maybe this is just my team but in general the majority of developers feel like writing tests is a waste of time for them. If we did more pairing and XP I don’t think this would not be an issue.

So overall its been very positive. I would suggest NOT adopting this practice as a first step to continuous delivery otherwise it’s pretty neat.

[obliviousdelivery-blog]

Build Automation and Managing Artifacts

One problem with a large monolithic Maven build’s (hah!) is ending up with artifacts from partial builds in your Artifactory/Nexus repository. Say you have project A with subprojects 1,2,3. subprojects 1 and 2 build and deploy but subproject 3 fails. Project version is toast but now you have some rouge artifacts sitting in your repo. One solution is to use a plugin for Jenkins that manages your artifacts until the build completes then uploads as a final step. Downfall is you are now dependent on the plug-in and haven’t we all been burned by that in the past. My solution is to just script Maven to use a staging location to hold the artifacts until everything compiles and unit testing is complete, then upload.

This also allows you to separate the build, and test steps to fail faster and complete builds quicker. (compile everything in parallel then unit test in series). Because we know how thread safe those unit tests are. 

export MAVEN_SETTINGS=“-Dmaven.repo.local=~/.repository –settings ~/maven_settings/maven_settings.xml” 

 # Compile 

${MAVEN_HOME}/bin/./mvn ${MAVEN_SETTINGS} -B -U -T 1C clean -DaltDeploymentRepository=repo::default::file://~/staging deploy -DskipTests=true 

# Now Test 

 ${MAVEN_HOME}/bin/./mvn ${MAVEN_SETTINGS} -B -U test 

# Now Deploy 

repo_url=“http://my_repo:8080/artifactory” tgt_repo=“libs-local-some-repo” dir=~/staging-repo echo -e “\033[2;32m UPLOAD ARTIFACTS \e[0m” find “$dir” -type f | while read f; do  rel=“$(echo ”$f" | sed -e “s#$dir##” -e “s# /#/#”)“;  if [[ $rel = *.md5 || $rel = *.sha1 ]]  then     continue  fi    sha1=$(sha1sum ”$f")    sha1=“${sha1:0:40}”    printf “\n\nUploading ’$f’ (cs=${sha1}) to ’${repo_url}/${tgt_repo}${rel}’”    status=$(curl -k -u $user:$pass -X PUT -H “X-Checksum-Deploy:true” -H “X-Checksum-Sha1:$sha1” –write-out %{http_code} –silent –output /dev/null “${repo_url}/${tgt_repo}${rel}”)    echo “status=$status”    # No checksum found - deploy + content    [ ${status} -eq 404 ] && { curl -k -u $user:$pass -H “X-Checksum-Sha1:$sha1” -T “$f” “${repo_url}/${tgt_repo}${rel}”    } done

You can use the multi-job plugin to link these steps together or, maybe better is the workflow plugin separate these steps into scripts and use one groovy file to orchestrate everything. 

[obliviousdelivery-blog]

Do you even plug-ins bro?

I have noticed at work it seems to be popular in to jam as much build architecture as possible into as many Jenkins plug-ins as one can find. I see more-and-more plug-in heavy Jenkins installations crunching along that are trading configuration complexity for some simple scripting or legacy tools. Maybe it’s all the pretty icons on the job page?

“But I get a pretty build report... and the linky is right on the results page with a pretty green icon...”

I am much more confident in a completely stripped vanilla Jenkins install full of boring freestyle jobs that look like this: 

1. pull script 

2. pull source 

3. run script

That seems like the golden ticket to me.

Infrastructure as code, not configuration.

Need to move from Jenkins to another CI? No problem, the orchestration is all scripted in scm not tied to some plug-in that probably has more bugs then you realize. https://goo.gl/9gnres

Need to adjust the logic? Just grab the script and modify. And hey, now you have a log of modifications made to your Jenkins jobs right from your scm. No need to guess which of your 12 Jenkins admins f’d the build job.

Yes, I do use plug-ins, quite a few actually, but my jobs don’t depend on them for basic functionality. 

Pretty test reports, graphs, job email notifications with cat pics? Yes please.    

[obliviousdelivery-blog]

One of my favorite retailers has a release process that rivals a NASA launch sequence. It starts in the afternoon and runs until the wee hours of the morning. In the early days, more than twenty people had active roles to play during the release. These days, that is down to less than a dozen. As you might imagine, any process involving that many people requires detailed planning and coordination. Because each release is arduous, they don’t do many of them a year. ... One solution—the easy but harmful one—is to slow down the release calendar. Like going to the dentist less frequently because it hurts, this response to the problem can only exacerbate the issue. The right response is to reduce the effort needed, remove people from the process, and make the whole thing more automated and standardized.

Michael Nygard -  Release It!