What is Canary Analysis ?
This a two-part blog series about Canary Deployment and Canary Analysis. In the first part, we have discussed canary as a release strategy, canary deployment stages, benefits of using canary and constraints with implementation. In the second part, we discussed how canary analysis is performed after canary deployments are done, selection and evaluation criteria for choosing different metrics and smoke tests to perform the analysis and canary hypothesis testing performed with incremental traffic to the canary instances. Here we will discuss Canary Analysis.
Canary analysis is a two step process where we will assess the canary based on a selected list of metrics and logs to infer if we are going to promote or roll back the new version.
The two steps to perform Canary Analysis are:
Correct metrics selection and evaluation
Selection and evaluation of metrics
Further the selection and evaluation criteria are based on the following:
Choosing the most important business metrics
Balanced set of fast and slow metrics
The last step of canary analysis is evaluating the values of the metrics also known as Canary Hypothesis Testing which enables us to assess if the canary instance should be promoted to production or not. It is performed with incremental traffic to the canary instances.
As software deployments move at scale and velocity it becomes highly crucial to have a proper system in place that assess these releases before they damage the production environment leading to losses for an organization and irate customers. canary deployments provide an excellent approach to reducing the risk of introducing a defect into production, are relatively low cost, and do not slow the process much as long as appropriate automation is involved. One way to increase the speed of canary analysis is to automate the verification process. To read more about canary analysis please read our blogs - What is Canary Analysis ? and “What is Canary Deployment ?”
