Updates to AWS CloudFormation: Change Sets Help Get to the Root Cause of Stack Update Failures

By Flux7 Labs
April 7, 2016


 On March 29, 2016, Amazon released  Change Sets for AWS CloudFormation, an important new update with far reaching benefits. Anyone using CloudFormation templates, anyone pursuing an infrastructure as code strategy on AWS, should pay attention.

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, allowing them to provision and update them in an orderly and  predictable fashion.

The truth is, updates can fail,  leaving the stack in an inconsistent state. The result is unwanted cost, wasted resources, delay and downtime.

As AWS partners, we push multiple AWS stack updates every week to help our customers bring new solutions to market quickly. At every update, there is a risk to business continuity; when changes have been made manually to a stack created by one template, a new update can spell failure for services built on the newly modified template.

Even when an AWS consultant was able to spend hours carefully reviewing updates, there was no way to know for certain what effects the update would have.  Thus, the engineers had to engage in the “Push and Pray” method, often requiring that additional resources  be on-hand in case the update failed part of the way through a stack, or resulting in the downtime of key applications or services.

This constant “fear factor” when updating the stacks inherently prevents engineers from innovation,  trying new changes, and impacts the company’s very ability to be creative, agile and efficient.

As an interim solution, AWS launched the “Continue Rollback Update” feature detailed in the AWS DevOps blog Continue Rolling Back an Update for AWS CloudFormation stacks in the UPDATE ROLLBACK FAILED state. This feature allows engineers to quickly revert back to the last known good state. While an immense help in mitigating downtime, it doesn’t really help engineers address the real root of the failure: Human errors (ie, cloudformation bugs or manual changes to the stack).

The new update, Change Sets, allows engineers to proactively ‘preview’ the impact of changes, catch and prevent failures earlier and with higher probability.  It means they can better make proactive fixes to prevent failures and maintain agility. No praying, no guesswork, no surprises.

We recommend that organizations using AWS CloudFormation regularly add a ‘Review Change Set’ step in their CloudFormation stack update processes, and require the change sets related to all stack updates to be reviewed before they are pushed.  This critical step can prevent loss of expensive engineering time and the potential loss of  business due to production failures.

To read more about CloudFormation, click the button below

Read More


If you are not yet using CloudFormation and would like to learn more about Infrastructure as Code and how it can help your company’s agility & efficiency, contact us to start the process.


Did you find this useful?  

Interested in getting tips, AWS News, best practices and commentary delivered regularly? Click the button below to sign up for our blog and set your topic and frequency preferences.

Subscribe to the Flux7 Blog