Too busy to read?

Many DevOps practices emerge if we continue to manage our work beyond the goal of "potentially shippable code" at the end of each iteration, extending it to having our code always in a deployable state, with developers checking into trunk daily, and that we demonstrate our features in production-like environments.

Similarly, we were all amazed that we had a qa environment available that matched Dev so early in the project. That, too, was unprecedented. We needed to make a bunch of adjustments to reflect that the Dev systems had considerably less memory and storage than qa, and qa had less than those in Production. But the vast majority of the environments were identical and could be modified and spun up in minutes.

Wes laughs and adds wryly, "Yeah, in the case of puccar, have the coroner stock up on a bunch of body bags, too. And a pr person ready to handle the angry phone calls from the business, saying that some customers were allergic to the foam we used." I laugh. "You know, that's an interesting idea. Let's let the business choose the foam. There's no reason why all the responsibility should rest on our shoulders. We can send an e-mail out to the business ahead of time and ask when the best implementation time would be. If we can give them data on the outcomes of previous changes, they may even withdraw the change." Patty is typing away. "Got it. For these types of changes, I'll have my staff generate some reports on the changes' success rates and any associated downtime. This will help the business make more informed decisions around the changes."

To transmit code through the value stream requires multiple departments to work on a variety of tasks, including functional testing, integration testing, environment creation, server administration, storage administration, networking, load balancing, and information security.

The simplest of the three patterns is called blue-green deployment. In this pattern, we have two production environments: blue and green. At any time, only one of these is serving customer traffic--in figure 20, the green environment is live. Figure 20: Blue-green deployment patterns (Source: Humble and North, Continuous Delivery, 261.) To release a new version of our service, we deploy to the inactive environment where we can perform our testing without interrupting the user experience. When we are confident that everything is functioning as designed, we execute our release by directing traffic to the blue environment. Thus, blue becomes live and green becomes staging. Roll back is performed by sending customer traffic back to the green environment.

By creating launch guidance, we help ensure that every product team benefits from the cumulative and collective experience of the entire organization, especially Operations. Launch guidance and requirements will likely include the following: Defect counts and severity: Does the application actually perform as designed? Type/frequency of pager alerts: Is the application generating an unsupportable number of alerts in production? Monitoring coverage: Is the coverage of monitoring sufficient to restore service when things go wrong? System architecture: Is the service loosely-coupled enough to support a high rate of changes and deployments in production? Deployment process: Is there a predictable, deterministic, and sufficiently automated process to deploy code into production? Production hygiene: Is there evidence of enough good production habits that would allow production support to be managed by anyone else?

"you need to create what Humble and Farley called a deployment pipeline. That's your entire value stream from code check-in to production. That's not an art. That's production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That's how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at."

"With the current process, two issues keep coming up: At every stage of the deployment process, environments are never available when we need them, and even when they are, there's considerable rework required to get them all synchronized with one another. Yes?" Wes snorts, saying, "No reward for stating something that obvious, but you're right." She continues, "The other obvious source of rework and long setup time is in the code packaging process, where it Operations takes what Development checks into version control and then generates the deployment packages. Although Chris and his team do their best to document the code and configurations, something always falls through the cracks, which are only exposed when the code fails to run in the environment after deployment.

Right now, we focus mostly on having deployable code at the end of the project. I propose we change that requirement. At each three-week sprint interval, we not only need to have deployable code but also the exact environment that the code deploys into, and have that checked into version control,

I think about Erik challenging me to think like a plant manager as opposed to a work center supervisor. I suddenly realize that he probably meant that I needed to span the departmental boundaries of Development and it Operations. "You guys are both correct," I say, interrupting Wes and William. "William, would you mind writing down all the steps on the whiteboard? I'd suggest starting at 'code committed,' and keep going until the handoff to our group." He nods and walks to the whiteboard and starts drawing boxes, discussing the steps as he goes. Over the next ten minutes, he proves that there are likely over one hundred steps, including the automated tests run in the Dev environment, creating a qa environment that matches Dev, deploying code into it, running all the tests, deploying and migrating into a fresh staging environment that matches qa, load testing, and finally the baton being passed to it Operations.

I indicate to Brent and Wes that one of them should continue where William left off. Brent gets up and starts drawing boxes to indicate the packaging of the code for deployment; preparing new server instances; loading and configuring the operating system, databases, and applications; making all the changes to the networks, firewalls, and load balancers; and then testing to make sure the deployment completed successfully.

Realizing the importance and enormity of the challenge in front of us, I walk to the whiteboard, and pick up the red marker. I say, "I'm going to put a big red star on each step where we had problems during previous launches." Starting to make marks on the whiteboard, I explain, "Because a fresh qa environment wasn't available, we used an old version; because of all the test failures, we made code and environment changes to the qa environment, which never made it back into the Dev or Production environments; and because we never synchronized all the environments, we had the same problems the next time around, too."