Too busy to read?

"Some of the wisest auditors say that there are only three internal control objectives: to gain assurance for reliability of financial reporting, compliance with laws and regulations, and efficiency and effectiveness of operations. That's it. What you and John are talking about are just different slides of what is called the 'coso Cube.'"

the Three Ways," he says. "The First Way helps us understand how to create fast flow of work as it moves from Development into it Operations, because that's what's between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery."

I have a sinking feeling in the pit of my stomach. How can we manage production if we don't know what the demand, priorities, status of work in process, and resource availability are? Suddenly, I'm kicking myself that I didn't ask these questions on my first day.

Chester, your peer in Development, is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful 'itties.

We're still struggling on how to prioritize our own seventy-three internal projects," she says, her expression turning glum. "There's still way too many. We've spent weeks with all the team leads trying to establish some sort of relative importance level, but that's all we've done. Argue." She flips to the second page. "The projects seem to fall into the following categories: replacing fragile infrastructure, vendor upgrades, or supporting some internal business requirement. The rest are a hodgepodge of audit and security work, data center upgrade work, and so forth."

Because of our ever-improving production monitoring of the infrastructure and applications, more often than not, we know about the incidents before the business does.

"It's like the free puppy," I continue. "It's not the upfront capital that kills you, it's the operations and maintenance on the back end." Chris cracks up. "Yes, exactly! They'll say, 'The puppy can't quite do everything we need. Can you train it to fly airplanes? It's just a simple matter of coding, right?'"

Many DevOps practices emerge if we continue to manage our work beyond the goal of "potentially shippable code" at the end of each iteration, extending it to having our code always in a deployable state, with developers checking into trunk daily, and that we demonstrate our features in production-like environments.

"There are five focusing steps which Sensei Goldratt describes in The Goal: Step 1 is to identify the constraint. You've done that, so congratulations. Keep challenging yourself to really make sure that's your organizational constraint, because if you're wrong, nothing you do will matter. Remember, any improvement not made at the constraint is just an illusion, yes? "Step 2 is to exploit the constraint," he continues. "In other words, make sure that the constraint is not allowed to waste any time. Ever. It should never be waiting on any other resource for anything, and it should always be working on the highest priority commitment the it Operations organization has made to the rest of the enterprise. Always."

"Dr. Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion. Astonishing, but true! Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck.

"In these competitive times, the name of the game is quick time to market and to fail fast. We just can't have multiyear product development timelines, waiting until the end to figure out whether we have a winner or loser on our hands. We need short and quick cycle times to continually integrate feedback from the marketplace. "But that's just half the picture," she continues. "The longer the product development cycle, the longer the company capital is locked up and not giving us a return. Dick expects that on average, our r&d investments return more than ten percent. That's the internal hurdle rate. If we don't beat the hurdle rate, the company capital would have been better spent being invested in the stock market or gambled on racehorses.

Similarly, we were all amazed that we had a qa environment available that matched Dev so early in the project. That, too, was unprecedented. We needed to make a bunch of adjustments to reflect that the Dev systems had considerably less memory and storage than qa, and qa had less than those in Production. But the vast majority of the environments were identical and could be modified and spun up in minutes.

Wes laughs and adds wryly, "Yeah, in the case of puccar, have the coroner stock up on a bunch of body bags, too. And a pr person ready to handle the angry phone calls from the business, saying that some customers were allergic to the foam we used." I laugh. "You know, that's an interesting idea. Let's let the business choose the foam. There's no reason why all the responsibility should rest on our shoulders. We can send an e-mail out to the business ahead of time and ask when the best implementation time would be. If we can give them data on the outcomes of previous changes, they may even withdraw the change." Patty is typing away. "Got it. For these types of changes, I'll have my staff generate some reports on the changes' success rates and any associated downtime. This will help the business make more informed decisions around the changes."

"We spent a lot of blood, sweat, and tears creating our old change management policy, and everyone still blew it off. What makes you think this will be any different?" I shrug. "I don't know. But we'll keep trying things until we have a system that works, and I'm going to make sure everyone keeps helping us get there. It's not just to satisfy the audit findings. We need some way to plan, communicate, and make our changes safely. I can guarantee you that if we don't change the way we work, I'll be soon out of a job."

I nod silently but refuse to say more. I always liked that phrase in Saving Private Ryan: "There's a chain of command: gripes go up, not down."

We make a pile of nearly fifty cards proposing changes to the Rainbow, Saturn, and Taser applications, and also changes to the network and certain shared databases, which could impact a significant portion, or even all, of the business. "Even looking at those cards makes my heart palpitate," Wes says. "These are some of the dangerous changes we make around here." He's right. I say, "Okay, let's mark all of these as 'fragile.' These are high risk and must be authorized by the cab. Patty, changes like this should be at the top of the pile during our meetings." Patty nods, taking notes saying, "Got it. We're predefining high-risk categories of change that not only must have change requests submitted, but must have authorization before being scheduled and implemented."

Imagine a world where product owners, Development, QA, IT Operations, and Infosec work together, not only to help each other, but also to ensure that the overall organization succeeds. By working toward a common goal, they enable the fast flow of planned work into production (e.g., performing tens, hundreds, or even thousands of code deploys per day), while achieving world-class stability, reliability, availability, and security.

"Yes, I think I can," I say. "At the plant, I gave you one category, which was business projects, like Phoenix," I say. "Later, I realized that I didn't mention internal it projects. A week after that, I realized that changes are another category of work. But it was only after the Phoenix fiasco that I saw the last one, because of how it prevented all other work from getting completed, and that's the last category, isn't it? Firefighting. Unplanned work."

Myth--DevOps Means Eliminating IT Operations, or "NoOps:" Many misinterpret DevOps as the complete elimination of the IT Operations function. However, this is rarely the case. While the nature of IT Operations work may change, it remains as important as ever. IT Operations collaborates far earlier in the software life cycle with Development, who continues to work with IT Operations long after the code has been deployed into production.

"you need to create what Humble and Farley called a deployment pipeline. That's your entire value stream from code check-in to production. That's not an art. That's production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That's how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at."

Is it a change or not? I can see both sides of the argument. After thirty minutes of arguing, it's still not clear that we know the definition of what a "change" should be. Was rebooting a server a change? Yes, because we don't want anyone rebooting servers willy-nilly, especially if it's running a critical service. How about turning off a server? Yes, for the same reason. How about turning on a server? No, we all thought. That is, until someone came up with the example of turning on a duplicate dhcp server, which screwed up the entire enterprise network for twenty-four hours. A half hour later, we finally write on the whiteboard: "a 'change' is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered."

"With the current process, two issues keep coming up: At every stage of the deployment process, environments are never available when we need them, and even when they are, there's considerable rework required to get them all synchronized with one another. Yes?" Wes snorts, saying, "No reward for stating something that obvious, but you're right." She continues, "The other obvious source of rework and long setup time is in the code packaging process, where it Operations takes what Development checks into version control and then generates the deployment packages. Although Chris and his team do their best to document the code and configurations, something always falls through the cracks, which are only exposed when the code fails to run in the environment after deployment.

Right now, we focus mostly on having deployable code at the end of the project. I propose we change that requirement. At each three-week sprint interval, we not only need to have deployable code but also the exact environment that the code deploys into, and have that checked into version control,

I think about Erik challenging me to think like a plant manager as opposed to a work center supervisor. I suddenly realize that he probably meant that I needed to span the departmental boundaries of Development and it Operations. "You guys are both correct," I say, interrupting Wes and William. "William, would you mind writing down all the steps on the whiteboard? I'd suggest starting at 'code committed,' and keep going until the handoff to our group." He nods and walks to the whiteboard and starts drawing boxes, discussing the steps as he goes. Over the next ten minutes, he proves that there are likely over one hundred steps, including the automated tests run in the Dev environment, creating a qa environment that matches Dev, deploying code into it, running all the tests, deploying and migrating into a fresh staging environment that matches qa, load testing, and finally the baton being passed to it Operations.

"Let's use the example of configuring a server. It involves procurement, installing the os and applications on it according to some specification, and then getting it racked and stacked. Then we validate that it's been built correctly. Each of these steps are typically done by different people. Maybe each step is like a work center, each with its own machines, methods, men, and measures."

Beyond the Goal,

"You probably don't even see when work is committed to your organization. And if you can't see it, you can't manage it--let alone organize it, sequence it, and have any assurance that your resources can complete

Beyond the Phoenix Project,

I indicate to Brent and Wes that one of them should continue where William left off. Brent gets up and starts drawing boxes to indicate the packaging of the code for deployment; preparing new server instances; loading and configuring the operating system, databases, and applications; making all the changes to the networks, firewalls, and load balancers; and then testing to make sure the deployment completed successfully.

Realizing the importance and enormity of the challenge in front of us, I walk to the whiteboard, and pick up the red marker. I say, "I'm going to put a big red star on each step where we had problems during previous launches." Starting to make marks on the whiteboard, I explain, "Because a fresh qa environment wasn't available, we used an old version; because of all the test failures, we made code and environment changes to the qa environment, which never made it back into the Dev or Production environments; and because we never synchronized all the environments, we had the same problems the next time around, too."

"Everyone knows that in manufacturing, as wip increases, due-date performance goes down.

Continuous Delivery

'Messiahs are good, but scripture is better.'"

Five Dysfunctions of a Team, by Patrick Lencioni.

Continuous Delivery.

Visible Ops Security.

data, which, incidentally, form two of the three legs of the 'confidentiality, integrity, and availability triangle' or cia."

"In the 1980s, this plant was the beneficiary of three incredible scientifically-grounded management movements. You've probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: wip is the silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it, you can't control wip."