Too busy to read?

Curious, she asks one of the them why he doesn't seem very worried. He replies with a puzzled look, "I'm a developer--I work on features. I give them to QA and Ops to test and deploy. Then I work on the features for the next release. That keeps me plenty busy."

Punishing failure and "shooting the messenger" only cause people to hide their mistakes, and eventually, all desire to innovate is completely extinguished.

"Business people can see features or apps, so getting funding for those is easy," he continues. "But they don't see the vast architectures underneath that support them, connecting systems, teams, and data to each other. And underneath that is something extraordinarily important: the systems that developers use in their daily work to be productive.

Without constant feedback from a centralized build, integration, and test system, they really have no idea what will happen when all their work is merged with everyone else's.

Everyone around here thinks features are important, because they can see them in their app, on the web page, or in the API. But no one seems to realize how important the build process is. Developers cannot be productive without a great build, integration, and test process.

This service calls out to a bunch of external APIs from the shipping providers, and some of those are failing. Brent suspects that they are being rate-limited by one of them, because they'd never had Parts Unlimited servers send so many queries like this before. Maxine can't believe that a service that seems so trivial is jeopardizing the entire launch. She smiles and makes a note of this, because she knows that this will likely be the new normal. But for something this mission-critical, there's no way we should depend on external services, she thinks. We need to gracefully handle the case when they're down or when they cut us off.

"It's funny: the tech giants assign their very best engineers to that bottom layer, so that every developer can benefit. But at Parts Unlimited, the very best engineers work on features at that top layer, with no one besides interns on the bottom working on Dev productivity.

expect leaders to buffer their people from all the political and bureaucratic insanity, not throw them into it.

"Without continuous builds, we are like a car manufacturer without an assembly line, where anyone can do whatever they want, independent of the plant goals," he continues. "We need to discover problems only when we are in the build or testing process, not during deployment or production.

This is like labeling all the deck chairs on the Titanic.

Prevention requires honesty, and honesty requires the absence of fear. Just like Norm Kerth says in the Agile Prime Directive, 'Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.'

production deployments are some of the most complex activities of any technology organization, because they require so much coordination between so many different parts of the organization.

Technical debt is inherently neither good nor bad--it happens because in our daily work, we are always making trade-off decisions," he says. "To make the date, sometimes we take shortcuts, or skip writing our automated tests, or hard-code something for a very specific case, knowing that it won't work in the long-term. Sometimes we tolerate daily workarounds, like manually creating an environment or manually performing a deployment. We make a grave mistake when we don't realize how much this impacts our future productivity."

She's learned over her entire career that when people can't get their builds going consistently, disaster is usually right around the corner.

She wonders what is happening: Too many promises to the market? Bad engineering leadership? Bad product leadership? Too much technical debt? Not enough focus on architectures and platforms that enable developers to be productive?

She had also forgotten that when you throw the Operations teams into the mix, the number of teams involved more than doubles, because getting all those applications running in production requires the server administration teams, virtualization teams, cloud teams, storage teams, networking teams

Ops, it's worse, because we have so many areas of specialization. Complex work like deployments hit every one of those areas. We have servers, databases, networks, firewalls ... heck, in the last decade, we've created even more silos, like storage, VLANs, automation teams, virtualization, hyperconverged infrastructure, and who knows how many more.