The Phoenix Project

Key idea 1

Disorder in the IT department leads to the downfall of the whole company.

It’s Tuesday morning and Bill is behind schedule for work. As he hurries along the highway, he receives a call instructing him to meet with the CEO, Steve, immediately upon his arrival.

Bill worries he might be losing his job. It wouldn’t be entirely unexpected. As the leader of a mid-sized technology group at Parts Unlimited, he has consistently performed his duties with excellence. Known for his reliability, efficiency, and straightforwardness, qualities honed from his time as a Marine, Bill has maintained a strong professional standing. However, the company has been facing significant challenges.

Bill observes the almost vacant parking lot as he arrives. It appears that Parts Unlimited is lagging behind its competitors who are continuously innovating, leaving Parts struggling to keep up. Over the past few years, with numerous layoffs, Bill’s department has been forced to achieve more with fewer resources.

However, Steve isn’t terminating Bill; he’s actually promoting him. As the newly appointed VP of IT Operations, Bill will now report directly to Steve and be responsible for the successful launch of the Phoenix Project. The company’s survival hinges on the success of Phoenix, Steve points out – something Bill is fully aware of. The project is crucial because it enables customers to shop from Parts both online and in physical stores. Without this capability, the company risks losing its customer base and potentially ceasing to exist.

The central point being made is that disorder within the IT department can lead to the downfall of the entire company.

Bill is reluctant to accept the promotion because it essentially means he would be responsible for resolving a massive IT disaster. Yet, he finds himself agreeing to it as he shakes Steve’s hand. Now, he’s faced with the daunting task of tackling the total chaos that the IT infrastructure has become.

John in the Information Security department is repeatedly causing major outages, classified as SEV1 incidents, which require immediate resolution. He does this by circumventing established protocols to expedite Development changes that he considers crucial for auditor reviews. Unfortunately, Operations is unable to test these changes beforehand due to the lack of a testing environment, stemming from budget constraints.

Patty, the director of IT Service Support, tells Bill that changes are not being recorded because no one is willing to use the company’s cumbersome change management tools. Additionally, attendance at the weekly Change Advisory Board (CAB) meetings is consistently low.

Bill wonders how anyone manages to keep track of operations. He comes to the realization that they simply don’t, which explains why everything is in such disarray.

Bill stares at his desk. His old laptop has crashed and shown the blue screen of death, leading him to get a replacement. However, the new one is an outdated, bulky relic, at least a decade old, with worn-off letters and a battery that’s held together with tape. He finds himself questioning if this is some sort of message from the universe.

Key idea 2

IT Operations, similar to Plant Operations, operates under the principles of the theory of constraints.

Phoenix is scheduled for deployment in ten days, yet only three out of the twelve critical tasks have been completed due to the IT department dealing with urgent issues. Specifically, Brent, the lead engineer, has been managing these emergencies. Poor Brent. Bill is baffled by how this one individual appears to be supporting the whole organization, excelling at fixing any issue that comes his way, which results in him facing a continuous stream of them.

The main point is that IT Operations, similar to Plant Operations, operates under the principles of the theory of constraints.

Just as Bill starts settling into his new role, he learns he needs to meet with Erik, a potential board member and reputed tech guru. Upon entering the conference room, he finds only a donut delivery man dressed in crumpled pants and a denim shirt. Surprised, Bill thinks to himself about the convenience of donut delivery as he piles some onto his plate.

Suddenly, the deliveryman turns, offers his hand, and introduces himself as Erik. Without missing a beat, he starts explaining to Bill the issues with IT Operations at Parts. Erik challenges the common misconception that IT is solely knowledge work, exempt from standardization and documentation, suggesting that IT could benefit from adopting practices from Plant Operations. To illustrate his point, Erik tells Bill to gather his belongings for a trip they’re about to take.

Five minutes later, they arrive at one of Parts’ manufacturing facilities. Inside, while observing the factory floor, Erik explains the theory of constraints: In many factories, a limited number of resources—whether materials, machines, or personnel—control the entire system’s output. These are the bottlenecks, or constraints.

To optimize productivity, begin by pinpointing the bottleneck. Without knowing where this bottleneck lies, you cannot effectively direct your improvement efforts. Improvements that do not address the bottleneck are essentially ineffective. Visualize the process as a linear sequence: making enhancements past the bottleneck only leads to increased waiting times. Conversely, improvements made before the bottleneck merely cause a buildup of inventory.

The second step involves fully utilizing the bottleneck. Ensure that your bottleneck is never idle or delayed by other resources, and it should constantly be working on the most critical task.

The final step is to subordinate everything to the constraint. Just as the pace of the slowest Boy Scout sets the speed for the whole troop, position this Scout at the front. Essentially, align all tasks to proceed at the rate that the bottleneck can handle.

Bill comes to understand that Brent, the engineer who always solves everyone’s issues, is their bottleneck. Naturally, this is why tasks that should only take 30 minutes end up taking Brent weeks to complete. He’s constantly working at full capacity, meaning any additional tasks assigned to him end up waiting unless they’re urgent.

It all makes perfect sense. Bill realizes he must align the pace of work with Brent’s capacity. Who would have imagined that IT operations could be similar to managing a factory?

Key idea 3

Grasping the four types of IT work is key to fulfilling your commitments.

With only a few days left until the Phoenix deployment, the project management meeting is tense. Bill has asked for an extension, but Sarah from Retail Operations insists on proceeding with the scheduled launch. As she criticizes the IT Operations team for stalling, Bill pauses to take a deep breath.

He’s familiar with this scenario. The project’s release can’t be postponed due to commitments made to customers or investors. Developers dominate the timeline, squeezing out adequate time for operations testing. Consequently, ridiculous shortcuts are taken, often yielding a final software product that is unstable or unusable. And who ends up working tirelessly, rebooting servers and applying quick fixes to compensate for the poor quality code? IT Operations.

Bill exhales heavily, thinking about something Erik pointed out: the four types of work. Erik had said that rigor and discipline have their limits. Until Bill fully understood the nature of the work in IT Operations, he wouldn’t be able to effectively handle project deliverables, outages, and compliance issues.

The main takeaway is that comprehending the four types of IT work is crucial for fulfilling your obligations.

Bill had explained to Erik that deploying Phoenix was a task in itself. However, Erik pointed out that official company initiatives, or business projects, represented just one type of work. This prompts Bill to think hard about what the other types of work could be, leading to a moment of realization.

The second category of work involves internal IT projects. This includes infrastructure or IT operations projects spawned by business initiatives, as well as enhancement efforts such as deployment automation or environment creation. These internal projects are frequently not monitored centrally, complicating the management of their workflow.

The third category of work, known as changes, typically stems from business and internal projects. These are activities that could affect the services provided. Often, because changes in IT Operations and Development are logged in different systems, errors and inefficiencies frequently occur, particularly during transitions.

Lastly, there’s unplanned work, which can be compared to firefighting. It involves addressing incidents or emergencies that arise from the other three types of work. This kind of work disrupts scheduled tasks and commitments, and it’s currently hindering the Phoenix launch.

Unplanned work prevents you from reaching your objectives, essentially acting as “anti-work.” It’s crucial to eliminate it whenever possible. While life is unpredictable and issues will inevitably arise, the key is to manage unplanned work effectively by identifying its sources.

Frequently, it arises from technical debt, which accumulates when you opt for expedient yet unwise shortcuts. Similar to financial debt, technical debt accrues compound interest, which increases over time. Failing to address this debt means your efforts are consumed by managing the repercussions—manifested as unplanned work.

Key idea 4

Outstanding teams are characterized by trust, communication, dedication, responsibility, and a shared emphasis on collective achievement.

Bill attempts to apply Erik’s teachings, but Steve, the CEO, disrupts this by demanding adherence to the traditional, unrealistic schedule. Predictably, the launch of Phoenix is a disaster. Unfazed, Bill decides to resign.

In his absence, the project’s decline worsens. The various teams, including IT Operations, Development, and business units, scramble to bring some semblance of order from the turmoil, yet they remain in conflict, continually shifting blame among themselves.

They realize they need Bill back. With some persuasion and the offer of an attractive bonus, he agrees to return.

The essential takeaway is that exceptional teams emphasize trust, communication, commitment, accountability, and a shared focus on success.

When Bill enters the conference room, Steve is reminiscing about his youth. He mentions that his family struggled financially. He recalls spending summers picking cotton with his brothers and being the first in his family to attend college.

Bill is not a fan of this sentimental approach, but he recognizes the urgency to end the continuous, unproductive cycle threatening Parts Unlimited. Therefore, all the department heads have gathered to discuss Steve’s favorite book, “The Five Dysfunctions of A Team.”

Steve emphasizes that trust is the cornerstone of effective teamwork, essential for tackling complex business challenges. A lack of trust represents the primary dysfunction within a team. When leaders don’t trust each other, it can lead to persistent conflicts and poor performance, and this lack of trust can cascade down, affecting the entire team.

To foster trust, begin by embracing vulnerability. Steve demonstrates this by initiating a personal history exercise. By sharing personal stories and motivations, he sets an example of vulnerability, encouraging everyone in the group to do the same.

The second dysfunction is fear of conflict. Preferring superficial harmony over honest, vigorous debates leads to stagnation. Open and frank discussions on difficult topics are challenging and demand that leaders and teams break from their habitual responses. However, engaging in these conflicts is essential for advancement.

The third dysfunction is a lack of commitment. When team members only pretend to agree with group decisions, it leads to uncertainty and disinterest within the organization.

The fourth dysfunction involves avoiding accountability. Failing to accept responsibility and hesitating to challenge colleagues on negative actions both contribute to a culture of mediocrity. Lastly, neglecting the team’s results in favor of personal achievements, status, or ego undermines collective success, creating a lose-lose scenario.

Key idea 5

The First Way pertains to the progression of work from Development through IT Operations and ultimately to the customer.

The leadership meeting doesn’t resolve all issues, but that isn’t its purpose. Bill recognizes something more valuable emerging: a unified vision as the team advances towards a solution. Now, it’s crucial for everyone to collaborate. Erik emphasizes that it’s the moment for DevOps, a methodology that merges software development (“Dev”) and IT operations (“Ops”).

Today, DevOps is revolutionizing IT much like the industrial revolution transformed manufacturing. Rather than focusing on the conversion of raw materials to products, DevOps enhances the IT value stream. It accelerates market entry, fortifies system flexibility and resilience, and incorporates testing to rapidly evaluate new innovations.

The business principles that guide DevOps practices are similar to those in manufacturing, according to Erik. He refers to them as The Three Ways, with the first focusing on optimizing workflow.

The main point is that The First Way is centered around the movement of work from Development through IT Operations to the customer.

A key aspect of the First Way is to expedite the workflow through Development and IT Operations. Rather than spending time adjusting to-do lists and reshuffling priorities during weekly executive meetings, implement a kanban board.

A kanban board visually represents all active IT projects in one location. This setup allows every team member to clearly see the scope of work and the tasks that need attention, eliminating the confusion of managing scattered emails, texts, or phone calls.

To set up a kanban board, you’ll need to section off a wall into three parts: Ready, Doing, and Done. Write down each current task on an index card and place it in the appropriate section. Importantly, you should restrict your Work In Progress (WIP) to only four or five tasks at any one time. This approach, a core component of the First Way, minimizes active tasks to enhance focus and speed in completion. As tasks progress, shift the cards from one column to the next, such as moving a card from Ready to Doing when work begins.

Erik’s First Way emphasizes the efficiency of the entire system over individual departments or work silos. Kanban boards play a crucial role not just in reducing Work In Progress (WIP), but also in helping teams select the appropriate tasks in the correct quantities, aligning with overall company performance. Essentially, value is only produced when code reaches production, so eliminating unnecessary tasks is often more beneficial than taking on additional ones. It’s vital to be selective with commitments—having less on your plate usually means greater effectiveness.

Key idea 6

The Second Way focuses on addressing quality issues at their origin to eliminate the need for later corrections.

Erik notes that a 2007 Suzuki Hayabusa dragster motorcycle is visually stunning and capable of speeds exceeding 230 mph. It features a six-gear constant mesh transmission and a #532 chain drive, but it lacks a reverse gear.

He’s illustrating a concept. Just as the high-speed motorcycle lacks a reverse gear, the workflow in IT should also not need to reverse. Any work that must backtrack due to defects or unclear instructions is essentially wasted effort. Ideally, work should only move forward.

Expanding on this automotive metaphor, the concept of the Second Way emphasizes preventative maintenance and regular oil changes to avoid issues down the line.

The main point is this: The Second Way focuses on addressing quality issues right at their origin to eliminate the need for redoing work.

The release cycles for The Phoenix Project are presently set at nine months, which is excessively lengthy if Bill aims to integrate quality from the beginning and maintain a smooth progression. He requires much quicker and more frequent feedback loops from customers or IT Operations back to Development.

As Erik puts it, “Forget about old Civil War cannons. Consider antiaircraft guns instead.”

The aim is to achieve single-piece flow, meaning that one unit of product moves continuously through various processes. Essentially, the focus is on handling one piece of work at a time. This approach reduces waiting time between tasks, maximizes output, minimizes work in progress (WIP), and lowers the likelihood of errors.

Consider the case of Toyota, known for its remarkable single-minute exchange of die. This concept illustrates how, through incremental enhancements, the company managed to streamline its processes to achieve single-piece flow, reducing the time it took to change the dies for stamping car hoods from three days to just under ten minutes.

In the IT sector, single-piece flow can be realized through a continuous delivery model that focuses on three primary strategies. First, it involves utilizing small batch sizes, which facilitates rapid feedback application and necessary adjustments. Second, it includes halting the production line whenever issues arise, specifically avoiding new tasks when failures occur in builds, tests, or deployments.

Third, develop rapid, automated testing procedures to confirm that the code is always deployment-ready. Code isn’t considered “finished” just when development completes; it’s truly done once it’s been fully tested and is functioning as intended. Erik recommends a significant adjustment to the release schedule for Phoenix. Rather than a single deployment every nine months, Bill should target ten deployments daily. Bill finds this idea daunting.

However, that’s precisely what Flickr managed to accomplish in 2009. While the majority of IT departments at that time were conducting quarterly or yearly deployments, Flickr, through a collaboration between Development, Operations, and business teams, and by applying the principles of the Second Way, developed a streamlined, one-step process for environment creation and deployment. This enabled them to achieve ten deployments a day, making them a thousand times faster than their peers at the time.

Currently, that figure has increased even more. Companies like Amazon, Netflix, and Google, which have embraced DevOps methodologies and cultural practices, are now capable of implementing thousands of production changes every day, all while maintaining outstanding stability and security.

Key idea 7

The Third Way focuses on fostering a culture that embraces ongoing experimentation, learning from failures, and continuous enhancement.

Bill is admiring his brand-new laptop. All his applications and data are in place, his email is up and running, and everything is working flawlessly. If Bill’s old laptop symbolized all the problems at Parts Unlimited, his new laptop signifies all that his team has achieved together recently.

SEV1 outages have decreased by two-thirds and the time it takes to recover from incidents has been reduced by 50%. The ticketing system has been cleared of superfluous tasks. There’s now a standardized process for creating environments – or build procedures – which ensures Development and Operations are aligned and variability is eliminated.

Developers are collaborating with Security to initiate preventive projects rather than securing applications post-deployment. This approach has satisfied the auditors and cut down on security-related tasks by 75 percent, freeing up resources to be allocated to other areas.

This has resulted in accelerated project workflow. The Phoenix project is progressing efficiently, with team members understanding their responsibilities and feeling satisfied and content in their roles. Furthermore, through the implementation of the Third Way, they are innovating more than ever before.

The central idea is that the Third Way emphasizes fostering a culture where continuous experimentation, failure, and enhancement are the norm.

The Third Way suggests that stagnation is tantamount to regression. It emphasizes three elements, starting with experimentation. To surpass competitors, it’s crucial to foster a culture that prioritizes innovation and encourages taking risks.

At Parts, a new initiative called “Unicorn” has been launched to foster innovation. By utilizing data and environments duplicated from Phoenix, engineers have established a distinct, separate database. This allows them to continuously develop and test features such as customer-targeted promotions without impacting the active application.

The second component involves embracing failure. To address this, Parts introduced their “Simian Army Chaos Monkey” project, which was designed to wreak havoc by generating catastrophic bugs that could terminate processes or even entire servers. Initially, this led to chaos, with the testing infrastructure frequently crashing. However, as the Development and IT Operations teams worked together to fortify the code and infrastructure, the IT services grew resilient and immune to failures.

The third aspect focuses on prioritizing the enhancement of daily operations over the routine tasks themselves. At Parts, every manager is required to make some form of improvement, no matter how small, every two weeks. These cycles, known as improvement kata, consistently apply pressure to the system, compelling it to evolve.

In martial arts, the concept of kata involves repeatedly practicing a set pattern until it becomes ingrained. It’s the development of habits that leads to expertise. Research indicates that practicing for five minutes every day is more effective than a longer session of three hours once a week, highlighting that frequent repetition improves proficiency.

In today’s tech-centric environment, IT is more than just a department; it’s a critical skill akin to reading or writing. Business managers must have this skill to strategically take risks and outperform competitors. When both business and IT understand their intrinsic connection and adopt DevOps principles and practices, the whole organization benefits.

Summary Overview

The essential takeaway from this book is:

In our current tech-centric world, the IT department is crucial to a company’s success. Disjointed interactions and workflows between Development and IT Operations can disrupt the entire organization. However, this turmoil isn’t inevitable. DevOps introduces a cohesive IT approach based on the “Three Ways” which focus on rapid feedback, effective communication, and fostering a culture of experimentation and continual improvement. By adopting these strategies, your IT department—and your company as a whole—can achieve remarkable levels of success.

Here’s another practical tip:

Set up your own kanban board. While kanban boards enhance productivity across organizations, they’re equally effective for personal use by helping you manage workload and track your progress. Replace your intricate to-do lists with a simple setup: a blank wall and some Post-it notes. Organize your tasks into three categories: Ready, Doing, and Done. Keep your work-in-progress limited to four or five items at a time, and start tackling them. As you advance, you’ll move the cards from left to right, potentially reaching your goals more swiftly than you thought possible.

Did you like the summary? You can buy the full book in Amazon and support us!