The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

“‘Absolutely,’ I say, nodding. ‘You want the IT systems to be reliable and available, and for the business to be able to depend upon them. You want disruptions to normal operations kept to an absolute minimum so that the business can focus on getting Phoenix done.’ Looking surprised, Steve nods. ‘Exactly. Yes, well put. Whatever you said, that’s exactly what I want.’”

Palmer’s summary of Masters’s request frames IT as a background process compared to the more valuable and relevant business departments. The key term to note, here, is Masters’s use of “whatever,” which reveals how little Masters cares about IT. Palmer essentially promises not to do a great job, but to stay out of the way as much as possible, while Masters could not care less about Palmer’s role so long as it does not draw attention to IT.

“Sarah is the SVP of Retail Operations, and she also works for Steve. She has an uncanny knack for blaming other people for her screwups, especially IT people. For years, she’s been able to escape any sort of real accountability. Although I’ve heard rumors that Steve is grooming her as his replacement, I’ve always discounted that as being totally impossible. I’m certain that Steve can’t be blind to her machinations.”

Moulton is the main antagonist of the novel, and here, Palmer outlines why he dislikes her. She blames IT for problems rather than taking responsibility for her contributions. It is important to note that the text’s depiction of Moulton’s character highlights some critical elements of misogyny in the modern workplace, such as the perception that women “cannot handle” responsibility or that they depend on “manipulation” to get ahead. Palmer is confident that Masters will not fall for Moulton’s tricks, suggesting his character is dedicated to hard work, compared to Moulton’s underhanded tactics.

“It’s amazing to me how handoffs between Development and IT Operations always get screwed up. But given the perpetual tribal warfare between the two groups, maybe I shouldn’t be surprised. I reply, ‘I get the picture. Look, make sure you dig into this Dev specification issue personally. We’ve got to get this nailed down—grab everyone involved, whether they’re in Dev or Ops, and lock them in a room until they come up with a written specification. Phoenix is so important, we can’t afford to screw this up.’”

Palmer’s analysis of Development and IT Operations aligns with his use of “tribal warfare,” in which “Dev” and “Ops” are critically opposed to one another. However, he makes the first step toward uniting them as “DevOps” by sending McKee to watch over Development. He is starting to understand that the departments can only succeed by working together, but he still sees this as a treaty between warring factions, rather than as cooperation within a single team.

“‘Normally, you would have months to prepare your response letter and execute your remediation plan,’ she continues, suddenly looking apologetic. ‘Unfortunately, the way the audit testing calendar worked out, we only have three weeks until the external auditors arrive. That’s regrettable. We’ll make sure to give it more time in the next audit cycle. But this time around, we require your response by…’ She looks at her calendar. ‘One week from Monday, at the very latest. Do you think you can make it?’”

The audit findings present a needed layer of “unplanned work” per Reid’s four types of work. From a narrative stance, the fact that the audit findings require a large amount of work in a small amount of time, concurrent with the Phoenix rollout, illustrates how IT is often buried in piles of tasks ranging from urgent to unimportant. Later, Palmer discovers that IT is almost entirely separated from the SOX-404 audit, making this unplanned work, which seemed so important at the moment, irrelevant.

“In another one of the endless Phoenix status meetings, I realize that the developers are even more behind than we feared. As Wes had predicted, more and more work is being deferred to the next release, including almost all of the testing. This means that we’ll be the ones finding the problems when they blow up in production, instead of the QA Department.”

Palmer and Davis’s discovery in the Phoenix status meetings sets up Reid’s later explanation of “technical debt”: the Development and IT Operations teams find themselves unable to execute the myriad tasks they agreed to complete for the Phoenix launch. Instead, these features and applications are delayed indefinitely, forming another bulk of unplanned work. As the narrative progresses, the teams will still need to execute these tasks, even as they continue to struggle with their projects, compounding their stress. This passage also sets up QA’s joining DevOps later on.

“He gestures broadly with both arms outstretched, ‘In the 1980s, this plant was the beneficiary of three incredible scientifically-grounded management movements. You’ve probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: WIP is the silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it, you can’t control WIP.’”

Reid’s analogy to the factory floor is a running motif text that allows him to introduce formal theories and concepts in business, manufacturing, and project management. Goldratt’s Theory of Constraints is especially critical to The Phoenix Project, formalizing the importance of isolating, elevating, and exploiting constraints like Brent. Reid also uses this avenue to introduce the concept that WIP is the same in manufacturing and IT, even though IT does not deal with a physical product.

“‘For these, we need to trust that the manager knows what he or she is doing. But I’d like Patty to verify that people have appropriately informed anyone they could affect, and gotten the “okay to proceed” from all of them.’ I think for a moment and say, ‘Take John’s tokenization application. Before that change request would even come to us, I would expect him to get the nod from the application and database owners, and also the business. If he’s done that, that’s good enough for me. I view our role as making sure that he’s dotted the i’s and crossed the t’s. At this level, I care more about the integrity of the process, not so much about the actual changes.’”

Palmer’s plan to deal with the overwhelming number of requested changes facing his team relies on a key distribution of responsibility. Rather than trying to integrate all the weight of changes into his own team, Palmer sees how changes can be diffused among other teams, such as IS. In the moment, this seems like part of the tribal warfare between teams, but in the larger narrative, it hints at the eventual merging of these teams into a unified, cohesive group. The success of Palmer and McKee’s change process decisions reinforces the importance of this cooperation.

“‘Wes, once again we’re getting bottlenecked by your people. Are there some personnel issues here that you need to be addressing?’ Wes turns bright red and is about to respond, when I quickly interject, ‘Kirsten, how many other tasks have been assigned to Brent?’ Kirsten quickly replies, ‘As of today, there are five outstanding tasks. Three were assigned last Wednesday, and two were assigned last Friday.’ ‘Okay, I’m on it,’ I say. ‘As soon as we’re done here, I’ll look into what’s going on. Expect a status report by noon today along with revised timelines for completion. I’ll let you know if we require anything.’”

Again, Brent emerges as the constraint across technology-related tasks at Parts Unlimited, but Palmer is already beginning to see how he can handle Brent as a resource. Fingle is upset about project deadlines passing, and Davis is upset at the accusation that he cannot organize his personnel, but Palmer sees through to the heart of the issue. Reid’s influence is starting to affect how Palmer addresses issues in his team and its projects. In addition, Palmer takes this chance to investigate for himself, showing his dedication to this project.

“William started sending back critical bug reports to the developers, many of whom had already gone home for the day. Chris had to call them back in, and William’s team had to wait for the developers to send them new versions. My team wasn’t just sitting around, twiddling our thumbs. Instead, we were frantically working with William’s team to try to get all of Phoenix to come up in the test environment. Because if they couldn’t get things running in a test environment, we wouldn’t have a prayer of being able to deploy and run it in production.”

Phoenix’s failure exposes the need for cooperation between Development, IT Operations, and QA, even though the characters have not yet figured this out. Mason, Allers, and Palmer struggle to work together, and the issues they face result from miscommunication and disorganization across teams. The developers fail to arrange their product sufficiently for testing, which causes QA to back up, which means IT Operations cannot put anything into production—all of which becomes critical to Project Unicorn’s successes and Project Phoenix’s failures.

“‘Look,’ I say, ‘I know that sometimes people think you’re not on our side, but I really need your help. You’ve got to keep the auditors off this floor. Maybe even out of this building. I’ll put up some curtains on the windows, or maybe even barricade the door.’ John looks at me and then nods. ‘Okay, I’ll handle the auditors. But I still don’t think you fully understand. As the custodians of cardholder data, we cannot allow hundreds of people to have access to it. The risk of theft and fraud is too high. We’ve got to destroy the data immediately.’”

This passage highlights two important elements of Pesche’s role in the novel. First, his job is truly important, and the data he protects the company from fraud. Second, Palmer, like the other department leaders, sees Pesche as an inconvenience, devaluing his work and minimizing his role to that of an interloper preventing normal business operations. Even though Pesche worked to prevent the disaster Palmer found in Ann’s war room, Palmer ignores that and sends Pesche to simply keep the auditors away.

“‘I haven’t seen you this stressed out since…’ She looks up for a moment, reviewing her memories. She looks back at me and says, ‘Ever! Half the time we’re driving in the car, you have this distant look on your face. The rest of the time you’re clenching your jaw, like you’re reenacting some terrible meeting in your head. You never hear what I’m saying, because you’re so preoccupied by work.’”

Paige is a niche character who serves a largely singular purpose: monitoring Palmer’s demeanor. Here, she notes how Palmer has never been as stressed about work as he is after the Phoenix failure, illustrating how his dedication to work is destroying his personal life. His wife misses him, and, though she understands his position, she laments his failure to balance work and life successfully. The purpose of this subplot is to show how disorganization impacts more than just the company—it also disrupts individual employees’ lives.

“‘Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever.’ He continues, ‘Remember, outcomes are what matter—not the process, not controls, or, for that matter, what work you complete.’”

Reid’s outline of the First Way focuses entirely on outcomes, which seems counterintuitive to Palmer at first. In trying to become more efficient, one would think that the place to start is with methods, processes, and controls. Instead, Reid argues that the work that gets done is less relevant than the speed and accuracy with which it is completed. If WIP is flowing smoothly and quickly, errors will be caught, quality improved, and critical tasks completed in a timely manner.

“‘There are two things I’ve learned in the last month. One is that IT matters. IT is not just a department that I can delegate away. IT is smack in the middle of every major company effort we have and is critical to almost every aspect of daily operations.’ He says, ‘I know that right now, nothing, absolutely nothing, is more important to the company’s success than how this leadership team performs. The second thing I’ve learned is that my actions have made almost all our IT problems worse. I turned down Chris and Bill’s requests for more budget, Bill’s request for more time to do Phoenix right, and micromanaged things when I wasn’t getting the results I wanted.’”

Masters’s shift from hindering to helping Palmer comes with one of the major messages of the novel. IT is critical to all business functions in the modern era, and Masters was not helping anyone by ignoring the IT Operations and Development leaders. Allers and Palmer, having direct contact and control over DevOps, need the freedom to direct their teams as required without Masters, who is distinctly separated from daily operations, hindering their progress. Even though Masters’s instinct is to micromanage, Allers and Palmer can provide results on their own.

“Well put, Bill. You’ve just described ‘technical debt’ that is not being paid down. It comes from taking shortcuts, which may make sense in the short-term. But like financial debt, the compounding interest costs grow over time. If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.”

Technical debt and unplanned work, though separate, operate similarly, and both hinder the normal operations of DevOps. As projects are rushed, developers and IT teams must take shortcuts and delay features, all of which must still be completed later on. As a result, the project might get done on time, but both teams will need to continue working on them, as unplanned work, until the debt is paid off. This pattern indefinitely delays any new projects or initiatives.

“‘Good,’ he says. ‘You’re standardizing Brent’s work so that other people can execute it. And because you’re finally getting those steps documented, you’re able to enforce some level of consistency and quality, as well. You’re not only reducing the number of work centers where Brent is required, you’re generating documentation that will enable you to automate some of them.’ He continues, ‘Incidentally, until you do this, no matter how many more Brents you hire, Brent will always remain your constraint. Anyone you hire will just end up standing around.’”

Part of Reid’s praise of Palmer’s plan to isolate and elevate Brent as a constraint is the necessity of documentation and automation. Davis noted previously that hiring new engineers takes time, and new hires rarely achieve productivity until they have been with the company for a year. Palmer’s decision to force Brent and others to document and automate their work means that new hires can actually learn what Brent does, rather than simply depending on him like everyone else. By elevating his constraint, Palmer also removes the constraint from some areas of work.

“This is important. Getting executives and workers the tools they need to do their jobs is one of our primary responsibilities. I’m not saying I don’t believe you, but let’s keep these time estimates to ourselves for now. If you can generate a week’s track record of hitting the dates, then let’s start publishing this to all the requesters and their managers, okay?”

McKee’s promises seem too good to be true, and Palmer’s skepticism reflects the difficulties his team and Development have had meeting deadlines. Palmer worries that the same pattern will emerge with McKee’s kanban boards. Nonetheless, the productivity of McKee’s plan is undeniable, and Palmer, too, is excited to see how their team is progressing.

“Suddenly, I recall my conversation with Wes right before Sarah and Chris decided to deploy Phoenix at Kirsten’s meeting. Wes complained about tickets related to Phoenix bouncing around for weeks, which delayed the deployment. It was happening then, too. That wasn’t a handoff between IT Operations people. That was a handoff between the Development and IT Operations organization, which is far more complex. Creating and prioritizing work inside a department is hard. Managing work among departments must be at least ten times more difficult.”

Palmer is starting to see the challenge of Reid’s First Way, in which it is insufficient to simply increase the productivity of IT Operations. Instead, Palmer needs to tackle the challenge of coordinating Development, QA, IT Operations, and IS, which requires overcoming the tribal mindset he has faced throughout the novel. Palmer’s note that it will be “ten times more difficult” serves to increase the sense of achievement when he accomplishes his goal.

“‘People think that just because it doesn’t use motor oil and carry physical packages that it doesn’t need preventive maintenance,’ Erik says, chuckling to himself. ‘That somehow, because the work and the cargo that it carries are invisible, you just need to sprinkle more magic dust on the computers to get them running again. Metaphors like oil changes help people make that connection. Preventive oil changes and vehicle maintenance policies are like preventive vendor patches and change management policies. By showing how its risks jeopardize business performance measures, you can start making better business decisions.’”

Throughout the novel, Reid compares IT to physical manufacturing practices, and this passage explains his reasoning. Most people, like Masters, see IT as a magical process that spontaneously resolves issues, while others, like Palmer, see IT as a unique, challenging field exempt from normal business practices. Reid cuts through these misconceptions with his comparison between updating software and getting an oil change, which highlights IT as a business component like any other, which needs to be maintained just like a factory.

“‘When R&D capital is locked up as WIP for more than a year, not returning cash back to the business, it becomes almost impossible to pay back the business,’ she continues. Holy crap. Maggie is starting to sound eerily like Erik, too. The need to continually reduce cycle times is part of the First Way. The need for amplification of feedback loops, ideally from the customer, is part of the Second Way.”

Reflecting the Theory of Constraints, Lee notes how R&D, though wildly successful, cannot function because it operates in front of a bottleneck: IT Operations and Development. Without the reporting and systems that IT Operations maintains, R&D can continually produce better products that never get to market because of the gap between Development, IT Operations, and Marketing. Palmer sees how his team can shorten this gap, allowing a greater flow of WIP and granting faster turnaround on feedback for even greater improvements.

“Our project backlog has been cut way down, partially from eradicating dumb projects from our queue. And John has delivered. We’ve cut a bunch of unneeded security projects from our audit preparation and remediation work, replacing them with preventive security projects that my entire team is helping with. By modifying our development and deployment processes, we’re hardening and securing both the applications and production infrastructure in a meaningful and systematic way. And we’re gaining confidence that those defects will never happen again in the future.”

The two elements highlighted in this passage are the elimination of irrelevant work and the refining of critical tasks. On one hand, some projects and tasks clog up the flow of WIP, including some that apply to outdated or unimportant systems. This is not work according to Reid’s four types of work. At the same time, though minor improvements are also not work in Reid’s model, they are useful because they allow real work to be completed faster and with greater accuracy. By clearing out useless work and refining tasks, Palmer is rapidly improving the efficiency of his budding DevOps team.

“Now you must prove that you can master the Second Way, creating constant feedback loops from IT Operations back into Development, designing quality into the product at the earliest stages. To do that, you can’t have nine-month-long releases. You need much faster feedback. You’ll never hit the target you’re aiming at if you can fire the cannon only once every nine months. Stop thinking about Civil War era cannons. Think antiaircraft guns.”

Reid’s challenge to DevOps is intense, increasing from one deployment every three weeks to 10 deployments per day. However, his comparison to weaponry serves to show how outdated Allers and Palmer’s prior schedule was. Reid’s proposition is not that DevOps needs to constantly produce new things, but that it can only respond to the market and business needs if it can produce quickly.

“‘If we had a common build procedure, and everyone used these tools to create their environments, the developers would actually be writing code in an environment that at least resembles the production environment. That alone would be a huge improvement.’ He takes the marker cap out of his mouth. ‘To build the Phoenix environment, we use a bunch of scripts that we’ve written. With a bit of documentation and cleanup, I bet we could cobble together something usable in a couple of days.’”

Part of Palmer’s plan to integrate DevOps involves automation, which, in turn, reduces the chance of miscommunication. Throughout the novel, a major issue has been configuring environments between Development, QA, IT Operations, and IS, since each step of production requires successful testing and execution. By automating the environment, Palmer eliminates the possibility of failure and forces Development to operate in the same environment as and with the assistance of the other three groups.

“Ironically, one of the developers suggested turning off all the real-time recommendations, which we had worked so hard to build. Why recommend more products to buy, he argued, if customers can’t even complete a transaction? Maggie quickly agreed, but it still took the developers two hours to change and deploy. Now, this feature can be disabled with a configuration setting, so we can do it in minutes next time around, instead of requiring a full code rollout. Now that’s what I call designing for IT Operations! It’s getting easier and easier to manage the code in production.”

Though it seems counterintuitive, the Unicorn team has to turn off one of their most successful features because it is, in effect, too successful. Their reporting and recommendations features have made Marketing so effective that they are running out of inventory. The fact that Unicorn figures out how to shut off this feature and sets up a configuration that allows it to be turned on and off quickly shows the development of Palmer’s team, which formerly could not even release a feature promptly.