DevOps can be interesting: Just read “The Phoenix Project”

I recently finished reading The Phoenix Project, which was a great read. It’s a fictional story that teaches DevOps (the intersection of the worlds of software development and application operations), and is actually much more fun to read than you’d expect based on that topic. The story made it an easy read, while there was real meat — it was like going to training on DevOps best practices, including concepts from ITIL, Lean, and continuous integration (CI); and other recommendations on books and concepts to study further (like chatting with a mentor).

If your career is related to software development or deployment, I recommend it. Here are a few quotes that jumped off the page for me:

  • “Remember, it goes beyond reducing WIP [Work in Progress]. Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever.”
  • “Left unchecked, technical debt will ensure that the only work that gets done is unplanned work!”
  • “Unplanned work has another side effect. When you spend all your time firefighting, there’s little time or energy left for planning. When all you do is react, there’s not enough time to do the hard mental work of figuring out whether you can accept new work. So, more projects are crammed onto the plate, with fewer cycles available to each one, which means more bad multitasking, more escalations from poor code, which mean more shortcuts. As Bill said, ‘around and around we go.’ It’s the IT capacity death spiral.”
  • “Everyone knows that in manufacturing, as WIP increases, due-date performance goes down”
  • “Brent is a worker, not a work center,” I say again. “And I’m betting that Brent is probably a worker supporting way too many work centers. Which is why he’s a constraint.” “Now we’re getting somewhere!” Erik says, smiling. Gesturing broadly at the plant floor below, he says, “Imagine if twenty-five percent of all the work centers down there could only be operated by one person named Brent. What would happen to the flow of work?”
  • “The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful.”
  • “Studies have shown that practicing five minutes daily is better than practicing once a week for three hours. And if you want to create a genuine culture of improvement, you must create those habits”
  • “I’m experimenting with putting kanbans around our key resources. Any activities they work on must go through the kanban. Not by e-mail, instant message, telephone, or whatever. “If it’s not on the kanban board, it won’t get done,” she says. “And more importantly, if it is on the kanban board, it will get done quickly. You’d be amazed at how fast work is getting completed, because we’re limiting the work in process. Based on our experiments so far, I think we’re going to be able to predict lead times for work and get faster throughput than ever.”
  • “The wait time is the ‘percentage of time busy’ divided by the ‘percentage of time idle.’ In other words, if a resource is fifty percent busy, then it’s fifty percent idle. The wait time is fifty percent divided by fifty percent, so one unit of time. Let’s call it one hour. So, on average, our task would wait in the queue for one hour before it gets worked. “On the other hand, if a resource is ninety percent busy, the wait time is ‘ninety percent divided by ten percent’, or nine hours. In other words, our task would wait in queue nine times longer than if the resource were fifty percent idle.” I conclude, “So, for the Phoenix task, assuming we have seven handoffs, and that each of those resources is busy ninety percent of the time, the tasks would spend in queue a total of nine hours times the seven steps…”
  • “’I’ve learned that while the finance goals are important, they’re not the most important. Finance can hit all our objectives, and the company still can fail. After all, the best accounts receivables team on the planet can’t save us if we’re in the wrong market with the wrong product strategy with an R&D team that can’t deliver.’ Startled, I realize he’s talking about Erik’s First Way. He’s talking about systems thinking, always confirming that the entire organization achieves its goal, not just one part of it.”
  • “People think that just because IT doesn’t use motor oil and carry physical packages that it doesn’t need preventive maintenance,” Erik says, chuckling to himself. “That somehow, because the work and the cargo that IT carries are invisible, you just need to sprinkle more magic dust on the computers to get them running again. “Metaphors like oil changes help people make that connection. Preventive oil changes and vehicle maintenance policies are like preventive vendor patches and change management policies. By showing how IT risks jeopardize business performance measures, you can start making better business decisions.
  • “She created these SOX-404 control documents for the finance team. It shows the end-to-end information flow for the main business processes in each financially significant account. She documented where money or assets entered the system and traced it all the way to the general ledger. “This is pretty standard, but she took it one step further: She didn’t look at any of the IT systems until she understood exactly where in the process material errors could occur and where they would be detected. She found that most of the time, we would detect it in a manual reconciliation step where account balances and values from one source were compared to another, usually on a weekly basis. “When this happens,” he says, with awe and wonder in his voice, “she knew the upstream IT systems should be out of scope of the audit.” “Here’s what she showed the auditors,” John says, excitedly flipping to the second page. “Quote: ‘The control being relied upon to detect material errors is the manual reconciliation step, not in the upstream IT systems.’
  • “He saw a presentation given by John Allspaw and his colleague Paul Hammond that flipped the world on its head. Allspaw and Hammond ran the IT Operations and Engineering groups at Flickr. Instead of fighting like cats and dogs, they talked about how they were working together to routinely do ten deploys a day! This is in a world when most IT organizations were mostly doing quarterly or annual deployments. Imagine that. He was doing deploys at a rate one thousand times faster than the previous state of the art.”
  • “It’s about continual experimentation, like Scott Cook did at Intuit, where they did over forty experiments during the peak tax filing season to figure out how to maximize customer conversion rates. During the peak tax filing season! “If you can’t out-experiment and beat your competitors in time to market and agility, you are sunk. Features are always a gamble. If you’re lucky, ten percent will get the desired benefits. So the faster you can get those features to market and test them, the better off you’ll be. Incidentally, you also pay back the business faster for the use of capital, which means the business starts making money faster, too.”

5 things I keep learning continually

I continue to see the concepts (said in many different ways) in my life and in what I’m reading and listening, so I thought I’d reflect on them in front of the internet:

  1. Objectives brick_wall– Define what’s actually important (goals, passions, values, etc.)
  2. Simplify – Remove things that aren’t important
  3. Automate – Setup recurring things to happen automatically through automation or delegation when you can
  4. Focus – Spend your energy on what’s important/critical/unique
  5. Adapt – Expect and embrace change

It’s amazing to me how these concepts, said so many different ways, are so valuable to everything we do, talked about so often, and consistently challenging to so many people.  Here are some examples: 

  1. Objectives – In careers, if you don’t define what’s important to you; you’ll end up bouncing around the professional world like a pinball from opportunity to opportunity without any passion (A great event recently talked about how “Passion follows purpose” meaning that it’s hard to get excited for something if you haven’t figured out what you actually want first).  Books like What Color is your Parachute are great for trying to figure this out.
  2. Simplify – It’s amazing how pervasive the concept is that people want to be well-rounded (it shows up in great work-life balance discussions, being well read across every topic, keeping our options open all the time) and yet no one seems to try to reconcile this with the fact that our culture doesn’t revere the most well-rounded people.  Quite the opposite — we don’t talk about Michael Jordan because he was a great all-around guy.  Or actors, or executives, or politicians. (see Seth Godin’s great point on this)  Someone we cram our lives with more and more things to be well-rounded, making ourselves increasingly stressed all while we get farther away from being an expert at anything.
  3. Automate – Personal finance can be stressful and exhausting if you’re checking your balances every month and constantly spending energy making decisions.  It can also be much simpler if you build a system that runs itself (see Ramit’s great post on this at  This automation sounds tough, but like so many great things; investing a little initially makes a huge difference long-term.  (And it’s certainly not limited to finances — try to automate everything you can, see
  4. Focus – It’s so easy to get distracted by things that aren’t important.  Think about the time management metaphor of the rocks, pebbles, and sand.  Or how everyone complains about responding (reacting) to email instead of “actually doing their job.”  Or the quadrant of important vs. non-important and urgent vs. not urgent.  All of these relate to the concept that we should spend energy on what’s actually important to us (see #1); though it’s entirely too easy to just react instead of lead our own life’s focus.  Be intentional on what you spend your time on, like Jim Collins allocating his time on his whiteboard.
  5. Adapt – The technology industry likes to talk about Agile all the time these days — the desire to be nimble and flexible.  We’re even talking about in the context of raising a family (see this WSJ article).  This sounds like a new concept, but the idea of adaptability isn’t new; but it’s certainly hard.  Make time in your schedule to reflect (think Agile retrospectives) and make changes (tune your life).

You should be wrestling with some of these all the time.

Agile discussion on Halfaker’s Blog

Below is a recent post from:

A Focused and Collaborative Approach Produces Powerful Results for Clients: CTO, Mike King, Discusses the Agile Framework

November 01, 2012

As Halfaker and Associates Chief Technology Officer, Mike King’s business growth leadership has focused on delivering tailored technologies and methodologies to Government clients who seek rapid, low-risk approaches to solve their organization’s challenges. Due to the dynamic nature of government contracting and the turbulent economic climate, Mike has introduced and championed the Agile framework in all aspects of Halfaker’s operations. The Agile approach to project management and software development activities have provided Halfaker an effective way to align closely with Government clients seeking flexibility and reduced risk as they pursue mission success.

Join us in taking a closer look at the Agile Framework with Halfaker’s CTO, Mike King:

Mike, can you describe the Agile framework for those not familiar with the process?

The Agile framework focuses on several key tenets. Two of these tenets include:

  • Increasing how fast solutions get into the hands of our clients through two to four week timed iterations that force prioritization, focus people’s energy, and produce incremental work products, and
  • Dramatically reducing risk by continually collaborating instead of relying solely on scope documentation to guide our priorities.

Furthermore, there are many methodologies that fall under the umbrella of Agile. These methodologies define some great tools including: project charters, estimating techniques, product backlog, meeting rhythms, test-driven development, burndown charts, and velocity graphs. We have used the Scrum Methodology on several projects and Kanban on internal projects. For more information on Scrum, check out this ten minute introductory YouTube video:

How does Halfaker use Agile concepts to improve how the company operates?

At Halfaker, we have found the greatest value of Agile in its ability to simultaneously increase the quality of the services we deliver and lower the risk to both Halfaker and our clients in delivering services. In addition, we are maturing our internal business processes by building on best practices from Agile and other process-driven frameworks such as CMMI and Lean Six Sigma. This process has revealed that each framework has strengths, weaknesses, and conflicts with each other. However, by building Halfaker’s business process architecture around our clients, we are able to reconcile these conflicting frameworks and continually improve how our clients experience Halfaker.

What Halfaker project has been most distinguished by this process? 

I’m very proud of the work we’ve done on VetGovPartner (VGP). We created a powerful web application platform that supports the small business Government contracting community at the intersection of social networking, market research, and conference experience. The tool allowed over 4,000 National Veteran Small Business Conference attendees to build their agenda from among the hundreds of conference sessions and navigate the chaos of such a large event. In addition, VGP allowed the Government acquisition, small business, and large business communities to connect with each other through the social layer of VGP. This was a powerful addition to the conference experience–it brought order to a large agenda and offered a powerful social layer that people used to connect effectively with each other.

Is the Agile framework commonly used across the industry?

Agile is an increasingly common buzzword that people throw around, but finding ways to apply these concepts effectively is still an immature process. Agile depends on teams working effectively both internally and with their clients. The stereotypes of the Government contracting industry do not align well with Agile teams—this framework breaks down when you don’t have a collaborative client relationship and a team that is able and encouraged to self-organize and attack challenges. We have found that there are great opportunities for organizations to use Agile for powerful results, but it’s a challenging undertaking.

What are Halfaker’s long-term plans with Agile?    

One of Agile’s principles is iterative refinement; we embrace that as a way to constantly improve Halfaker as an organization. Because Agile is a mindset and not a set of specific methodologies, we will continue to use it to improve our organization. My vision for Halfaker is that we will continue to grow closer to the ideals of a purely Agile organization that effectively prioritizes its clients over other priorities.

Mike King is a 2012 NVTC/Washington Technology Government Contractor CTO Innovator Awards Finalist. The CTO Innovator Awards honor chief technology officers within the region’s government contracting community for their critical contributions to achieving results for their customers and their leadership within their own companies.

Initial reactions to Lean Startup Machine DC

This weekend I experienced the intensity that was Lean Startup Machine DC (#LSMDC).  I went with without lots of expectations — I haven’t read The Lean Startup book, I haven’t been to any similar events, and I didn’t know anyone who had.  It turned out to be an intense, great experience.  The highlight for me was meeting all kinds of fascinating people thinking about or already doing cool stuff in the DC startup/entrepreneurship scene. My initial reaction to lean, with has some significant overlap with agile, is focused on quickly validating business models through real tests of your assumptions.  Not thought experiments, not talking it over with you buddy; but real, tangible ways to test your assumptions by removing assumptions.  For example, an early idea of the team I was on was:

  • Assumption:  people would share job openings on their social networks if they would get paid for helping getting someone hired
  • Realization:  people are protective of their social status, as part of their online identify, and only want to share opportunities that help a friend and/or are associated with a cool company

Much of the weekend was spent wrapping my head around using tools to validate or invalidate assumptions (using tools like Google Ad Words, Facebook ads, talking to domain experts (finding them using your network and social media), and pitching concepts).  In addition, there was a huge focus on collecting “currency” or tangible validation of your idea, which could include letters of intent from vendors, clicks on ads (see, email addresses on a launch page (see, initial payments, etc. The weekend started with people pitching ideas on Friday evening and forming teams and ended with a Sunday afternoon series of presentations of each group, focused on their process, what they learned, and their results.  In between, there were some great local people sharing their expertise in entrepreneurship and lean concepts; as well as a team of mentors around to help guide people through the process. Some highlights included:

  • Peter Corbett talking about iStrategyLabs
  • Discussions with Teague Hopkins about the future of the office of CIO
  • Maksim Tsvetovat discussing how social networks grow
  • Stephanie Hay’s discussions on initial tests instead of getting stuck in talking about ideas
  • Great ideas from Chitra Sivanandam about focusing on pain points (and hearing about some of the super-cool stuff she does)
  • Hearing Brian Sowards’ story of how he used lean to iterate a weak idea into a multi-million dollar valuation

Don’t go to a meeting empty-handed

Meetings can be horrible, horrible events.  My least favorite part is when a new work product (document, spreadsheet, website, etc.) needs to be created, and no one has taken the initiative to create the first draft.  This first draft seems so overwhelming to create initially, sometimes because there’s a lack of clear vision or direction; sometimes because creating the first of something is daunting for fear of doing it wrong; and often because we’re lazy.  However, there’s so much value in someone stepping up and making something – a shell, an outline, a sketch, a “scarecrow”; whatever you want to call it, to get the ball rolling.  So next time you’re in a discussion with a vacuum of tangible content, grab a blank sheet of paper, an empty PowerPoint slide, or a whiteboard and make something.

There is a risk that the person with the initiative can unintentionally become the owner of the artifact.  But owners are often (or at least should be) rewarded in organizations for stepping up and creating change instead of just talking about it.

Tools to build Web Apps without Coding

I’m not even sure what you call it, but I’d like to find a open source or hosted (software as a service) service solution to allow me to create simple web applications, primarily related to simple database and workflow systems. There seem to be several services in this space, with a variety of positions in the market.

I’m excited to tinker with these and see what I can find that works well — any suggestions?

Ideas at the Intersection of Agile, Startup scaling, Business Process Management, and Organizational Design