Talk notes: Changing Culture To Enable DevOps (DevOps Day)

DevOps DaySanta Clara, CA
June 25, 2010

I'm here to participate on a panel called "DevOps Outside of Web Operations."

Keynote:

Excellent video of DevOps background.  Showed DevOps problem statements and outlines some of the principles through Charlie Chaplain movie clips by Patrick DuBois. Well done.

Changing Culture To Enable DevOps

John Allspaw, Etsy
Israel Gat (Agile Executive)
Lee Thompson (DTO Solutions)
Lloyd Taylor (NetElder Associates)

Moderator: Andrew Shafer (Cloudscaling)

Introductions:

• Introductions
  • John Allspaw: Etsy. Formerly at Flickr. Has seen what is possible.
  • Lloyd Taylor: now at investor firm. VP Tech Ops: Formerly Google and LinkedIn.
  • Israel Gat (Agile Executive): now consultant. formerly with MOM. Now has Agile bug.
  • Lee Thompson (NetElder Associates): has experience in financial vertical.
• Question: lots of separation of between Dev and Ops tribe, with different goals and measurements.  Please comment. (everything below are quotes)
  • Israel: "governance is so complex. blah, blah..."
  • Lee: "in beginning, no one represents non-functional requirements, so things grow into a big mess. Agile Startup now brings this up. Not doing this early will lead to problems."
  • John: "the story often begins, "look at how bad things are." Shipping features late cost money. But so do outages. In successful organizations, developers will often set Nagios thresholds, and configurations settings are in the same place as the source code."
  • Lloyd: "grounding everything in control theory. The goal of a business is to make money, so everything should subordinate and support that goal. When personalities cause problems especially trust issues between engineering and development, things tend to get worse. Things will reach a level of stability, but at a very low level of performance. That's why I think the tribe issue is so important. If Engineering and Ops hang out each other and have lots of cross-socialization. Hard to demonize people you hang out with. When you spend time together, you defeat the stove-piping. Both ship products, but one is much more efficient, and probably makes more money."
    • A great book on this is "Great Boss, Dead Boss" which is all about tribes.  The author is a VP at a solar panel manufacturing, and is a Theory of Constraints Jonah (like me).
• Question: Conway law: quality of product will be a reflection of the culture: Please reflect.
  • Lee: "Look at cross-incentives. Ops is measured on siteups. Dev is on new features. This is the core conflict. What's the best way to keep site up? No changes. Just having faces to each organization makes it more difficult to say 'that moron.'  Google has a very good process, which has a chokepoint called 'machine allocation,' which is where they predict which services has the most economic potential."
  • Israel: "Is culture in IT similar than in revenue recognition. Try to assess what culture exists in Development and in IT operations. If they're different, that's interesting.
• Question from audience: "Many people moving QA out of Dev. Thoughts?"
  • John: "QA exists, even if it doesn't exist organizationally."
  • Lloyd: "Facebook doesn't have QA group." 
    • My new buddy Chris Westin tells me that in the Velocity Conference talk yesterday, "Developers do their own pushes. So obviously they're doing testing."
  • Lee: "1) where is the pain: If developers suffer when deployments fail, you must ensure that those who cause the issue also bear the pain.  Pain driven development. 2) you have to look at how difficult it is to make a mistake: if APIs are poorly documented, then it leads to fragility. These are part of non-functional requirements. 3) Facebook does this well: how fast do you detect screwups?"
• Question: "Global community of practice: many people are working on this problem, so how can we leverage each other?"
  • John: "30% of you write Internet facing code? That's the development side. Then the other community is IT ops.
• Question: "Why DevOps now?"
  • Lloyd: "More simple apps like Rails. And with cloud, you can't do things the old way. That's why I'm trying to create DevOps Tool Chain."
• Question: "What happens when you just don't have time to spend with your own kids, let alone learn about Mary's kids (your peers in the other silo?)"
  • John: "Ask where is the pain?"
  • Israel: "Scrum attendance is a good indicator to tell when the other group is too busy to even show up to the daily scrum meetings." (Yes, this is very effective.)
• Question: "How about in global distributed orgs, when you never see pictures of the kids of Toronto staff?"
  • John: "That's hard."
  • Lloyd: "Lesson after 30 years in DoD is that you have to be physically present. Maybe Facebook generation will change that."
• Question: "As developer, how do I get IT ops to trust us? They're busy."