Talk notes: Ignite Talks #2 (#DevOps Day)

DevOps Day, Santa Clara, CA
June 25, 2010

I'm here to participate on a panel called "DevOps Outside of Web Operations."

Ignite Talks #2

5 minutes, all slides advance every 30 seconds

I wasn't able to catch everyone's attributions.

• Netomata: like puppet for network equipment
• Petascale storage: DIY! (@tlossen: http://openstoragepod.org)
  • open source hardware project for scaleable storage
  • software by training, but find hardware by interest
  • 988 exabytes: amount of data that will be produced in 2010
  • Cost of a petabyte (see pic)
  • Use laptop disks
  • Wow, ATX style motherboard, and stack 12TB on top of it in one enclosure
  • Only needs 600W
  • Put them into pods of racks: 1.2 petabytes (6 KW of power)
  • (This is hilarious...)
  • Okay, let's scale it down so normal people might find it useful.  So you can put it in your living room.
• Systems thinking & Value stream mapping: 
  • talking about tools he brings into large engagements
  • MIT sloan beer game: retailer, wholesaler, distributor, factory
    • talking about the overordering happens, resulting in huge backlogs and then huge excess.  stable system turns very unstable.
  • software production pipeline
  • continuous integration, functional testing, UAT, staging, production
  • stumbled upon lean value stream mapping: optimize flow (concept into cash)
  • model current state: collect metrics: p/t, l/t, c&a (time wasted, production issues raised)
    • note that it crosses organizational boundaries
  • envision future state
• cloud supporting tools: http://libcloud.org
  • cloud service providers all have different APIs
  • we now provide/support 16 different cloud service providers
  • APIs
    • list_nodes(), reboot_note(), destroy_node(), ...
  • Add data: location, price-per-hour
  • deployment steps: mercury for drupal
  • want to go beyond just booting machines
  • it's open source: in Apache incubator
  • related projects
    • jclouds (author is here)
    • Apache Deltacloud
• A simple story of how one company spent 10M dollars and saved nothing
  • called the "zero effect"
  • the victim: Gloria (like a movie script)
  • the hero: Daryl Zero
  • the villain: Gregory Start
  • the company: High's (haha)
  • the vendor: Rivoli (haha)
  • story begins: $8.9M won out of $10M RFP
  • the analysis: $20K
    • Windows: "we've already got one" (Sponge Bob)
    • AIX dudes: "too busy, won't call back" (Teenage Ninja Turtles)
    • Oracle guys: "you are not worthy"
    • Security guys: "you want freaking root"
    • Linux folks: "hello, you guys should check out Nagios"
    • HP dudes: "they were interested" ("they liked us because no one cared about them")
    • Bob the CIO: "I want my damned dashboard"
  • The mothership steps in and decides to do end-to-end operations analysis
  • final cost: $10M
  • two years later, they switched to BMC.
  • Now that's a tragic story!
• Erik Sowa, Lyris: Latent Code: Lessons Learned Implementing Feature Bits (@eriksowa)
  • latent code: stuff not activated, stuck in "concept to cash" cycle
  • no big bangs: at mercy of lowest performing team
  • the deployment pipeline
  • Inspiration
    • John Allsaw and Paul Hammond video
    • Jez Humble and David Farley book
  • technology allows prod manager to turn it on
    • feature flags from Flickr and Twitter
  • lessons learned
    • design pressure is good: need loosely coupled and tightly cohesive
    • manage the lifecycle: retire aggressively
    • maintain production quality: code hidden behind feature bits is subject to same requirements as production code
    • default state: decouple code rolls
    • naming convention matters
    • do not overload: for latent code, not for controlling who has gold/silver feature
    • limit the overhead (dependencies)
    • customer facing releases
    • beta- and split-testing: you can use this to do customer testing, turning on for particular group of users