Abstract
People think that DevOps, Cloud Native, Agile, GROWS, etc. are all rainbows and roses. You start small, work your way up or you decide as an organizational unit to change. These are two patterns associated with a Jedi-type maturation process of DevOps. Like in Star Wars, DevOps has a dark side. This talk provides examples of successful and failed DevOps transformations as well as some lessons learned along the way.
Description
The Ops Mutiny
The Setup
- Former employer needed to change to stay competitive
- IT team that needed changing was resisting
- Development teams had started to speed up on flagship product
- There was a mutiny and the majority of the IT team left the company en masse
- No formal processes
The Moment
- The company had been acquired for a huge sum of cash.
- Architecture review was not for me, the new lead, it was for the company that had bought them.
- Faulty HVAC in office comm closet that housed a few production pieces.
- Windows NT 4.0 in 2015
- I started nmap’ing the network looking for obvious vulns
- Desktop support person said there was concern about the load I was putting on the network (from my laptop)
- Desktop support was a holdover from the old regime and was trying to hinder my investigation
The Straw That Broke the Camel’s Back
- Flagship product was moving to a cloud provider (containers, microservices, 12 Factor, etc.)
- Third party integrator was building new solution intending to hand it off to my team
- Team was actively resisting new things and third party presence grew
- I had no allies on the team
- Shortly before a major US holiday, I determined enough was enough
- After three weeks, I was putting in my two weeks notice
- Ruthless tactics to get me to stay
- Firing people in the name of DevOps before a holiday is not okay (I am human)
The Dev Rebellion
The Setup
- At another employer, I took part in a full blown DevOps rebellion against the monolithic IT department.
- DevOps was brought to bear within a small team of people as they worked on a new open source product.
- The product team was going to utilize a PaaS despite the IT organization stating “cloud” was not an option.
- The assumption from the IT staff and paper pushers was that this new product was deploying in-house.
- However, all the documentation clearly stated Heroku was the workload’s destination.
- Storage was the only thing staying in-house (because it was actually cheaper and compliant).
- IT department approval authorities signed off on utilizing cloud resources despite having said they never would.
- This proved their ITIL and ITSM death grip process was not meeting the desired objectives.
The Moment
- The team used Docker, Papertrail, Postgres, Slack
- It was truly exciting to prove safe, secure innovation could work with cloud resources
- The bluster and resistance of the IT leadership quickly changed to attempts to satisfy the development teams’ needs.
- A sponsored OpenStack instance as a developer playground was going to take a year or more.
- Tooling to enable API calls for IT resources was outsourced to a vendor with little feedback from our teams.
- IT did not “get it” but the team was working around the bottlenecks.
Victory
- The “no cloud” folks quickly became the “cloud is okay”
- They still tried to build a private cloud but the genie was out of the bottle
- Cloud was tolerable and they had to start detailing what it cost to run their infrastructure
Conclusion
- This is not to say Ops is bad and Devs are great.
- The Dev Rebellion had people at all levels pointing at successes of the team. The Ops Mutiny had a core group of people trying to cast off change regardless of outcomes.
- The DevOps journey is a combination of people, processes, and tooling.
- The culmination of these is a team of allies pushing the envelope and never resting on their laurels.
- There is no such thing as a completed DevOps transition.
- Resting on your laurels is not a DevOps mindset.
- DevOps and its allies should be iterating and improving upon what they have learned daily.