Over 10 years ago, I was a senior Linux engineer working in a cross-functional team with a number of Java developers, some QA folks, a network engineer and another Linux engineer.
We were working in a SCRUM like approach, we were automating the build, bootstrap and upgrade processes of all the components we were engineering together during sprints. It was hard work, but we were collaborating on a platform together.
One of the really great things was that we made much faster progress than the other teams I had worked in before. We were iterating really quickly, adding functionality bits by bits while keeping the product up and running. We didn’t have a name for it yet but we knew that the way we were working was beneficial for the company and that we were also improving ourselves at the same time. We grew into a tight-nit, cross functional team that had a very clear mission: delivering value for the business.
The Start of the DevOps Movement
In 2009, we ran a conference that brought together experienced folks from both the Open Source and Agile movement with early cloud adopters.
Patrick Debois named that conference “devOps” days, inspired by a Velocity talk from John Allspaw: “dev and ops, 10 deploys a day”.
The first devOps Days edition, Ghent 2009, © Krys Buytaert
We never set out to create a global movement: we wanted to learn from each other how we could improve software delivery, not only for ourselves, but also for others in the industry. Without knowing it we wanted to create a community of practice.
A lot of the best practices we already followed came back: using Open Source, Agile Software development, automated testing, continuous delivery, infrastructure as code, improved monitoring and metrics, but mainly the fact that we worked together as one team, developers and ops, collaboration was the starting point.
Thus “devOps” was born. The conference was a hit, people liked what we talked about and after a couple of years of hosting one conference a year on each side of the Atlantic things accelerated, every large city wanted to have its own event. London did two, Amsterdam, Berlin, Paris and also many more American cities started their own events. devOpsDays was on a roll and grassroots communities all over the world were sharing their experiences on how to build better software faster.
The DevOps Culture
What in Europe was pushed mostly by Ops who were tired of being on call for -pardon the language- ‘crappy’ software, was in the valley pushed by developers who wanted more agile operations, pushing the Ops to spin up stacks faster and deploy new features more often.
In the middle it was the same set of discussions. Both Patrick and I had our experience in larger organizations, certainly not the typical smaller startups such as the ones that ended up being the poster children of our movement.
“Look at what those startups do, Look at how lean they are” was the feedback we got from the larger organizations, the banks, the governments.
In successful organizations, teams work together with a common goal: the business.
It is true that in the early years we could take a 100-person IT organization, guide them from chaotic manual ad-hoc operations and low quality artifact delivery to monitored automated infrastructure and continuous delivery of the applications they build in 3 to 6 months time, because they were already using Open Source by default and they could tear down the silos they were about to build. It wasn’t that trivial for the larger organizations.
Successful Engineering Teams
In successful organizations, teams work together with a common goal: the business. They are capable of designing, building, deploying and running a component of software that delivers value to either their end users, or their peers in the organization.
They have introduced a level of automation that allows them to implement changes in their code base and push them to production in a controlled way, leveraging testing best practices, often in a matter of minutes.
They define better metrics, and build better monitoring and orchestration tooling around their software so that they can understand how their software behaves under different workloads, and can quickly react when things start to behave differently than they expect. They operate as a team, taking responsibility for the entire life-cycle of the software, from dev to prod.
Ten years ago, organizations that had adopted an agile methodology, open-source and cloud practices moved fast. Sometimes they were startups, sometimes they were only smaller groups within larger organizations. But the common traits is that the team members collaborated and worked towards a common goal.
What they didn’t do however was to buy tools. Tools of the proprietary kind, tools from the vendors that were actually re-branding their legacy software and putting a nice #devOps stamp on it. Some of them indeed changed their toolset, others improved it, automated the manual parts, removed the friction, increased the auditability and security by having it all automated.
Then the 1000+ person IT organizations wanted to do the same, but their challenges were different. It takes both senior leadership and engineering teams to want to make this radical change, and all the people on the floor need to be retrained and need to unlearn to behave the way they have been taught to behave for years.
The first challenge—changing leadership and culture—is still the hardest one, as it takes people out of their comfort zone. They fear like when going through any other change that they are going to lose control.
Challenges for Large Organizations when adopting DevOps Practices
When a large organization tells us today they are adopting devOps practices, it’s always a challenge to figure out if they are actually doing something different than before, or if they just renamed all of their existing processes, and moved from pen and paper to using an issue tracker (an example, but one that is too often encountered).
One of the typical symptoms of this is when large corporations that are moving to the (private) cloud get stuck in lengthy resource request procedures, with multiple approval steps trough the entire management chain, as opposed to giving their developers direct access to the API they want to use.
Where smaller organizations are often agile and lean by nature, but with no specific process, larger organizations tend to be stuck in an incomplete forms of agile: structures such as SAFe that fundamentally are not much different from waterfall and base their recruitment tactics on certifications rather than experience.
And like Michael Brunton-Spall says: “At some point the antibodies kick in”.
The reverse effect is also true, like Yves Hanoulle wrote in his farewell post at one of his customers: “The experienced folks have a utopian view of the combination of all the great things they have seen in other organizations”.
Don’t get me wrong: lots of large organizations have high performing teams inside their organizations, but that doesn’t mean the organization as a whole is agile, practices continuous delivery, and has good monitoring and metrics. It also doesn’t mean these organizations have made no progress. But the larger an organization, the harder it is to turn around the ship.
A Journey of Continuous Improvement to Bring Business Value
It doesn’t really matter where an organization currently is… as long as they realize they are on a journey to continuously improve the way they work.
As we near 2019, the year in which we will hopefully celebrate the 10th anniversary of devOpsDays in Gent, the topic ‘Is #devOps dead’ obviously pops up in numerous conversations.
With the numerous vendors and recruiters abusing the word, #devOps became hollow for a group of people. People who have never seen it succeed and stopped believing in it, people who worked as devOps engineers doing manual work with enterprise devOps tools in devOps teams that were the additional silo between the developers, the release management, the QA, and the ops team in organizations that claimed to be agile.
But devOps is alive and kicking! There are more and more people joining the ranks of a community that realize that they are on a journey. Every year brings us new people to the numerous devOpsDays all over the world.
And when they are at devOpsDays they learn that it is not about the tools they buy, but it is about collaborating as teams with a common goal.
The force is strong in devOps. We are still improving, learning from each other and breaking down the silos. We are still trying to bring more empathy in our engineering teams, empathy for the users, empathy towards our fellow co-workers. We are still fighting the urge to put tools and processes ahead of humans. And finally we still need to keep our eyes on the ball, and deliver value to our businesses.
I hope to see you all in Gent in 2019 for the 10th year anniversary of DevOpsdays.