On Value Stream Management in DevOps, and Seeing Problems as Solutions

You know that thing when a term emerges and you kind of get what it means, but you think you’d better read up on it to be sure? Well, so it was for me with Value Stream Management, as applied to agile development in general and DevOps in particular. So, I have done some reading up.

Typically, there seems to be some disagreement around what the phrase means. A cursory Google of “value stream DevOps” suggests that “value stream mapping” is the term du jour, however a debate continues on the difference between “value stream mapping” (ostensibly around removing waste in lean best practices) and “value streams” – for once, and for illustration purposes only, I refer to Wikipedia: “While named similarly, Lean value stream mapping is a process-based practice that seeks to identify waste, whereas value streams provide a higher-level overview of how a stakeholder receives value.”

Value streams are also seen (for example in this 2014 paper) as different to (business) processes, in that they are about making sure value is added, versus being about how things are done. This differentiation may help business architects, who (by nature) like precision in their terminology. However the paper also references Hammer & Champy’s 1993 definition of a process, which specifically mentions value: “Process is a technical term with a precise definition: an organized group of related activities that together create a result of value to the customer.” Surely a process without value is no process at all?

Meanwhile analysts such as Forrester have settled on Value Stream Management, which they reference as “an emerging market” even though at least some of the above has been around for a hundred years or so. Perhaps none of the terminological debate matters, at least to the people trying to do things with whatever the term means. Which is what, precisely? The answer lies in the restating of a problem as a solution: if value stream management is the answer, the challenge comes from a recognition that things are not working as well as they could be, and therefore are not delivering value as a result.

In the specific instance of DevOps, VSM can be seen as a direct response to the challenge of DevOps Friction, which I write about in this report. So, how does the pain manifest itself? The answer is twofold. For people and organisations who are already competent at DevOps, particularly those cloud-native organisations who are DevOps-by-default (and might wonder what other approach might exist), the challenge is knowing whether specific iterations, sprints and releases are of maximum benefit, delivering something of use as efficiently as possible.

In this instance, the discipline of value stream management acts as Zen master, asking why things are as they are and whether they can be improved. Meanwhile the ‘emerging market’ of VSM refers to tooling which smooths and simplifies development and operational workflows, enabling the discipline to be implemented and hopefully maximising value as a result. Which gives us another “problem-as-solution” flip — while many of the tools available today are API-based, enabling their integration into workflows, they have not always been built with end-to-end value delivery in mind.

A second group feeling the pain concerns organisations that see DevOps as an answer, but are yet to harness it in a meaningful way beyond individual initiatives — many traditional enterprises tend to fall into this category, and we’ve held various webinars about helping organisations scale their DevOps efforts. For these groups, value stream management offers an entry point: it suggests where effort should be focused, not as DevOps as an end in itself but as a means for delivering increased, measurable value out of software.

In addition, it creates a way of thinking about DevOps as practical workflows, enabled by automation tools, as opposed to ‘just’ a set of philosophical constructs. The latter are fine, but without some kind of guidance, organisations can be left with a range of tooling options but no clear idea about how to make sure they are delivering. It’s for this reason that I was quite keen on GitHub’s announcement around actions, a couple of weeks ago: standardisation, around not just principles, but also processes and tools, is key to efficiency.

The bottom line is that, whatever the terminology, we are moving away from thinking that ‘DevOps is the answer’ and towards ‘implementing the right kind of DevOps processes, with the right tools, to deliver higher levels of value’. Whether about principles or tooling, value stream management can therefore be filed in the category of concepts that, when they are working right, they cease to exist. Perhaps this will become true in the future but right now, we are a long way from that point.

Afternoon: If you want to read up on the notions of value management as applied to business outcomes, I can recommend this book by my old consulting colleague Roger Davies.

Five questions for: Mike Burrows of Agendashift

My travels around the landscape of DevOps brought me to Mike Burrows, and the work he was doing around what he terms Agendashift, an outcome-based approach to continuous transformation. While these words could be off-putting, I was more intrigued by the fact that Mike had set up a Slack site to articulate, test and improve his experience-based models – as he says, there’s 500 people on the site now, and as I have experienced, it’s very participative. So, what’s it all about – is there life beyond prescriptive lean and agile approaches? I sat down with Mike (in the virtual sense) to find out the background of, and hopes and dreams for, Agendashift.
1. What led you to write a book about lean/agile/Kanban — what was being missed?
Good question! I’m one of those people that laments the rise of prescription in the Lean-Agile space, and though I found it easy to find people who were in sympathy with my view, I didn’t find a lot of constructive alternatives. I myself had developed a consistent approach, but calling it “non-prescriptive” only told people what it wasn’t, not what it was! Eventually, I (or perhaps I should say “we”, because I had collaborators and a growing community by this time) landed on “outcome-oriented”, and suddenly everything became a lot clearer.
2. How would you explain Agendashift in terms a layperson might understand?
The central idea is principle #2 (of 5 – see agendashift.com/principles): Agree on outcomes. It seems kinda obvious that change will be vastly easier when you have agreement on outcomes, but most of us don’t have the tools to identify, explore, and agree on outcomes, so instead we jump to solutions, justify them, implement them over other people’s resistance, and so on. I believe that as an industry we need to move away from that 20th century model of change management, and that for Agile it is absolutely essential.
Around that central idea, we have 5 chapters modelled on the 5 sessions of our workshops, namely Discovery (establishing a sense of where we are and where we’d like to get to), Exploration (going down a level of detail, getting a better sense of the overall terrain and where the opportunities lie), Mapping (visualising it all), Elaboration (framing and developing our ideas), and Operation (treating change as real work). Everything from a corporate ambition to the potential impact of an experiment is an outcome, and we can connect the dots between them..
3. You went through an interesting development process, care to elucidate?
Two key ingredients for Agendashift are to be found in the last chapter of my first book, Kanban from the Inside (2014). The first is the idea of “keeping the agenda for change visible”, a clue to where the name “Agendashift” came from, and worthwhile to develop further how one might populate and visualise such a thing (and I took inspiration not just from Kanban, but also from Story Mapping). The second was the kind of bullet point checklist you see at the end of a lot of books.
I and a few others independently around the world (Matt Phillip most notably) realised that we had the basis for an interesting kind of assessment tool here, organised by the values of transparency, balance, collaboration and so on (the values model that was the basis for my book). In collaboration with Dragan Jojic we went through several significant iterations, broadening the assessment’s scope, removing jargon, eliminating any sense of prescription, and so on. We found that the more we did that, the more accessible it became (we now have experience using it outside of IT), and yet also more thought-provoking. Interesting!
Other collaborators – most notably Karl Scotland and Andrea Chiou – helped move Agendashift upstream into what we call Discovery, making sure than when we come to debriefing the assessment that we’re already well grounded in business context and objectives. The unexpected special ingredients there has been Clean Language (new to me at the time, and a great way to explore outcomes) and Cynefin (already very familiar to me as model, but now also very practical once we had the means to create lots of fragments of narrative, outcomes in Agendashift’s case).
4. Who is the Agendashift book aimed at, is it appropriate for newcomers, journeymen or masters?
I do aim in my writing for “something for everyone”. I accept though that the complete newcomer to Lean-Agile or to coaching and facilitation may find that it assumes just a bit too much knowledge on the part of the reader. My third book (working title “Right to Left: The digital leader’s guide to Lean-Agile”, due 2019) will I think have the broadest possible appeal for books in this space. We’ll see!
5. How do you see things progressing – is nirvana round the corner or is that the wrong way to think about it?
We’re coming up to the 2 year anniversary of the public launch of the Agendashift partner programme, 2 years into what I’m told is likely a 3-year bootstrap process (I have some fantastic collaborators but no external investment). General interest is definitely growing – more than 500 people in the Agendashift Slack for example – and I’m seeing a significant uptick in demand for private workshops, either directly from corporates or via partner companies. Its potential as a component of leadership development and strategy deployment is gaining recognition too, so we’re not dependent only on Agile transformation opportunities. I believe that there is potential for Agendashift in the digital and DevOps spaces too.
There is a lot of vested interest in imposed Agile, and in all honesty I don’t see that changing overnight – in fact I tell people that I can see the rest of my career (I’m 53) being devoted to outcomes. Over time though, I believe that we will see more success for transformations that are based on genuine engagement, which can only be good for the likes of Agendashift, OpenSpace Agility, and so on. Eventually, the incongruity of imposed Agile will be exposed, and nirvana will be achieved 🙂
 
My take: Not the weapon, but the hand
I’m all for methodologies. Of course, I would say that – I used to run a methodology group, I trained people in better software delivery and so on. From an early stage in my career however, I learned that it is not enough to follow any set of practices verbatim: sooner or later (as I did), edge cases or a changing world will cause you to come unstuck, which goes a long way to explain why best practices seem to be in a repeated state of reinvention.
I was also lucky enough to have some fantastic mentors. Notably Barry McGibbon, who had written books about OO, and Robin Bloor, whose background was in data. Both taught me, in different ways, that all important lesson we can get from Monty Python’s Holy Grail: “It’s only a model.”
Models exist to provide a facade of simplicity, which can be an enormous boon in this complex, constantly changing age. At the same time however, they are not a thing in themselves; rather, they offer a representation. As such, it is important to understand where and when they are most suited, but also how they were created, because, quite simply, sometimes it may be quicker to create a new one than use something ill-suited for the job.
And so it is for approaches and methods, steps we work through to get a job done. Often they are right, sometimes less so. A while back, myself, Barry and others worked with Adam and Tim at DevelopmentProcess to devise a dashboard tool for developers. So many options existed, the thought of creating something generic seemed insurmountable…
… until the epiphany came, that is: while all processes require the same types of steps, their exact form, and how they were strung together, could vary. This was more than just a, “Aha! That’s how they look!” as it also put the onus onto the process creator to decide which types of step were required, in which order.
Because of this, among many other reasons, I think Mike is on to something. In another recent conversation, Tony Christensen, DevOps lead at RBS, said the goal had become to create a learning organisation, rather then transforming into some nirvanic state. True Nirvana, in this context at least, is about understanding the mechanisms available, and having the wherewithal to choose between them.
 
Image: Agendashift
 

Agile DevOps: A Path to the Common Ground of Productivity

 
Agility has become the buzz word around the enterprise. whether it is agility around storage, networking, cloud operations, or most any other IT service is not really the point here, it all comes down to agility as an ideology.
Take for example the burgeoning data analytics market, which is driven by big data and business intelligence, where implementing agile ideologies could be the secret to success. After all, an agile business needs to be able to react to trends and discoveries to remain competitive, and waiting on analytics does not bode well for those looking to make intelligent decisions as quickly as possible.
In other words, best of breed analytics solutions must bridge the gap between data science and production to unify development and deployment into an agile methodology. With that in mind, Florian Douetteau, CEO of Dataiku, has put together an interesting guidebook that discusses how to achieve that level of synergy to build a data project that embodies the ideologies of agility.
Douetteau has identified the key strategies that illustrate how to bring agility to a data science project, those strategies include adopting:
–              Consistent Packaging and Release
–              Continuous Retraining of Models
–              Multivariate Optimization
–              Functional Monitoring
–              Roll-Back Strategy
–              IT Environment Consistency
–              Failover Strategies
–              Auditability and Version Control
–              Performance and Scalability
Ultimately, the goal here is to bring agility to the data team, where a data science team and IT production can work hand in hand to deliver results in an agile fashion.
In an Interview with GigaOM, Douetteau offered additional advice, he said “One of the most valuable tips I can offer is that IT should provide a common platform, which gives users across the different groups access to the tools and technologies they are familiar with. Ideally, visual drag and drop tools for should be provided for less technical team members, while the ability to code, should be provided for advanced members. What’s more, monitoring, security options and role based administration tools should be made available to those responsible of deployments.”
Nonetheless, previous attempts to achieve the goal of agile decision making has been an almost impossible task, thanks to the silos surrounding data science development and the deployment of operational applications that can illustrate results.
Douetteau says “the biggest challenge of most data science projects is getting everyone on the same page in terms of business goals, technical requirements, project challenges, and responsibilities. More often than not, there is a disconnect between the worlds of development and production. Some teams may choose to re-code everything in an entirely different language while others may make changes to core elements, such as testing procedures, backup plans, and programming languages.”
It is that isolationism that prevents many data science projects from becoming an overall success, and worse yet, lead to incorrect conclusions and assumptions. Much of the blame can be placed upon the waterfall development ideologies of the past, which have hampered the adoption of agility in the area of data sciences.
Douetteau adds “preventing failures takes a manager who is willing to act as tech stack and programming language dictator, who will force the team into a fixed technology for a solution. That manager should also ensure that team members adopt a big picture approach, where they are able to help each other complete tasks outside of their comfort zone. Individual silos of knowledge will hinder a team’s effectiveness, and collaboration is the key to success.”
For enterprises to truly become agile, they must eschew those waterfall development processes and switch to agile methods across the board. However, data science projects seem to be the most opportune place to start in today’s on demand, instant results world.
Douetteau adds “Providing a platform that caters to all members of the team promotes collaboration and communication, two elements that are essential to the success of any devops/data analysis project that involve multiple departments.”
What’s more, the lessons learned on data science projects can be readily applied to other areas of IT and business operations, making agile an achievable goal, as long as you know where to start.
Douetteau says “Finding a common ground between your data team and IT department will undoubtedly ease the process of creating a data product for your organization.  If all of your teams are aligned from the start of a project, each department knows their role and what technologies they are familiar with and specialize in to accomplish the task. Data scientists can build a solution and the IT department can deploy it.   Once a best practices procedure is established it can be reproduced and your organization can more quickly and effectively make use of new predictive data opportunities… making your organization truly agile”

Batman & Robin: How DevOps pairings can succeed

Part 1 of 4 in an analysis series sponsored by CenturyLink.

Making Development & Operations Partners

DevOps, while a big topic for most enterprises today, is often misunderstood and not approached well. This usually leads to a lack of success in implementing and using DevOps or not fully gaining the benefits DevOps brings. Many of the common approaches that fail involve putting a developer into an on-call rotation; this doesn’t work because the developer doesn’t have a stake in Operations.

One of the best ways to learn about DevOps is to look at an organization that has already successfully implemented it. CenturyLink’s Cloud team is an example of such an organization.

Why did CTL Cloud choose DevOps?
Jim Newkirk, VP of Engineering, joined Tier 3 in 2012 – prior to its acquisition by CenturyLink in 2013. Jim has been a Board Member of the Agile Alliance since 2009. Jim saw DevOps as a way to avoid fluctuations and chaotic behavior within the Engineering team whenever important bugs came in.

As a result, as new bugs are being reported, they are assigned to an individual developer — building empathy for customers and a better partnership with Operations. Developers stopped seeing themselves as performing Operations functions and Operations started seeing themselves as Developers too, since both sides are working on things that relate to the CenturyLink Cloud Platform.

Whilst good leadership can help organizations make any type of cultural shift deliver value, it is only a beginning. In this case, it required both the Development and Operations teams to work together. This initiative was started as a way to create developer accountability.

The Dynamic Duo – Batman & Robin

DevOps is the system of cooperation and participation by both the Development (Batman) and Operations (Robin) teams of an IT organization with the shared goal of delivering high quality, easily maintainable software to users. Batman and Robin (the characters from DC Comics fame) are an inspiration behind an analogy that CenturyLink uses to describe how they use DevOps internally.

Who is Batman?
Batman is a Developer on-call.
Batman is a detective and a problem solver.
Batman is also responsible for stability – to the point of asking for better logging and better data points in order to troubleshoot customer issues.
Batman now understands user issues and customer side use-cases.
Batman is driving innovation and the customer on-boarding process onto the CenturyLink Cloud platform.
Batman is not a specialist. He is a generalist. Batman has to work with many different technologies and learn many techniques in order to do his job.
Batman is constantly taking on new challenges that are thrown at him.

New Developers are put onto the front line call rotation after three weeks on the job. By giving Developers direct interaction with end users, Developers gain an appreciation for how their software is being used and where the operational pain is. This is what creates an empathetic cultural shift for Developers in understanding Operations by allowing the Developers to “get into the minds” of Operations. Developers also need to be accountable for their solutions in production, when the user is using them.

Developers need to be generalists as well, being able to debug code, run and understand SQL queries, and scripting, as examples, because without broad breadth, you cannot be Batman. This generalist approach allows many people to effectively solve problems across the entire stack, instead of creating a dependence on a single individual to solve a specific class of problems. The generalist can solve most problems because of their full understanding of how the system as a whole works, giving them the insight needed to get to the root cause of the problem and fix it.

Where does Robin fit in?
Robin is a customer service-centric thinker
Robin is a consumer of Development – a Systems Developer
Robin uses APIs – a functional programmer.

Operations figures out how to streamline the deployment and automation of applications written by the Development team. To do this, they write their own code against APIs supplied by the Development team, making them users of the code and platform as well. Because Operations is actually using the code and not just Operating the systems running code for end users, they are exposed to any weaknesses much earlier and understand how the applications and systems actually work. This is why Operations is Robin. Batman (Developers) can only do so much, often times Batman needs Robin to help him solve many of the problems that an individual can’t solve alone. Robin has the authority to make fixes live to solve customer issues.

What about the Utility Belt?

Batman relies on his skills, but he also takes full advantage of his Utility Belt. Batman’s Utility Belt is purpose built to solve a large number of general limitations that he faces as he tries to solve problems. In DevOps there are a large number of tools that can be used to automate and solve problems in both Development and Operations activities.

Here are some examples of items in CenturyLink’s Utility Belt:

Source: Gigaom Research

Source: Gigaom Research

Benefits

The CenturyLink Cloud team already had good structure in place before adding DevOps practices into their organization. For the purposes of this report, we are considering a very small subset of the prior structure:

  • Knowledge Base for support
  • Deployment time of 5 hours over 3 data centers

Since adding the DevOps-Batman & Robin ethos:

  • Knowledge Base has grown by 4x in 12 months
  • 40% of tickets are solved immediately through the Knowledge Base
  • Deployment time of 1 hour over 12 data centers
  • Still 1 Batman/Dev on-call with 3x VM inventory in 12 months and 4x Cloud capacity

The Knowledge Base is fed Batman (Dev) & Robin (Ops) teams. Better operational efficiencies have been realized through this new process even with the CenturyLink Cloud’s VM inventory tripling in last 12 months and capacity has quadrupled.

This same team now also owns infrastructure engineering – further removing dependencies on a separate a team for the CenturyLink Cloud platform.

Both Batman and Robin are key to keeping the CenturyLink Cloud growing and running and its customers happy – regardless of whether they are internal groups such as Savvis, or external groups such as partners or a customer of the platform.

Everyone is an Engineer.

Join me and the CenturyLink Cloud team this week on a webinar where we will discuss this post and other findings.

RedCritter Tracker: Gamifying agile project management

RedCritter Tracker is a thoughtfully gamified project management tool for software development teams that’s set to launch later this month. Founder and CEO Mike Beatty walked me through the app’s key capabilities, including project management support, visibility into employee skill sets, customizable rewards, and badges.