Multi-team Software Delivery Assessment

The Multi-team Software Delivery Assessment is a simple, easy-to-execute approach to assessing software delivery across many different teams within an organisation. Devised by Matthew Skelton of Conflux, it is used as a key part of the Software Delivery Assessment at Conflux, but can be used freely by anyone (subject to the CC BY-SA license below).

The assessment uses and builds on the well-known and proven Spotify Squad Health Check model.

Translations: Japanese (ja 🇯🇵)

The assessment covers ten dimensions in total:

These ten dimensions cover key aspects of modern software delivery in a form that enables teams to self-assess their strengths and practices.

🚀 Overview: see slides 32-38 in Continuous Delivery at scale

🃏 Card deck: make the assessment fun and interactive by using the Software Delivery Assessment printed card deck from Agile Stationery. Developed in collaboration with Conflux, the card deck has Tired and Inspired indicators for each of the assessment criteria, together with emoji cards for quick-fire voting from team members. The card deck works for remote assessments too!

Copyright © 2018-2021 Conflux Digital Ltd

Licenced under CC BY-SA 4.0

Permalink: SoftwareDeliveryAssessment.com

Purpose of the assessments

The aim of the assessments is to promote and sustain a positive working environment for building and running software systems where:

Changes to software are built, tested, and deployed to Production rapidly and safely using Continuous Delivery practices
Processes and practices are optimised for the flow of change towards Production
Software is designed and built to enable independent, decoupled deployments for separate families of systems
Software is designed and built in a way that addresses operability, testability, releasability, security, and reliability
Problems in Production are always detected by teams before customers and users notice
Responsibility and accountability for software changes lead to empowerment and ownership
Working with software is rewarding and interesting
Being on-call and supporting the software is sustainable and valuable
People feel confident to challenge poor practices and approaches
Teams have a clear mission and well-defined interaction patterns with other teams

Fundamentally, the assessments should help to unblock and enable teams so they can succeed. The assessments should help teams to improve how they build, test, and deploy software systems through identifying different kinds of improvements:

Team-focused improvements
Product-focused and Service-focused improvements
Organisation-wide improvements

The assessments should NOT be used to penalize teams, but to provide a shared drive towards improving practices and quality.

Teams included in the assessments

Every team writing code, scripts, and/or configuration for application software or infrastructure will benefit from being included in the assessments:

Teams building user- and customer-facing websites and services
Teams building internal services
Teams building infrastructure to support other systems (including Platform teams)
Teams building build & deployment tooling and scripts
Teams configuring and testing COTS products as part of the software & infrastructure estate
Any other teams with a primary focus on building, configuring, and testing software and infrastructure

By "team", we mean 6-10 person group that works together closely, usually called a Squad, Scrum team, Product team, or Stream-aligned team.

Assessment criteria

The criteria for each dimension are taken from existing published books and online sources:

Team Health - based on the criteria from Spotify Squad Health Check with some additions
Deployment - based on key questions from the book DevOps for the Modern Enterprise by Mirco Hering as discussed on Mirco's blog post Mirco’s self assessment questions of DevOps Maturity
Flow - based on criteria from the book Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim, plus some details from The Principles of Product Development Flow by Don Reinertsen
Continuous Delivery - based on selected criteria the book Continuous Delivery by Jez Humble and Dave Farley and the summary of the book at CDchecklist.info
Operability - based on selected criteria from the book Team Guide to Software Operability by Matthew Skelton, Alex Moore, and Rob thatcher, together with some questions from OperabilityQuestions.com
Testing and Testability - based on selected criteria from the books Agile Testing by Lisa Crispin and Janet Gregory, Continuous Delivery by Jez Humble and Dave Farley, Growing Object-Oriented Software by Steve Freeman and Nat Price, Working Effectively with Legacy Code by Michael Feathers, Team Guide to Software Testability by Ash Winter and Rob Meaney, and TestabilityQuestions.com.
Reliability and SRE - based on selected criteria from the books Site Reliability Engineering by Betsy Beyer, Chris Jones, Jennifer Petoff, & Niall Murphy, The Site Reliability Workbook edited by Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, & Stephen Thorne, Seeking SRE edited by David N. Blank-Edelman, Team Guide to Software Operability by Matthew Skelton, Alex Moore, & Rob Thatcher.
On-call - based on selected criteria from the books Site Reliability Engineering by Betsy Beyer, Chris Jones, Jennifer Petoff, & Niall Murphy, The Site Reliability Workbook edited by Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, & Stephen Thorne, Team Guide to Software Operability by Matthew Skelton, Alex Moore, & Rob Thatcher.
Security and Securability - based on selected criteria from the books Agile Application Security by Laura Bell, Michael Brunton-Spall, Rich Smith, Jim Bird; Alice and Bob Learn Application Security by Tanya Janca; Secure by Design by Dan Bergh Johnsson, Daniel Deogun, Daniel Sawano; Continuous Delivery by Jez Humble and Dave Farley; and Threat Modeling: Designing for Security by Adam Shostack; together with the OWASP Top Ten list of application security risks.
Team Topologies team interactions - based on selected criteria from the books Team Topologies by Matthew Skelton and Manuel Pais and Dynamic Reteaming by Heidi Helfand, together with ideas from the Team Topologies area on GitHub.

How to run the assessments

The assessment session itself should feel somewhat like a team retrospective session. The main difference from a normal retrospective session is that in the team assessment session the facilitator guides the discussion more firmly. There are many questions to discuss and it's important that the team discusses all of the criteria in the time available.

At the end of the assessment session, the team should feel encouraged and empowered to decide on what actions they want to take to improve their processes and practices based on the discussions.

Cadence

Many organisations find that running team assessments every 3 months provides a good result.

Preparation

Find someone to Facilitate the assessment. This should be someone from outside the team, who is familiar with running team retrospectives.
Book a time slot for 2 or 3 hours: for online sessions you could split this over 2 or more video call sessions, and for in-person sessions book a room large enough for the team.
For online sessions, either use the card deck from Agile Stationery and screenshot or note down the results in a spreadsheet, or use an online poll tool to capture responses from people in the session. For in-person sessions, either use the card deck from Agile Stationery, or print the assessment sheets for each set of criteria, either using the ready-made A1 PDF (see Releases), or the individual assessment pages at A1 size if possible (use small margins):
For online sessions, show the Tired and Inspired criteria on the screen alongside the question. For in-person sessions, either print the details pages as a guide or have the pages open on-screen to understand the context and details of each of the assessment criteria:
It is valuable to capture details and nuances of the discussions around each question. For online sessions, have someone take notes in a document or shared whiteboard. For in-person sessions, bring lots of marker pens or whiteboard markers: red, blue, and green are best.
Include someone who is familiar with facilitating retrospectives (possibly a scrum master) in the session. They will be shadowing the facilitator during the session so the person from your team can facilitate other assessment sessions later.

Make sure that the Facilitator understands the purpose of the session and is familiar with the assessment pages and questions.

Facilitators

The facilitator should familiarise themselves with the Spotify Squad Health Check approach before running the session. See How I Used the Spotify Squad Health Check Model for a good experience report, Squad Health Checks from SkyBet, and download instructions from Spotify (PDF).

During the assessment:

Keep the team on schedule by asking for some discussions to be held outside the session

Write down the team scores and notes on the printed assessment sheets or ensure that scores and notes are captured in a digital tool.

Take photographs of the completed assessment sheets for in-person sessions.

Get feedback from the team on the VALUE and EXECUTION of the engineering assessment - smiley faces are sufficient!

Timings

Each team assessment runs for around 3-4 hours, and the facilitator will run the team through 10 sets of questions:

Team health check - 35 mins
Deployment health check - 10 mins
Flow check - 10 mins
Continuous Delivery check - 20 mins
Operability check - 20 mins
Test coverage check - 20 mins
Reliability and SRE - 30 mins
On-call - 15 mins
Security check - 30 mins
Team Topologies check - 20 mins

These timings leave space for two 10 minute breaks during the assessment.

Running the Assessment session

Each section has several questions. Each question should be answered as follows:

The team (either as individuals or as a team) rate each of the criteria using SAD (1 OR 2) / MEH (3) / YAY (4 OR 5) based on the Tired and Inspired guidelines
- Tired aligns to a low rating (1), and Inspired aligns to a high rating (5)
- If you used individual ratings, tally the ratings and/or decide on a single team score from 1 to 5. You may find it useful to use different coloured pens on the printed sheet to indicate visually the different ratings.
The Trend since the previous time is identified (going up, staying roughly the same, going down), if applicable
An Action agreed to improve the score for that question over the coming months.
Use the Notes column to indicate further information that you think is valuable for the coordinating team to know about.
Make sure to complete to the Date/Name/Facilitator details
For in-person sessions, take a photo of each completed sheet and send to the person coordinating the assessments
Get team members to rate the assessment session itself in terms of: Value, Execution (sad, meh, happy faces)

Viral facilitation

Each team assessment session has present one person who is shadowing the facilitator so that they can themselves facilitate future sessions. Each new facilitator should facilitate at least 2 sessions with other teams. In this way, the number of facilitators expands rapidly, enabling a minimal burden on the initial facilitators.

Coordinating and interpreting the results

After teams have each run an assessment session and sent their results, the coordinating group should collate the results from different teams to identify any areas that need improving across the organisation. Ask questions such as:

Why does team ABC rate themselves as 1 for test coverage? What is hindering them?
What can we do as an organisation to help more teams with deployments?
Is there an aspect of the Platform that needs improving so teams can go faster?

Do not attempt to rank or compare teams directly. Instead, use the signals from the teams to understand the organisational dynamics better and then prioritise organisation-wide improvements.

ConfluxHQ/software-delivery-assessment

ConfluxHQ

Reviews

Repository Details