More and more I am reading about, and experiencing, organisations who have adopted “Scrum” as their process, but are complaining that they aren’t achieving any of the benefits that they expected from “going agile”. This often manifests itself in broad claims that agile is a failure as a methodology.
I’ve come to believe that there is a fundamental misunderstanding around what agile actually is, how it is supposed to help, and in particular how it relates to Scrum. In this post I want to try to address this confusion.
Misunderstanding Scrum
The most common mistake that I tend to see is organisations and teams adopting all the ceremonies of the Scrum process, but then tending to treat their estimation and planning just like a more traditional project. E.g.:
- Start with a detailed requirement and a relatively fixed scope
- Break the work down into a set of epics, stories and tasks
- Estimate the size of each ticket using some magic number (e.g. story points)
- Determine the team velocity based on previous metrics for an unrelated project (or make a guess)
- Place the tickets into separate sprints based on their size and the team velocity
- Generate a plan from this sprint breakdown
- Determine a delivery date and communicate it to senior management
The team then execute on the plan using the Scrum ceremonies. Assuming the requirements were perfect; the technical complexity as expected; the team capacity correct; and there were no unknowns - then it’s likely delivery will be close to the planned date.
However, what normally happens is that things don’t always run perfectly. Sometimes we get lucky and things prove simpler than expected and the project can deliver early. More often…
- The requirements might be unclear or we may not be delivering what the customer actually wants
- There might be some unknowns that can’t be easily be resolved prior to estimation
- There might be some unknown unknowns that aren’t discovered until some time into the project
- This project might be more technically complex than the previous ones that the velocity was based on
- The capacity of the team might not be consistent with what was planned due to things like: other/new business commitments; changes to members; holidays and sickness; or technical support needs
When a project comes up against this reality there’s a good chance that it no longer matches the planned delivery. Rather than just accept that very few projects run perfectly true to an initial plan, the common outcome is then to start blaming Scrum and the agile methodology using reasons like:
- “We followed all the processes and ceremonies and we still didn’t deliver on time”
- “The productivity of the engineers didn’t improve like we expected, they must not have been working hard enough”
- “We used agile but still didn’t manage to deliver what the customer wanted”
Agile gets the blame because Scrum was used against a traditional project planning and delivery approach: this doesn’t provide any benefit. Let’s look at how agile should really work and how the Scrum process then fits into that…
The Purpose of Agile
Agile was never intended as an estimation and planning tool. The primary purpose of the agile methodology is to manage uncertainty and change and ensure the team is building what the customer really wants. Uncertainty comes in a number of forms:
- Requirement Uncertainty - are we building the right thing? is it what the customer actually wants? does it solve their problem?
- Technical Uncertainty - how confident are we that we know how to build this? are there any unknowns that we can’t easily resolve? how do we adapt to challenges that we just didn’t expect at all?
- Resource Uncertainty - will the team be stable? will there be unexpected sickness or holiday? what about sudden new and urgent business needs? how about an unplanned major support issue?
For all of the above we manage this uncertainty by creating fast, short feedback loops. We learn from the feedback and adapt the project based on this. Examples of some of the fast feedback loops we might use include:
- Having a customer on the project team to immediately validate that any decisions are correct
- Releasing features to customers in small increments, as early as possible, in order to gain feedback that we are solving the correct problem
- De-risking an important integration by building and releasing it early in the project so that it can be used and monitored and any issues detected and resolved before they become a significant blocker
- Building and running suites of automated tests early and regularly to instantly spot any breaks in functionality as new features are added
- Re-adjusting planned capacity frequently, to account for more dynamic resourcing patterns
In all of the above examples, the team uses the feedback from the most recent activities as input to adapt what they are producing and how they are creating it. This adaption is usually realised as a change of project scope or a change to when the project may be delivered (usually some combination of both). Sometimes there may also be a change in the project resource allocation, but this is less common unless it’s related to a departing team member.
A note about planning and estimation: while agile is not an estimation and planning tool, it’s still possible to plan out a project. After updating the backlog to include the latest feedback it is possible to break the work down into multiple small releases and predict when they will be available (given the current level of uncertainty). The key is to understand that the delivery is likely to change every time feedback is applied to the backlog and that therefore planning is continuous and ongoing process, rather than a fixed point-in-time commitment.
Enter Scrum
These feedback cycles are where Scrum fits into the picture. If a team is not super experienced working with constant change, then the multiple overlapping feedback cycles can be rather daunting to deal with. By breaking cycles down into fixed time-boxed sprints, this provides a cadence for managing feedback.
Take a two-week sprint…
The team starts out by grooming their backlog. They take onboard feedback and learning from previous sprints and releases. They talk to the business and customer to understand which work has the highest importance (this may regularly change due to new business understanding or feedback from early releases to customers). They also look at risks and unknowns that they need to resolve.
By its very nature, this grooming is likely to change both the scope and delivery timescales for the project each time it takes place. It is the responsibility of the team and product owners to balance these two competing factors to meet whatever the business deems to be more important: the set of features delivered or the date that the project is complete. You can’t have both if your project contains any uncertainty (and almost all non-trivial projects do).
Next up the team plans their sprint. They take on board the newly updated backlog and information about availability of team members for the next two weeks. They then aim to select the correct amount of work to fully utilise all the available engineering time in the sprint. The selection of items should be based on the combinations of the priority of the backlog and the items that are likely to give the most useful feedback about areas of high uncertainty.
Aside: I personally don’t believe that sizing tickets with some magic story point number and then selecting tickets that match some approximate velocity is the correct way to determine what should be included into a sprint. My feeling is that it should be much more of a judgement call on what the team think they are confident to get done - sometimes it will be more and sometimes it will be less, and this mostly depends on how much uncertainty there is around each story. In my experience engineer intuition is usually more accurate than arbitrary story points and velocity calculations!
The team then go on to complete their sprint, ideally releasing software that provides them with additional feedback that can be used to further reduce uncertainty. They also conduct a retrospective to discover what they have learnt and how they can improve, which becomes another source of feedback that is fed into the next sprint.
So we see that Scrum comes about as a set of ceremonies that help manage the process of uncertainty, feedback and change. In itself, it is not agile, it’s just a way of making agile easier to work with and control.
Conclusions
If an organisation isn’t dealing with uncertainty and change in an agile way, then introducing Scrum won’t be of any help. It is not the process that creates agility: it’s a way to make agility more controlled and manageable.
To work in an agile way, organisations have to accept that there is uncertainty everywhere and the goal is to continuously aim for its reduction. Short feedback loops are a way of doing this, but they result in constant change. The key advantage of accepting this feedback-change cycle means organisations are much more likely to deliver solutions that meet the customer’s requirements and which have less technical and quality issues.
It is vital to understand that in this way of working the trade off is generally between scope and delivery timescale. All uncertainty and change impacts on delivery and the more uncertainty there is, the more variable the delivery will be. If a fixed delivery timescale is essential then the only way to achieve this and still be responsive to uncertainty is by adjusting scope (or visa-versa).
If the requirement from the organisation is to deliver fixed-scope projects to a specific, committed delivery time then there’s no point adopting an agile approach or using Scrum. If an organisation really wants to work this way then they are better to just plan and run the project using more traditional techniques. However:
- Estimates must contain a significant measure of contingency based on the perceived level of uncertainty
- It makes it much more essential that the requirements are clear and accurate as there is less opportunity to adapt
- The project is significantly more likely to deliver solutions that aren’t what the customer really wanted or that don’t solve their problems
- Technical quality might be lower as there will be less scope to adjust to technology unknowns that are discovered along the way
Please don’t adopt Scrum unless you are also being agile, and don’t blame agile when Scrum fails to deliver against projects with fixed scope and delivery timescales.
comments powered by Disqus