Scrum lessons for R&D projects

This document describes an adaptation of the Scrum agile methodology aimed at Applied Research.

The structure and some content of the methodology described here are derived from the official Scrum guide, written by Ken Schwaber and Jeff Sutherland, the originators of Scrum and available at scrum.org. Also, this work is inspired by the SCORE or “SCrum fOr REsearch” method, by Michael Hicks and Jeffrey S. Foster ¹.

Author: Vitor Casadei

---

1. Scrum Events and Ceremonies

1.1 The Sprint

1.1.1 Description

Sprints are the heartbeat of Scrum, where ideas are turned into value.

They are fixed length events of usually 4 weeks or less to create consistency. A new Sprint starts immediately after the conclusion of the previous Sprint.

All the work necessary to achieve the Research Goal, including Sprint Planning, Daily Scrums, Sprint Review, and Sprint Retrospective, happen within Sprints.

During the Sprint:

• No changes are made that would endanger the Sprint Goal;
• Quality does not decrease;
• The Research Backlog is refined as needed.

1.1.2 In Practice

Sprints enable predictability by ensuring inspection and adaptation of progress toward a Research Goal. When a Sprint’s horizon is too long the Sprint Goal may become invalid, complexity may rise, and risk may increase. Shorter Sprints can be employed to generate more learning cycles and limit risk of cost and effort to a smaller time frame. Each Sprint may be considered a short project, however, in the research space, many research leads may take several sprints to be completed, therefore, it’s necessary to create exceptions to this rule and enable more flexible sprints where histories may be worked on multiple sprints and the results may not be ready at the end of each sprint.

Various practices exist to forecast progress, like burn-downs, burn-ups, or cumulative flows. While proven useful, these do not replace the importance of empiricism. In complex environments, what will happen is unknown. Only what has already happened may be used for forward-looking decision making.

On research projects, there will be cases where a fixed timeframe is set to explore a certain subject: for a given hypothesis the team can create a history and the team must agree on a maximum time/effort that will be employed in this history. For example, the team can agree that 2 or 3 sprints may be used to better understand a given subject and that, at the end of the period, the team will assess the knowledge and decide whether it’s necessary and valid to continue the research. This process is done during the Sprint Planning.

At the end of the Sprint, some sort of Research Documentation must be delivered as a way to perpetuate the knowledge and serve as a knowledge base for writing articles or revisiting information.

1.2 Sprint Planning

1.2.1 Description

Sprint Planning initiates the Sprint by laying out the work to be performed for the Sprint. This resulting plan is created by the collaborative work of the entire Scrum Team.

The Owner ensures that attendees are prepared to discuss the most important Research Backlog items and how they map to the Research Goal. The Scrum Team may also invite other people to attend Sprint Planning to provide advice.

1.2.2 In Practice

Sprint Planning addresses the following topics:

1.2.2.1 Topic One: Why is this Sprint valuable?

The Owner proposes how the research could increase its value and utility in the current Sprint. The whole Scrum Team then collaborates to define a Sprint Goal that communicates why the Sprint is valuable to stakeholders. The Sprint Goal must be finalized prior to the end of Sprint Planning.

1.2.2.2 Topic Two: What can be Done this Sprint?

Through discussion with the Owner, the Researchers select items from the Research Backlog to include in the current Sprint. The Scrum Team may refine these items during this process, which increases understanding and confidence.

Selecting how much can be completed within a Sprint may be challenging. However, the more the Researchers know about their past performance, their upcoming capacity, and their Definition of Done, the more confident they will be in their Sprint forecasts.

1.2.2.3 Topic Three: How will the chosen work get done?

For each selected Research Backlog item, the Researchers plan the work necessary to create an Increment that meets the Definition of Done. This is often done by decomposing Research Backlog items into smaller work items of one day or less. How this is done is at the sole discretion of the Researchers. No one else tells them how to turn Research Backlog items into Increments of value.

The Sprint Goal, the Research Backlog items selected for the Sprint, plus the plan for delivering them are together referred to as the Sprint Backlog.

Sprint Planning is timeboxed to a maximum of eight hours for a one-month Sprint. For shorter Sprints, the event is usually shorter. Usually, it’s advised to keep Sprint Plannings in under 2 hours.

1.3 Daily Scrum

1.3.1 Description

The purpose of the Daily Scrum is to inspect progress toward the Sprint Goal and adapt the Sprint Backlog as necessary, adjusting the upcoming planned work.

1.3.2 In Practice

The Daily Scrum is a 15-minute event for the Researchers of the Scrum Team. The Researchers can select whatever structure and techniques they want, as long as their Daily Scrum focuses on progress toward the Sprint Goal and produces an actionable plan for the next day of work. This creates focus and improves self-management.

Daily Scrums improve communications, identify impediments, promote quick decision-making, and consequently eliminate the need for other meetings.

The Daily Scrum is not the only time Researchers are allowed to adjust their plan. They often meet throughout the day for more detailed discussions about adapting or re-planning the rest of the Sprint’s work.

In a research environment, according to the size of the Team, there may be multiple Lines of Research being explored at the same time. Therefore, it’s advised to have specific meetings between those involved in the Line of Research to discuss the details of the Research.

1.4 Sprint Review

1.4.1 Description

The purpose of the Sprint Review is to inspect the outcome of the Sprint and determine future adaptations. The Scrum Team presents the results of their work to key stakeholders and progress toward the Research Goal is discussed.

1.4.2 In Practice

During the event, the Scrum Team and stakeholders review what was accomplished in the Sprint and what has changed in their environment. Based on this information, attendees collaborate on what to do next. The Research Backlog may also be adjusted to meet new opportunities. The Sprint Review is a working session and the Scrum Team should avoid limiting it to a presentation.

The Sprint Review is the second to last event of the Sprint and is timeboxed to a maximum of four hours for a one-month Sprint. For shorter Sprints, the event is usually shorter, for example, an one hour session.

1.5 Sprint Retrospective

1.5.1 Description

The purpose of the Sprint Retrospective is to plan ways to increase quality and effectiveness.

1.5.2 In Practice

The Scrum Team inspects how the last Sprint went with regards to individuals, interactions, processes, tools, and their Definition of Done. Inspected elements often vary with the domain of work. Assumptions that led them astray are identified and their origins explored. The Scrum Team discusses what went well during the Sprint, what problems it encountered, and how those problems were (or were not) solved.

The Scrum Team identifies the most helpful changes to improve its effectiveness. The most impactful improvements are addressed as soon as possible. They may even be added to the Sprint Backlog for the next Sprint.

The Sprint Retrospective concludes the Sprint. It is timeboxed to a maximum of three hours for a one-month Sprint. For shorter Sprints, the event is usually shorter, for example, 30 minutes to one hour sessions.

2 Scrum Artifacts

Scrum’s artifacts represent work or value. They are designed to maximize transparency of key information. Thus, everyone inspecting them has the same basis for adaptation.

Each artifact contains a commitment to ensure it provides information that enhances transparency and focus against which progress can be measured:

• For the Research Backlog it is the Research Goal.
• For the Sprint Backlog it is the Sprint Goal.
• For the Increment it is the Definition of Done.

These commitments exist to reinforce empiricism and the Scrum values for the Scrum Team and their stakeholders.

2.1 Research Backlog

The Research Backlog is an emergent, ordered list of what is intended to be researched. It is the single source of work undertaken by the Scrum Team.

Research Backlog items usually are planned to be finished within a sprint. However, in the research environment, many items may not fit in a sprint and cannot be split either, in these cases, they may take more than one sprint to be completed, as long as the Team agrees on a timeframe composed of a maximum number of sprints that could be invested to complete the work and gather as much information as possible during this time.

Research Backlog refinement is the act of breaking down and further defining Research Backlog items into smaller more precise items. This is an ongoing activity to add details, such as a description, order, and size. Attributes often vary with the domain of work.

The Researchers who will be doing the work are responsible for the sizing.

2.1.1 Commitment: Research Goal

The Research Goal describes a future state of the research which can serve as a target for the Scrum Team to plan against. The Research Goal is in the Research Backlog. The rest of the Research Backlog emerges to define “what” will fulfill the Research Goal.

The Research Goal is the long-term objective for the Scrum Team. They must fulfill (or abandon) one objective before taking on the next.

2.2 Research Documentation

Research documentation is an extremely important part of the research process as it serves as a possible source of information for future research and also for the writing of research articles.

There isn’t a correct way of doing the documentation and the Team must come to an agreement on what is the best way to proceed. However, two approaches are suggested here:

1. A document that is written for each sprint, outlining the work and progress that has been done in that particular sprint;
2. A document that is not organized by sprints, but by Lines of Research. At the end of each sprint, a new section is added to the appropriate section describing what was done in the ending sprint.

2.3 Sprint Backlog

The Sprint Backlog is composed of the Sprint Goal (why), the set of Research Backlog items selected for the Sprint (what), as well as an actionable plan for delivering the Increment (how).

The Sprint Backlog is a plan by and for the Researchers. It is a highly visible, real-time picture of the work that the Researchers plan to accomplish during the Sprint in order to achieve the Sprint Goal. Consequently, the Sprint Backlog is updated throughout the Sprint as more is learned. It should have enough detail that they can inspect their progress in the Daily Scrum.

2.3.1 Commitment: Sprint Goal

The Sprint Goal is the single objective for the Sprint. Although the Sprint Goal is a commitment by the Researchers, it provides flexibility in terms of the exact work needed to achieve it. The Sprint Goal also creates coherence and focus, encouraging the Scrum Team to work together rather than on separate initiatives.

The Sprint Goal is created during the Sprint Planning event and then added to the Sprint Backlog. As the Researchers work during the Sprint, they keep the Sprint Goal in mind. If the work turns out to be different than they expected, they collaborate with the Owner to negotiate the scope of the Sprint Backlog within the Sprint without affecting the Sprint Goal.

2.4 Increment

An Increment is a concrete stepping stone toward the Research Goal. Each Increment is additive to all prior Increments and thoroughly verified, ensuring that all Increments work together. In order to provide value, the Increment must be usable.

Multiple Increments may be created within a Sprint. The sum of the Increments is presented at the Sprint Review thus supporting empiricism. However, an Increment may be delivered to stakeholders prior to the end of the Sprint. The Sprint Review should never be considered a gate to releasing value.

Work cannot be considered part of an Increment unless it meets the Definition of Done.

2.4.1 Commitment: Definition of Done

The Definition of Done is a formal description of the state of the Increment when it meets the research criteria defined by the team, that is, when the Team has reached some result on the Research Line, when the time defined to be employed in that Research has been reached or when the Team decides that the results are satisfactory.

The moment a Research Backlog item meets the Definition of Done, an Increment is born.

The Definition of Done creates transparency by providing everyone a shared understanding of what work was completed as part of the Increment. If a Research Backlog item does not meet the Definition of Done, it cannot be released or even presented at the Sprint Review. Instead, it returns to the Research Backlog for future consideration.

The Researchers are required to conform to the Definition of Done.