Set-Based Design

As a young Toyota engineer, you attack a problem with relish. You carefully identify the cause of the problem, taking care to do a thorough five-why analysis. You then think and think and come up with a brilliant solution. You detail the solution and run in to share it with your mentor. Instead of evaluating the idea on its merits and congratulating you, he asks, “What other alternatives have you considered? How does this solution compare with those alternatives?” You are stopped dead in your tracks, as you were convinced you had the best approach. [1]

... a Scrum Team has been formed within a Community of Trust. The Scrum Team is working from a Product Backlog that is continuously being refined. Some Product Backlog Items (PBIs) near the top of the Product Backlog are ready to go into a Production Episode, while others nearer the bottom of the backlog are far beyond the current horizon of concern. In between we find PBIs that seem like likely candidates for Product Increments. Several of them may be alternatives for other PBIs on the backlog. Or the Scrum Team may have alternative PBIs in mind for those listed on the backlog. Yet other PBIs have been firmed up to the degree that the team is confident of the requirements they must fulfill, but the team is considering the “how” of providing these requirements and has several ideas in mind.

✥       ✥       ✥ 

There are many possible solutions to a complex problem. It is often impossible to predict which is best.

Selecting a single option and developing it aggressively adds additional work later when new information emerges that forces team members to reconsider their initial decision. In a large system, with many sub-systems and many teams, this rework causes delays by forcing work to stop while the Scrum Team deliberates the new decision and works out consequences for other parts of the system.

While we value embracing change, we also value eliminating wasteful rework. It is easy to see in hindsight where we were stuck in analysis paralysis or how we initially lunged headstrong in the wrong direction. The challenge is to find the middle path.

Consider this graph from a classic article by Jakob Nielsen, [2] a father of UI and UX design, on Iterative User-Interface Design:

Interface quality as a function of the number of design iterations: Measured usability will normally go up for each additional iteration, until the design potentially reaches a point where it plateaus. [2]

In this article, which makes a strong case for using iteration to improve product quality, Nielsen states, “... we do not really know what leads to the creative insights necessary for a fundamentally novel and better interface design.” Consequently, we can say that iteration is insufficient for innovation. Customer feedback on regular product increments will improve the product, but it will not fundamentally reshape the conception of the product itself. For critical business needs which require differentiation from existing offerings, a different solution is needed.


develop many options in parallel, fixing only the critical constraints up front.

To accomplish this, each development episode proceeds to develop a chosen set of design alternatives in parallel. Each alternative in the set should be estimated by the people working on it, and they need to keep the estimates current. Then when all alternatives reach the halfway point, there is a meeting to trade notes and to evaluate which options to eliminate, and to discuss new options that may have been discovered. If a clear winner has emerged, the parallel work stops and the plan of record adopts the winner. If not, the team (always with input from the Product Owner) will budget themselves an additional work increment to gain additional understanding. The process of reducing options over time can be viewed as such:

Intentional phases of convergence and divergence [3]

As team members develop each option through successive iterations, they gradually increase the fidelity of the work output [4]. An initial work increment may only produce a hand-wavy sketch, a second might produce a throw-away prototype, and a final work increment would produce an initial real implementation. So each time the Product Owner and Developers decide to eliminate some options, they also increase the level of detail of the remaining options.

A single option will vary multiple parameters (e.g., algorithm selection, data structure, screen layout) to explore different design alternatives. Great care should be given to how many parameters to vary across the multiple options. As you grow in understand of how to run set-based design, you can increase the number of parameters being varied. Be thoughtful about the degree of overlap between the options. Independent options will cover more of the design space, but fixing a parameter will increase robustness of the result. When done well, we can minimize the lead time, using a reduced number of experiments. The point here is that you can design your learning, rather than proceeding from surprise to surprise.

The number of options increases with each iteration, while the number of exploration sets decreases.

✥       ✥       ✥ 

Depending on the scope and the extent of the Set-Based Design exercise, the results can feed directly into a Sprint or their insights can inform decisions as the team builds a Refined Product Backlog.

When many options are being considered to a problem, dependent teams cannot lock in their system designs. This does not mean that work must wait until a decision is made however. By working within the critical constraints the design of a sub-system can proceed, while being prepared to fit within the overall solution space of the larger system. In fact, some parameters may never be fixed, and rather left open until production, and in some cases even offered to the customer to decide.

Converging interdependent components over time [5]

A key consideration is when to begin converging toward the final solution. Ford and Sobek modeled Toyota’s development process using real-options and ran 100 simulations of a 900-day long project. [6]

Expected quality of point-based and set-based development projects. [6]

Expected values of flexibility in set-based development relative to point-based development. [6]

These findings suggest that the optimal time for convergence is near the middle of the overall project span. However the authors caution that if there are delays early in the project, it may be wise to push convergence even a little later than the midpoint. Therefore, the Product Owner must have visibility into the overall progress of the divergent work.

At Toyota it is preferred to have the same people involved in developing all of the options. This way the group develops a feeling for the problem space, which can guide the design of new options. The process can be accelerated by developing several of the options at the same time rather than sequentially. This rapid comparing of options as they’re being built creates the shared experience needed to develop one’s intuition. It’s also worth pointing out that set-based design ensures people keep capabilities in areas that haven’t used in a product in a long time. This preserves options that may be valuable in future products.

Google’s Design Sprint is one approach for rapidly evaluating many design ideas. The authors claim it “gives teams a shortcut to learning without building and launching.” The process begins with mapping out the problem and identifying an important area to focus, and concludes with testing a high-fidelity prototype with customers. The whole process takes 1 week or less. [7]

Last, at these key moments where the work product of team members is eliminated, we must consider the human dimension: having one’s work judged by others can hurt and potentially even be seen as a personal slight. This loss creates the need for a “rite of passage” of sorts. Robert Heath, author of “Celebrating Failure” recommends that leaders should “acknowledge failures with positive comments. Congratulate everyone on their efforts and remind them of the strengths and wisdom they gained from the experience. Put in perspective, a failure can do as much to motivate a team as a win.” [8]

These in-the-moment comments, however, will mean nothing if they do not take place within a larger culture of continuous learning. Google’s culture of “blameless postmortems” is one such example. It draws from experience in other areas like, healthcare and avionics, where failure can be fatal. In these workplaces the focus after a failure is not on the individuals involved but on the larger system, and uncovering the underlying reasons why people had partial or incorrect information which led to the undesirable result. In Site Reliability Engineering: How Google Runs Production Systems, Lunney and Lueder write that “you can’t ‘fix’ people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems.” [9] This idea of collective improvement must be embedded in the team for this pattern to be applied successfully.

The Product Owner Team may apply this approach to evaluate which of several alternative PBIs should be retained on the Product Backlog, or to explore alternatives for a high-risk or high-uncertainty PBI or entire Product Increment. Such a design exploration may cross several Sprint boundaries.

Alternatively, the Development Team may use this approach within a Sprint to evaluate risk across multiple implementation alternatives, and can use the resulting insight to select one alternative. When doing Set-Based Design within a Sprint the Development Team should time-box the exploration. Because the process creates a high rate of emergent requirements (that’s why they do it) it is difficult to estimate how much the team will accomplish in the time box. Therefore, when using this approach, the team must accept the fact that there is a risk that the exploration may not result in a potentially shippable Product Increment, but it is nonetheless worth the time spent for the insight gained.

Whether the results of a given Set-Based Design make it to the market or not, the team gains confidence in such features’ presence (or not) in the product, increasing Product Pride.

[1] Jeffrey K. Liker. The Toyota way: 14 management principles from the world’s greatest manufacturer. New York: McGraw-Hill, 2014, p. 263.

[2] Jakob Nielsen. “Iterative User Interface Design.” In IEEE Computer 26(11), 1993, pp. 32-41, (accessed 2 November 2017).

[3] David J. Singer, Norbert Doerry, and Michael E. Buckley. “What is set-based design?” In Naval Engineers Journal 121(4), 2009, pp. 31-43.

[4] David J. Singer, Norbert Doerry, and Michael E. Buckley. “What is set-based design?” In Naval Engineers Journal 121(4), 2009, pp. 31-43.

[5] Allen Ward, Jeffrey K. Liker, John J. Cristiano, and Durward K. Sobek, II. “The Second Toyota Paradox: How Delaying Decisions can Make Better Cars Faster.” In Sloan Management Review 36(3), January 1, 1995, p. 43.

[6] David N. Ford and Durward K. Sobeck, II. “Adapting Real Options to New Product Development by Modeling the Second Toyota Paradox.” In David N. Ford, and Durward K. Sobek, II, eds., IEEE Transactions on Engineering Management 52(2), May 2005.

[7] Jake Knapp (Google Ventures) et al. “The Design Sprint.”, n.d., no permalink, accessed 2 November 2017.

[8] Ralph Heath. Celebrating failure: The power of taking risks, making mistakes, and thinking big. Warne, NJ: Career Press, 2009.

[9] John Lunney & Sue Lueder. “Postmortem Culture: Learning from Failure.” In Betsey Beyer et al., eds., Site Reliability Engineering: How Google Runs Production Systems, O’Reilly, 2016, Chapter 15.

Picture from: Pixabay (under CC0 license).