Mindset practice questions

Safe-to-fail experiments limit how many users a change reaches and make it quick to reverse, so teams can learn in production early.

The team needs real customers (to see the effect on purchases) and limited damage. A feature toggle on a small cohort gives both. A full release maximises damage, staging has no real customers, and waiting for automation postpones the learning.

1. List the two needs: real customers and limited damage.
2. Look for an option that controls exposure.
3. Check it can be reversed quickly.
4. Reject options that miss either need.

Exam tip: Feature toggles plus small cohorts make production experiments cheap to fail.

Teams innovate only when experimenting has real, planned capacity — it cannot live in the gaps of a fully loaded iteration.

The barrier is capacity: every slot is filled with committed work. Agreeing reserved capacity with the product owner, who orders the work, removes that barrier. After-hours work is unsustainable, annual planning is far too slow, and per-idea sponsor approval adds friction.

1. Identify the barrier: no capacity for trying ideas.
2. Recognise who controls capacity trade-offs: the product owner.
3. Pick the option that builds experiments into normal work.
4. Reject options that delay ideas or add approval friction.

Exam tip: Make room for experiments in the plan; innovation squeezed into spare time rarely happens.

Experiment early on the riskiest assumption: the one that is both most uncertain and most damaging to the product if it turns out false.

The stem gives evidence for the commission and map assumptions. Owner trust and booking-channel preference are both untested, so impact decides: if owners won't trust strangers, nobody books walks, whereas a wrong channel guess only changes the design. Testing trust first gives the most learning per unit of effort.

1. List each assumption and the evidence the stem gives for it.
2. Rate each for uncertainty and impact if wrong.
3. Pick the one that is high on both.
4. Plan the cheapest experiment that tests it.

Exam tip: Test first what is least known and most fatal if wrong.

A trustworthy product experiment has a falsifiable hypothesis with a success threshold set in advance and a comparison that isolates the change.

Setting the expected change before launch makes the experiment a real test, and a concurrent control group isolates the effect of the shorter form. Page views measure the wrong thing, a sales campaign adds noise, and ratings from completers are biased opinions.

1. Restate the hypothesis: shorter form raises completion rate.
2. Ask what defines success before the data arrives.
3. Ask what isolates the form's effect from other causes.
4. Reject measures that are off-target or biased.

Exam tip: Agree the success threshold first, and compare against a control.

Vanity metrics (views, totals) can rise for reasons unrelated to the change. An experiment must report the metric its hypothesis names.

The hypothesis is about a rate (share completing setup). The report shows totals that a campaign would inflate regardless of the redesign. The practitioner should redirect the team to the completion rate, ideally against a control. Rolling out or gathering more views repeats the same mistake.

1. Find the metric the hypothesis names: setup completion rate.
2. Check whether the report contains it.
3. Notice the campaign could explain the reported rise.
4. Ask for the hypothesis metric before deciding.

Exam tip: Judge an experiment by the metric in its hypothesis, not by numbers that merely went up.

Match the experiment to the question: workflow and usability questions are answered fastest by low-fidelity prototypes tried by real users.

The question is about people following a workflow, the deadline is two days and no code is wanted yet. A paper or clickable mock-up meets all three. Working software is too slow, a specification tests reading not doing, and a spike tests technology.

1. Classify the question: usability, not technology.
2. Note the constraints: two days, no code.
3. Pick the cheapest artifact real pickers can try.
4. Reject options that address another question.

Exam tip: Use the lowest-fidelity prototype that still answers the question.