Design and judge an A/B test of subject lines

Overview

A sales team sends the same email with two subject lines. Subject line A gets 8 replies, subject line B gets 10. Is B better? Most people say yes. This lesson shows why the honest answer is "we do not know yet", and what a test needs to give a real answer. All companies and figures are invented.

Materials

• Board or projector, and calculators with a square root key.

• A spreadsheet program for the simulation (any will do), one device per pair.

• The planning sheet and the practice cases below, printed or projected.

• Optional for homework: the outreach campaign dataset.

Step by step

1. Hook (10 minutes)

Write on the board: "Subject line A: 200 emails, 8 replies. Subject line B: 200 emails, 10 replies." Ask for a vote and for reasons. Typical answers are "B is 25 percent better" (10 is 25 percent more than 8) or "B has a 5 percent reply rate, A only 4 percent". Leave the answers on the board; you come back to them at the end.

2. Simulation (15 minutes)

Pairs build two columns of 200 cells. Every cell is one email that gets a reply with a chance of 5 percent, in both columns. So A and B are, by design, exactly as good as each other.

A1:  =IF(RAND()<0.05,1,0)     copy down to A200
B1:  =IF(RAND()<0.05,1,0)     copy down to B200
D1:  =SUM(A1:A200)            replies for A
D2:  =SUM(B1:B200)            replies for B

Each pair recalculates five times (F9 in most programs, or edit any empty cell) and notes the two numbers each time. Collect all results on the board.

What the class will see: results anywhere from about 4 to 16 replies, and big differences between two identical subject lines. Mathematically, in about 73 runs out of 100 the two columns differ by 2 or more replies, and in about 42 out of 100 by 4 or more. A gap of 8 against 10 is completely normal for two subject lines that are equally good.

3. The rules of a fair test (15 minutes)

1. One variable. Change only the subject line. Same text, same sender, same day, same list.

2. Random split. Give each contact A or B by chance, for example with a random number column that you sort by. Never split by alphabet, region or "the first 500", because those groups can differ in other ways.

3. Same time. Send both versions in the same time window, so the weekday or a holiday cannot explain the difference.

4. Decide the metric first. Use replies or positive replies. Opens are unreliable: some mail apps load tracking images automatically and others block them.

5. Decide the size and the end first. Fix how many emails and when you will look at the result. Checking every hour and stopping as soon as one version is ahead ("peeking") makes chance winners much more likely.

6. Write the hypothesis down, for example: "A subject line with the customer's company name gets more replies than a generic one."

How big does a test need to be? The table gives rough sizes per variant for a fair chance (80 percent) of detecting a real difference at the usual 5 percent significance level. It is calculated with the standard formula for two proportions.

The message for learners: small improvements need very large tests. A team with a list of 400 contacts can only detect big differences, so it should test bold changes, not one word.

4. Design task (10 minutes)

Fourth Coffee, a fictional coffee roaster, sends a monthly email to the 1,200 offices that already buy from it and asks whether they want to try a new blend. Groups fill in the planning sheet.

5. Worked example: the two-proportion z-test (20 minutes)

The z-test asks: if A and B were really equally good, how unusual would the difference we saw be? The steps:

1. Reply rate of each variant: pA = replies A ÷ emails A, and pB = replies B ÷ emails B.

2. Pooled rate, as if there were one big group: p = (replies A + replies B) ÷ (emails A + emails B).

3. Standard error: SE = √( p × (1 − p) × (1 ÷ emails A + 1 ÷ emails B) ).

4. z = (pB − pA) ÷ SE.

5. If z is 1.96 or more (or −1.96 or less), the difference is called significant at the 5 percent level. Otherwise it could easily be chance.

Example 1: 8 against 10 replies, 200 emails each.

• pA = 8 ÷ 200 = 0.04 and pB = 10 ÷ 200 = 0.05.

• p = 18 ÷ 400 = 0.045, so p × (1 − p) = 0.045 × 0.955 = 0.042975.

• 1 ÷ 200 + 1 ÷ 200 = 0.01, so SE = √(0.042975 × 0.01) = √0.00042975 = 0.0207.

• z = (0.05 − 0.04) ÷ 0.0207 = 0.48.

0.48 is far below 1.96. A difference this large or larger would turn up in about 63 out of 100 tests even if both subject lines were equally good. Decision: no evidence that B is better. Keep either one, or run a much bigger test.

Example 2: 40 against 62 replies, 1,000 emails each.

• pA = 40 ÷ 1,000 = 0.04 and pB = 62 ÷ 1,000 = 0.062.

• p = 102 ÷ 2,000 = 0.051, so p × (1 − p) = 0.051 × 0.949 = 0.048399.

• 1 ÷ 1,000 + 1 ÷ 1,000 = 0.002, so SE = √(0.048399 × 0.002) = √0.000096798 = 0.00984.

• z = (0.062 − 0.04) ÷ 0.00984 = 2.24.

2.24 is above 1.96: significant at the 5 percent level. Decision: use B from now on, and check that the extra replies are not mostly negative ones. Significant does not mean certain: when there is no real difference, about 1 test in 20 still crosses the line by chance.

6. Practice (15 minutes)

Groups calculate z for both cases and write one sentence with their decision.

=(C2/D2-A2/B2)/SQRT((A2+C2)/(B2+D2)*(1-(A2+C2)/(B2+D2))*(1/B2+1/D2))

7. Exit question (5 minutes)

Go back to the votes from the start. Each learner writes two sentences to a manager who wants to switch to B after 8 against 10 replies.

Differentiation

• Easier: skip the formula. Learners compare the real result with the spread they saw in the simulation: "Would this gap have surprised us when A and B were identical?" Give them the spreadsheet formula to calculate z instead of doing it by hand.

• Harder: turn z into a p-value with a normal distribution table or =2*(1-NORM.S.DIST(ABS(z),TRUE)). Discuss multiple testing: a team that tests 10 subject lines against each other, none better than the others, has a 1 − 0.95¹⁰ = 40 percent chance of at least one false "winner". Discuss why the z-test is only an approximation when there are very few replies.

Homework

1. Write a complete test plan (the planning sheet) for a subject line test of your own choice, with a list of 600 contacts. Which size of difference can you realistically detect?

2. In the outreach campaign dataset, 5 of the 45 contacts who got subject line A replied, and 8 of the 45 who got B. Calculate z and write down your decision.