The story behind the data
Fabrikam, a fictional maker of warehouse shelving, had a stand at a trade fair in late August 2026. 90 visitors left their business card and agreed to receive information. Fabrikam then sent them a sequence of up to three emails:
Step 1 on Tuesday 8 September 2026,
Step 2 on Monday 14 September,
Step 3 on Tuesday 22 September.
Each contact was randomly given subject line A or B for all of their emails, 45 contacts each. The sequence stopped for a contact after a reply, an unsubscribe or a hard bounce. A soft bounce did not stop it: the next step was still sent.
The file fictional-outreach-campaign.csv (in the Downloads on this page) has one row per email sent. Every company, domain and figure is invented; all domains end in .example, which can never be a real address.
Data dictionary
Column | Spreadsheet column | Type | Meaning |
|---|---|---|---|
send_id | A | text | One email, S0001 to S0238, in the order of sending |
contact_id | B | text | The person it was sent to, C001 to C090 |
sent_at | C | date and time | When it was sent, sender's local time, as YYYY-MM-DD HH:MM |
step | D | number | Position in the sequence: 1, 2 or 3 |
variant | E | text | Subject line A or B |
recipient_domain | F | text | Domain of the recipient's company, for example pine-labs.example |
industry | G | text | Logistics, Manufacturing, Retail, Software, Hospitality or Healthcare |
country | H | text | Germany, Austria, Netherlands, United Kingdom, France or Ireland |
delivered | I | 0 or 1 | 1 if the receiving server accepted the email |
bounced | J | 0 or 1 | 1 if it bounced |
bounce_type | K | text | hard, soft or none |
opened | L | 0 or 1 | 1 if the tracking image loaded (unreliable, see task 6) |
replied | M | 0 or 1 | 1 if a person answered this email |
reply_sentiment | N | text | positive, neutral, negative or none |
unsubscribed | O | 0 or 1 | 1 if the recipient unsubscribed from this email |
meeting_booked | P | 0 or 1 | 1 if the reply led to a booked meeting |
Rules the data keeps
Every row is either delivered or bounced:
delivered + bounced = 1.A bounced email has no opens, replies, unsubscribes or meetings, and its
bounce_typeis hard or soft. A delivered email hasbounce_typenone.reply_sentimentis none exactly whenrepliedis 0.A meeting is only booked after a positive reply.
Each contact replies at most once, because a reply ends the sequence.
variant,recipient_domain,industryandcountrystay the same for all emails to one contact. Some contacts share a company domain.
Facts to check against
238 emails to 90 contacts at 81 company domains: 90 in step 1, 78 in step 2, 70 in step 3.
227 delivered and 11 bounced (6 hard, 5 soft).
13 replies: 4 positive, 5 neutral and 4 negative. 3 meetings, 3 unsubscribes.
How to work with it
SQL: open the SQL Query Playground, load the CSV and query the table
data. Nothing is uploaded. In other SQL tools, import the file as a table calleddataand the queries below work unchanged.Spreadsheets: open the CSV, select all, insert a pivot table. The 0 and 1 columns can simply be summed: the sum of
repliedis the number of replies.Each contact has exactly one step 1 row, so filtering on
step = 1gives one row per contact. That is the easy way to count contacts in a pivot table.
Tasks
Get to know the data. How many emails, contacts and company domains are in the file? How many emails were sent in each step?
Delivery and bounces. Calculate the delivery rate and the bounce rate of all emails. How many bounces were hard and how many soft, and in which steps? Why are all hard bounces in step 1?
Reply rate per step. For each step, calculate replies ÷ delivered emails. What does the result say about sending step 2 and step 3?
Reply rate per variant. What share of contacts replied, for subject line A and for B?
What did people say? Count the replies by variant and sentiment, and the meetings and unsubscribes by variant. Does the picture from task 4 still hold?
Opens and replies. Calculate the open rate (opened ÷ delivered) per variant. Find the emails that got a reply but were never counted as opened. How is that possible?
Industries. Calculate the share of contacts who replied for each industry, highest first. Which industry would you target next, and how much do you trust this ranking?
Countries. Build a pivot table of emails, bounces and the bounce rate per country. What could explain the differences, and what should Fabrikam check?
Who left the sequence early? Find every contact who got fewer than three emails, and explain why for each group.
Is B really better? Use your numbers from task 4 for a two-proportion z-test, as in the lesson on A/B tests. Is the difference significant at the 5 percent level?
Solutions
The results below were calculated from the file with a script and checked in the SQL Query Playground and in SQLite.
Solution 1: get to know the data
SELECT COUNT(*) AS emails, COUNT(DISTINCT contact_id) AS contacts, COUNT(DISTINCT recipient_domain) AS domains
FROM dataSELECT step, COUNT(*) AS emails
FROM data
GROUP BY step
ORDER BY step238 emails, 90 contacts and 81 domains. Step 1: 90 emails, step 2: 78, step 3: 70. In a spreadsheet: a pivot table with step as rows and the count of send_id as values.
Solution 2: delivery and bounces
SELECT COUNT(*) AS sent, SUM(delivered) AS delivered, SUM(bounced) AS bounced,
ROUND(100.0 * SUM(delivered) / COUNT(*), 1) AS delivery_rate,
ROUND(100.0 * SUM(bounced) / COUNT(*), 1) AS bounce_rate
FROM dataSELECT step, bounce_type, COUNT(*) AS emails
FROM data
WHERE bounced = 1
GROUP BY step, bounce_type
ORDER BY step, bounce_type227 of 238 emails were delivered: a delivery rate of 95.4 percent and a bounce rate of 4.6 percent. 6 hard bounces, all in step 1 (6 of 90 step 1 emails, 6.7 percent), and 5 soft bounces: 2 in step 1, 1 in step 2 and 2 in step 3.
All hard bounces are in step 1 because a hard bounce ends the sequence: an address that does not exist gets no step 2. Addresses from business cards often contain typos, and some visitors had already left their company.
Solution 3: reply rate per step
SELECT step, COUNT(*) AS sent, SUM(delivered) AS delivered, SUM(replied) AS replies,
ROUND(100.0 * SUM(replied) / SUM(delivered), 1) AS reply_rate
FROM data
GROUP BY step
ORDER BY stepStep | Sent | Delivered | Replies | Reply rate |
|---|---|---|---|---|
1 | 90 | 82 | 6 | 7.3 percent |
2 | 78 | 77 | 6 | 7.8 percent |
3 | 70 | 68 | 1 | 1.5 percent |
Step 2 brought as many replies as step 1, so the first follow-up was clearly worth it. Step 3 brought only one reply. Note that later steps go only to people who have not answered yet, so the groups are not the same.
Solution 4: reply rate per variant
SELECT variant, COUNT(DISTINCT contact_id) AS contacts, SUM(replied) AS replies,
ROUND(100.0 * SUM(replied) / COUNT(DISTINCT contact_id), 1) AS reply_rate
FROM data
GROUP BY variant
ORDER BY variantA: 5 of 45 contacts replied, 11.1 percent. B: 8 of 45, 17.8 percent. Contacts are the right base here, because each contact replies at most once, however many emails they got.
In a spreadsheet: count the contacts with a pivot table filtered on step = 1 (45 per variant), and the replies with a pivot table of all rows, variant as rows and the sum of replied as values.
Solution 5: what did people say?
SELECT variant, reply_sentiment, COUNT(*) AS replies
FROM data
WHERE replied = 1
GROUP BY variant, reply_sentiment
ORDER BY variant, reply_sentimentSELECT variant, SUM(meeting_booked) AS meetings, SUM(unsubscribed) AS unsubscribes
FROM data
GROUP BY variant
ORDER BY variantVariant | Positive | Neutral | Negative | Meetings | Unsubscribes |
|---|---|---|---|---|---|
A | 2 | 3 | 0 | 2 | 1 |
B | 2 | 2 | 4 | 1 | 2 |
B's extra replies are negative ones. Both variants got 2 positive replies (4.4 percent of contacts each), and A even led to one more meeting. Judged by the goal, meetings, B is not better.
Solution 6: opens and replies
SELECT variant, SUM(delivered) AS delivered, SUM(opened) AS opened,
ROUND(100.0 * SUM(opened) / SUM(delivered), 1) AS open_rate
FROM data
GROUP BY variant
ORDER BY variantSELECT send_id, contact_id, step, variant, reply_sentiment, meeting_booked
FROM data
WHERE replied = 1 AND opened = 0Open rate: A 45 of 115 delivered emails, 39.1 percent; B 63 of 112, 56.3 percent. Three emails got a reply without ever being counted as opened: S0092 (C087), S0150 (C016) and S0167 (C036). S0167 even led to a meeting.
An open is only counted when the tracking image loads. Mail apps that block images hide real opens; others load images automatically and create opens nobody made. So B's much higher open rate says little: what counts is replies and meetings.
Solution 7: industries
SELECT industry, COUNT(DISTINCT contact_id) AS contacts, SUM(replied) AS replies,
ROUND(100.0 * SUM(replied) / COUNT(DISTINCT contact_id), 1) AS reply_rate
FROM data
GROUP BY industry
ORDER BY reply_rate DESCIndustry | Contacts | Replies | Reply rate |
|---|---|---|---|
Healthcare | 3 | 1 | 33.3 percent |
Software | 15 | 3 | 20.0 percent |
Manufacturing | 19 | 3 | 15.8 percent |
Logistics | 26 | 4 | 15.4 percent |
Hospitality | 13 | 2 | 15.4 percent |
Retail | 14 | 0 | 0.0 percent |
Logistics and Hospitality tie at 15.4 percent, so tools may list them in either order. Healthcare looks best, but it is one reply from three contacts: one reply more or less moves the rate by 33 points. The ranking is far too uncertain to decide on. A good answer says so and suggests collecting more data, for example from the next trade fair, before targeting an industry.
Solution 8: countries
SELECT country, COUNT(*) AS emails, SUM(bounced) AS bounced,
ROUND(100.0 * SUM(bounced) / COUNT(*), 1) AS bounce_rate
FROM data
GROUP BY country
ORDER BY bounce_rate DESCCountry | Emails | Bounced | Bounce rate |
|---|---|---|---|
Austria | 23 | 2 | 8.7 percent |
Netherlands | 52 | 4 | 7.7 percent |
Germany | 64 | 3 | 4.7 percent |
Ireland | 36 | 1 | 2.8 percent |
United Kingdom | 39 | 1 | 2.6 percent |
France | 24 | 0 | 0.0 percent |
The counts are tiny: Austria's rate comes from 2 bounces. Possible causes worth checking are typos when the business cards were typed in, or contacts who had already changed jobs. Fabrikam should look at the bounced addresses themselves rather than conclude anything about countries.
Solution 9: who left the sequence early?
SELECT contact_id, COUNT(*) AS emails, MAX(bounced) AS bounced, MAX(replied) AS replied, MAX(unsubscribed) AS unsubscribed
FROM data
GROUP BY contact_id
HAVING emails < 3
ORDER BY emails, contact_idSELECT step, SUM(bounce_type = 'hard') AS hard_bounces, SUM(replied) AS replies, SUM(unsubscribed) AS unsubscribes
FROM data
GROUP BY step
ORDER BY step20 contacts got fewer than three emails; the other 70 got all three.
One email (12 contacts): 6 hard bounces in step 1 (C009, C042, C047, C065, C071, C072) and 6 replies to step 1 (C002, C010, C033, C045, C057, C059).
Two emails (8 contacts): 6 replies to step 2 (C016, C036, C054, C062, C082, C087) and 2 unsubscribes in step 2 (C028, C049).
The reply and the unsubscribe in step 3 end nothing, because step 3 is the last email anyway. In SQLite and in the playground, bounce_type = 'hard' counts as 1 when true and 0 when false, so SUM counts the hard bounces.
Solution 10: is B really better?
From task 4: A 5 of 45 (0.1111), B 8 of 45 (0.1778).
Pooled rate: 13 ÷ 90 = 0.1444, so p × (1 − p) = 0.1444 × 0.8556 = 0.1236.
1 ÷ 45 + 1 ÷ 45 = 0.04444.
Standard error: √(0.1236 × 0.04444) = 0.0741.
z = (0.1778 − 0.1111) ÷ 0.0741 = 0.90.
0.90 is far below 1.96, so the difference is not significant at the 5 percent level; a gap this large would appear in about 37 of 100 tests even if both subject lines were equally good. With so few replies the z-test is only a rough approximation anyway. Together with task 5 (more negative replies for B, no more meetings) the honest conclusion is: this test cannot tell which subject line is better.
License
The data is released into the public domain under CC0 1.0. Use it however you like, no credit needed.
