At a glance
- Role
- Product Manager, owned the polling feature and the campaign
- Team
- Me, as product manager, designer and engineer
- Timeline
- Q4 2025, one test over about two weeks
- Users
- Patients on a rare disease health platform
- Metrics
- Poll participation 31% with the simpler email against 18% with mine, across about 1,200 patients
Problem and stakes
Community polls let patients answer one quick question and see how others on the platform answered. The feature was new, and its adoption depended on getting people to come into the platform for it. For most patients the first thing they would see was an email.
My hypothesis was that a longer, more illustrative email, one that explained what the poll was for and what the community would learn from it, would bring more people in. The counter-view was that a short, plain email would do better. We could have argued it. I set up a test instead.
What I did
-
01
Tested the message before the campaign, not after
The poll email was the first of a series. Getting the tone right once meant every later message could reuse it, and getting it wrong would have been repeated across the whole series.
Tradeoff. The campaign started about a week later than it could have. In return, every email after it was built on a result rather than a guess.
-
02
Put my own hypothesis in as Version A
If I had only tested variations of the simpler email, I would have carried my assumption into the series untested. The point of the test was to find out whether I was right, so my version had to be in it.
Tradeoff. It is uncomfortable to lose your own test in front of the team. That discomfort is the price of a result nobody has to argue about.
-
03
Measured participation, not opens
An email can be opened and ignored. The feature only wins if the patient actually answers the poll, so the test counted patients who completed one, not patients who clicked.
Tradeoff. A stricter metric means smaller numbers and a slower read. It also means the number we shipped on was the one that mattered to the feature.
The test
Patients were split evenly between the two emails, with everything the same except the copy. The hypothesis I was testing was mine.
My hypothesis was that a longer email explaining what the community would learn from the poll would get more patients to take part than a short one.
"Every answer in a community poll adds to a picture of life with your condition that no clinic visit can capture. The results go back to the community, so you can see how others are managing the same things. We would like to know what matters to you this month. The poll takes two minutes."
"We are asking patients one question this week. Tap below to answer."
Result. The simpler version won by a clear margin: 31% of patients who got it answered the poll, against 18% for mine, across about 1,200 patients. I was wrong about what patients wanted to read, so we shipped the simpler messaging and used that tone for the rest of the lifecycle series.
What went wrong
My first hypothesis was wrong, and I had been ready to build around it. If I had shipped the lifecycle emails without testing the messaging first, the series would have carried the longer, more explanatory tone that patients ignored. The A/B test was cheap and it changed the whole series. The lesson I keep from it is to put my strongest opinion into a test first, especially when I feel sure.
Results
- The simpler poll email outperformed my own version, 31% participation to 18%.
- The simpler tone shipped and became the standard for the rest of the lifecycle email series.
What I learned
Put your strongest opinion into a test first. The more sure you feel, the cheaper it is to find out.
What I'd do next
- Test whether showing one community result inside the email itself lifts participation further than the plain ask.
- Run the same two-version test on the in-app prompt for polls, to see whether the simpler tone holds outside email.
Why this matters
Growth and lifecycle roles live on experiments. This is a small one, but it shows the habit: a clear hypothesis, a fair test, a metric tied to the feature rather than the email, and the willingness to ship the result that beat my own idea.
Next case study
AI in the product workflow
I designed, shipped and ran four agent workflows that the whole company picked up, saving five hours of manual triage a week and cutting triage cycle time by 17%.
- AI agents
- Internal tools
- Human in the loop
- Process design