The setup
Every product team has asked its users why they left. Most have done it with a survey, because a survey scales and an interview does not. We wanted to know what that trade costs, so we asked the same twelve questions two ways.
640 users who had churned in their second month. Half got a twelve-question survey with free-text fields. Half got a six-minute conversation with ProductSignal that opened with the same twelve questions and followed up wherever the answer was thin.
What surveys got
Surveys came back fast. 71% completion, median four minutes. The answers were clean and short, and almost all of them named a reason we already had on the roadmap: price, missing integrations, “didn’t have time.”
Which is the problem. When you give someone a text field under a question about pricing, they write about pricing. We got 640 confirmations of what we already believed, and very little we didn’t.
What interviews got
Interviews took longer to come back, and fewer people finished: 58% completion, median six minutes. But the content was different in kind, not just in length. Two in three transcripts contained something the survey never surfaced.
“Honestly, that they’d see the board. Instead they got an empty workspace.”
The empty first screen came up 92 times. It was not one of our twelve questions. It was not on the roadmap. It became the most-quoted pain of the study, and the first thing we fixed.
The second question
The difference is almost entirely the follow-up. When someone answers “I didn’t have time,” a survey moves on. An interviewer asks what they were trying to do when they ran out of it. That second question is where the useful answers live, and it is the one thing a form cannot ask.
We looked at where in each transcript the empty first screen first came up. In 84% of cases it was in response to a follow-up, not to one of the original twelve questions.
What we changed
We stopped treating the twelve questions as the study and started treating them as the opening. ProductSignal now spends most of an interview on follow-ups, and stops when it understands rather than when the list is done. Surveys still have a place, for counting things you already know how to name. For finding the things you don’t, ask the second question.