Product-market fit pipeline: Make the most of your PMF surveys with Python
Closing the feedback loop using Intercom, Typeform, Airtable, pandas, Plotly and Segment.
Product-market fit is “the moment when a startup finally finds a widespread set of customers that resonate with its product” (Eric Ries). This moment is hard to measure, and most, like Marc Andreessen, suggest you should just “feel” it.
It can be disconcerting to blindly wait for a feeling you haven’t experienced before. To give us some guidance, the father of growth hacking Sean Ellis proposed a way to measure your product market fit using a simple survey:
Just ask a user, “how would you feel if you could no longer use the product?”
Based on Sean’s experience running growth at Dropbox, LogMeIn, and Eventbrite, and comparing results at other companies, achieving product-market fit requires at least 40% of users saying they would be very disappointed without your product.
Sean’s thesis eventually evolved into a time-tested PMF Survey (formerly survey.io) that not only evaluates product-market fit, but also identifies how to improve it.
Even if you are skeptical about the 40% threshold, or confident that you have a product-market fit already, sending out regular PMF surveys and tracking results over time will be invaluable to prioritize your roadmap and evidence your company’s progress to stakeholders.
A lot has been written about the methodology, and how it is used at companies such as Buffer, and Superhuman.
I’ll assume you’re already convinced you’d like to know your PMF. In this post I’ll show you how to distribute, analyze, and act on your PMF Survey.
I will show you how to:
- distribute your survey with Intercom and Typeform,
- ingest and label responses in Airtable,
- enrich them with demographic and usage data,
- segment and visualize them with Python and Plotly,
- and finally channel them to other services using Segment.
Prepare the PMF survey
Although Sean Ellis has his own official website that you can use to send out the survey, I recommend you use Typeform instead.
First, Typeform lets you customize the prompts as you learn more about your users. For example, if you’re targeting teens with your product, you may want to replace LinkedIn with Discord in the “How did you discover us?” question.
Second, Typeform offers official an integration with Airtable, which we’ll be using to label our data.
The questions
I will not go into the details of creating a Typeform survey, but below are the PMF Survey questions in a format that’s easy to copy over. When creating the prompts, make sure to mark them all as Required.
Typeform’s Airtable integration does not yet support the Other option in multi-choice questions, so you’ll have to implement a custom solution with Logic Jumps instead (indented in the list above):
You will also need to add a Hidden Field called user_id, which we will pass down from Intercom to connect responses to additional customer data (more on that below).
Connecting to Airtable
I’ve prepared an Airtable template that you can easily copy and use for your own survey.
You can then follow official instructions to connect your Typeform to Airtable, and once you get to field mapping step, use the following setup:
Send the survey through Intercom
It’s best to send out a PMF survey on a regular schedule, every 3-6 months.
It is unfortunately impossible to set up a schedule like this in Intercom, so you’ll have to trigger a new campaign manually every time. Alternatively, you could create a cron job on your own server that dispatches the survey through SendGrid, or your transactional email provider of choice.
You want to only get feedback from people who have recently experienced ‘real usage’ of the product.
Depending on your business, this could be customers who ordered a meal, made a sale, listened to at least one song, or created a new document in, say, the last 30 days.
The more recent the users, the better, but aim to survey enough customers that’ll give you at least 300 responses, to have have some flexibility in segmenting the data without giving up on statistical significance.
I won’t go into the details of creating a new Intercom campaign, but in my experience, it’s best to keep the message simple and casual, and make it clear the survey won’t take long to complete.
There is no need to offer any kind of reward — this could bias results.
Passing the user ID to Typeform using hidden fields
When you link to your survey, make sure to append #user_id={{ user_id }} at the end of the URL.
This will pass the user ID from Intercom down to Typeform, and allow us to enrich our survey results with customer data from other sources.
Label customers in Airtable
Next, we’re going to add short and consistent tags based on the long-form answers in each.
This step is laborious, but indispensable to make the most out of your survey results.
I’ve seen one too many teams send out a PMF survey, check they’re above the magical 40%, take but a quick glance at full-text responses, and forget it. The only thing that benefits from this exercise is their ego.
Just as the true power of the Net Promoter Score (NPS) lies not in a single number, but trends and follow-ups to respondents, the PMF survey is worthless unless you segment, dig deep, and continuously act on your findings.
It can be tempting to outsource labeling to a Mechanical Turk, but I would strongly recommend this is done by a core member of the team responsible for the product:
- Respondents often use imprecise language, abbreviations, and even metaphors. A good understanding of your business domain is therefore indispensable for this task.
- Carefully reading through every response will give you a visceral understanding of your customers, and their pain points, which will unconsciously guide your future decisions.
The first two fields you’ll have to process are Source: Other and Alternative: Other.
Simply go through the write-in responses and convert them into standardized labels in the Source and Alternative columns.
For example, this:
Becomes:
Tip: If you find the same Source keeps appearing in Other, consider adding it to the multi-choice in the survey.
Next, you will summarize long-form responses in the Benefit, Request, and Persona columns. People tend to mention several benefits and suggestions, and you will add a separate tag for each.
I would recommend you focus on Benefits mentioned by Very disappointed (high product-market fit) respondents, improvement requests by on-the-fence Somewhat disappointed respondents, and disregard the rest.
This will help you maintain your core customer value, pick the right features to work on next, and last but not least, save you time tagging hundreds of records.
Process responses in Python
Now we’ve labeled all our data, it’s finally time to do some data crunching and visualization.
Below, I will assume you have a working knowledge of Python, pandas and Jupyter notebooks.
If you can’t code, and have a dedicated data scientist on your team, you can let them take over from here.
If not, I highly recommend you don’t outsource this to your engineering team, and instead try to explore the data in Excel, Tableau, or another tool you’re more comfortable with.
Loading the data from Airtable
Instead of exporting a CSV and reading the file into Python, we can load the data directly with the Airtable API.
To get started, first install the Airtable Python wrapper (remove the ! if you’re running this in a terminal, rather than a Jupyter cell).
You can now import pandas, and the Airtable library:
Set your base ID, table name, and Airtable API key:
And fetch the data:
Enrichment with user traits from Intercom
PMF survey responses are most useful when segmented based on customer demographics and past purchasing habits.
Below, I assume that you have this data available as user attributes in your Intercom. You could of course import it from any other software, or directly from your SQL warehouse.
To get started, follow the official instructions to create a new Intercom app.
The Intercom Python library is unfortunately no longer maintained. To access your customer data, you’ll therefore need to downgrade your app’s API version from 2.0 to 1.4 (2019-09-26).
You can do this from the Configure > Basic information screen, after creating the app.
Next, install the library:
Initialize your Intercom client:
And finally enrich your responses with Intercom location data and attributes:
Exploratory data analysis with Plotly
The possibilities for segmenting and data exploration are practically endless, depending on your business, and the data you loaded up from your other sources.
Below, I’ll share but a few basic examples to get you started.
I’ll be using Plotly to visualize the data, but feel free to use seaborn or Altair instead.
Compare survey distributions to your overall customer base
There’s a good chance that your data is slightly biased.
For example, if you serve customers across demographics, you may get a disproportionate number of responses from teenagers on a holiday compared to busy professionals.
To keep this bias in mind when analyzing results, I like to plot the survey’s distributions, and compare them to those of all users.
Or, if you have population data at hand:
Plot product-market fit over time
You can use the rangeslider_visible setting to output an interactive plot which makes it easy to see how your PMF evolved over time:
Segment, segment, segment
Ever wondered how your product-market fit compares by country?
Or whether frequent customers are more likely to know of a substitute to your product?
The possibilities for exploration are endless, and highly specific to your business, and the demographic and usage data you have available.
All this data can become overwhelming. I therefore like to document all my findings, as I go, in a separate Notion document.
There, I include the questions I’ve asked, image exports of relevant data, and conclusions I drew for the business.
Updating user analytics with Segment
The final step is to upsert your new survey data to Intercom, and other services.
This way, you’ll be able to quickly message all customers that requested a particular feature once you release it, schedule a call with customers who agreed to answer follow-up questions, or create lookalike audiences to target your Facebook Ads.
You could push the data to Intercom using the library we used above. Here, I will show you how to do this with Segment, which will make the data available to all the different services you use to analyse and communicate with your customers.
First, install the Python library:
Import it and set your Segment write key:
Enable debugging messages:
And identify users one by one with your new survey data:
You could also track a Segment event in the same function, such as “PMF Survey Completed”.
To backdate your track and identify calls, you can add a timestamp argument to both functions, set to the Date column from your survey data.
What next
You’ve come a long way, and after doing all this analysis, you’re probably teeming with ideas for how you can improve your product.
How do you pick what to focus on first?
This questions is too big to answer here, but one approach is to spend half your time doubling down on what Very disappointed users have loved, and half — on requests by respondents who were on the fence.
To quote Superhuman, who use PMF Surveys to prioritise their roadmap:
Vision-driven teams are apt to double down on what users love. Feedback-driven teams are apt to double down on addressing feedback. But it turns out that neither extreme is optimal.
From there, the most important thing is to keep running the surveys at a regular schedule, and track how your product changes have affected results for different segments.
You should also try to set up calls with at least a couple of customers every week. You’d be surprised how much you can learn from even just a 15 min casual chat with one of your users. (If you want to get better at user interviews, this book is a must-read: The Mom Test: How to Talk to Customers and Learn If Your Business is a Good Idea when Everyone is Lying to You.)
Finally, if you’re part of a larger organization, be sure to share this data with your management, sales and marketing teams, who can use it to steer the company vision, or optimize ad campaigns.
Liked this post?
Follow me here on Medium and Twitter @seifip.
Read all the way here, but still need help?
Get in touch at philip@seifi.co to learn how I can help you achieve product-market fit through data analysis, and other product management consulting services.
