Reusable Audience

As a SaaS company, UserTesting faces a common real-world business problem: the end users (UX research and design teams) are not the buyers (Head of Design, Head of Research, CTO…). How do we increase UserTesting’s perceived value from buyer’s point of view, as well as actual customer value?

After getting input from our customers, we see a clear trend, where end-users care a lot about the quality of their own generative research and evaluative research; while buyers care a lot about operationalizing and democratizing research, as a way to ensure UX quality.

How big is the market? After surveying and interviewing our enterprise customers, we realized that our customers can be categorized into 4 quadrants: “the consultants”, “the wild west”, “the gatekeepers” and “the empowered”.  The “consultants” means that the research team is centralized and takes requests from product and design teams. “The wild west” is what we call those companies where anyone in the company can do any form of research and synthesis. “The gatekeepers” are those companies that researchers are embedded within product squads, but they are not a fan of having non-researchers do research to gate for insight quality. “The empowered” are those research teams that focus on generative research for product opportunities, and empower their fellow non-researchers to do their own evaluative research. 

The empowered model is our current target market. Our strategy is to support those advanced research model first, and open it up to other markets while evangelizing research democratization. By investing in “the empowered”, not only can we support quality research for these teams, but also can we help set up industry best practices and evangelize the advanced research model in the long run.

Based on this initiative, the metrics for my squad is to increase the number of “enabled tests”, which means that we need to drive up test creation empowered by researchers.

From needs survey and interviews, we understand that the biggest concern of researchers to democratize research to their team members is low quality of research. If there is a way for them to set up guardrails or educate research novices, they would love to open up evaluative research to their designers or PMs.

To kick off the thinking around the new initiative, I facilitated a design spike, where the team spent 3 hours focusing on sketching ideas and user journeys. The idea of this design spike is to gather all stakeholders on the same initiative to think holistically and align on the top opportunities that we might pursue. I invited my PM partner, qualitative research partner, data science parter, marketer and engineers on my squad. There were 11 people in the spike, and over half of them were fully remote, so I used Mural (online whiteboarding tool) and created several storyboarding and user journey templates to digitalize this collaborative session. (The prompt for the design spike and the Mural templates were later on recycled by other product squads.)

Two great concepts really stood out from the sprint, “UserTesting research community” and “customized test component templates”. (The idea of “UserTesting research community” is around building an online space for research experts, research novices and research curious to be able to interact and learn from best practices. This idea is more aligned with some other marketing initiative, “UserTesting Learning Navigator”, so another squad took it over.)

“Customized test component templates” essentially is a streamlined workflow where research experts could create and save commonly-used test components, aka test plans and audiences, for their research novice team members to use. Looking through our customer feedback channel, this actually has been requested by our customers for the past 3 years.

After the sprint, I quickly created wireframes to illustrate major user journeys, so that the engineering team could have some artifacts to discuss feasibility of the solution and a rough timeline to complete the entire vision. This would help the squad to negotiate the resources and scope of MVP and future milestones.

Meanwhile, to understand more around this opportunity, my qualitative research partner, data scientist partner and I dived deep into quantitative behavioral data around how our users create tests. It was shocking to see that 80% of tests are created with a current feature “create from similar” vs directly starting from scratch. This feature allows user to copy the entire test, including both test plan and audience. We hypothesized that there were 4 scenarios for users to copy old tests when creating a new one:

1. User is conducting a continuous study that could be testing iterations of a design or product, so they are reusing the same test plan and audience.
2. User is testing the same design or product across different devices.
3. User only wants to reuse part of the original test, either the audience or the test plan, but right now there is no sufficient way to copy just one part.
4. User is doing advanced counter-balancing study, which is not a common use case.

With these hypothesis in mind, we sampled 100 pairs of “create-from-similar” tests to understand trends. Only around 25% of those “create-from-similar tests” were launched with the same audience and test plan that were copied from the original tests, which is case #1 and #2. We didn’t see an example of #4 in the samples. The majority of “create-from-similar” behavior was meant to only re-use part of the test, specifically audience. This is fairly easy to understand, since the target audience or persona of a company or a product area rarely shifts too much. Is it safe to assume that across our customers? We needed to validate it with researchers in “the empowered” group.

Templating test component could mean a lot of things. Do customers care more about reusing test plan or audience? What level of reusability should we consider? If it is audience, is it to save single screener question, e.g. popular job function question? Or should it be a group of questions, e.g. online shopping habit questions? Or it could be an entire audience, including both demographic filters and screener questions? My research partner Anthony, PM partner Leigh and I surveyed 20 customers around their preferred level of reusability. Literally all of them marked high on reusing the entire audience. Our hypothesis was validated.

“Can we make templates for personas (i.e. Freelancers) and mix them with templates for tasks (i.e. benchmarking studies)? Right now, I can only make a bunch of drafts for my team, and then we use the drafts as a template library.”

MW, UX Researcher @ Microsoft

I wish there was a way to save and organize screener questions and demographics as a persona preselects, because we use the same ones over and over again… Right now we have to comb through screener questions documented on google docs and select those 20 states that our hospitals are in, or 12 states that our Urgent Care’s are in over and over. =(

CM, UX Lead @ HCA Healthcare

Now that it is clear that saving audience as template is a desired outcome, designing the workflow is quite straightforward. The most important use case is for research experts to save the audience when creating it in study launcher.

What first came to my mind was to allow user to save new audience as well as update existing one (shown on the left). The mental modal behind this design is that an audience (e.g. Brick and Mortar) would be a lego piece, so that research novice would be able to plug and play when creating their own test.

After showing this to engineering team, tech lead partner Suan told me that being able to update existing audience requires real-time tracking and updating data base, which could be fairly complicated and time-consuming. If we could omit tracking the subject and only having the “save as new” functionality, it would be much simpler from backend point of view. I started to think of other solutions, since I’d like to unblock my squad and be able to deliver customer values faster.

In this iteration, Iswitched the mental model into considering those templates only a starting point for user to create test. When they are in study launcher, they are not able to do any edits to the templates real time. In this way, it does reduce a lot of technical and UX complexity.

I tested this design to real users, and it totally worked out just like the previous version. So the team decided to move on with this version of design.

Another important JTBD is for novices to select the saved audience during test creation. My assumption was that there needs to be a lot of information around the audiences to be presented at different levels to help the novices make a choice. Below were the iterations I’ve tried.

After testing, it turns out that most users were fine with choosing the right audience just by looking at the title; description that their research experts set up is helpful too. Additional information around when and who created the audiences could be helpful, only in scenarios where the user forgets which ones to use, or have some side questions about the research. So I decided to go with the first design, and change the icon color to blue, which aligns with our design system.

We also discovered a bonus for this new approach where the user could select multiple audiences: they naturally understand that they could use multiple audiences within one test now. It was hard to discover that in the previous interface.

Another important JBTD is for research experts to be able to organize, edit and delete these saved audience templates. Last year there was another squad launched “screener question templating” feature without providing this management functionality, and soon customers found themselves buried in infinite duplicates and stopped using that feature. We definitely learned the lesson this time, and want to make sure that the management feature is sufficient in their workflow.

Feel free to check out the prototype (WIP happy path): https://usertesting.invisionapp.com/share/ARTC9FDZP8B

Originally, we split the project into 2 milestones: MVP (where research expert could save and delete audiences) and M1 (where research expert could also edit and duplicate audiences).

During MVP Beta, we were seeking an answer to the question: would customers be creating audiences across test types, or they could be satisfied with test-type-specific audiences?  Desirability-wise, we knew that we should deliver universal audience, e.g. user could test Brick and Mortar for Live Conversation as well as unmoderated tests; feasibility-wise though, it’s really hard to distribute to testers, that’s why we took the easy route in MVP and only allowed user to create audiences for specific test types. As expected, feedback flooded in around not being able to find the audience in other test types. It was not intuitive at all for user to understand why they need to duplicate the same “persona” for all test types. Apparently we need to invest more to fix this problem now, which would be delivered in M3.

Other than that, qualitative feedback has been positive in general. Within 10 min after Beta release, we had a customer shouting us out when he was saving an audience for designers on his team to reuse.

Update: In Q3 2019, Saved Audience went GA. The old workaround “creating from similar tests” (essentially coping all metadata from a previous test) has drastically declined ever since. Customers lean toward this new way of reusability with intentional decisions. And the success of project leads us to explore more in reusability world. “Reusable Test Plan” is in build phase and Beta right now. Please stay tuned.

(Disclaimer on June 20, 2019: as the project continues, I will definitely gain more insights and reflect more on the process.)

So far what really struck me is how important to locate the right opportunity for the right market. The whole team and I feel thrilled when working on solving a real problem for the users. The moment of hearing customers saying YES was the peak of team moral.

Initially we took quite a detour when trying to achieve our original metric, “increase number of launched tests”. It was too vague and too generic. Without a clear top-down vision, the team got pulled left and right, experimenting on totally different opportunities led by different opinions. We could’ve made up our mind and chose one opportunity to work on, instead of being spread thin and trying to accomplish more than we could. Luckily leadership specified our squad’s main focus from “everything that UserTesting core platform does” to “scale research in empowered organization”. So that our metric became “to increase enabled tests”, which is more defined and actionable. From this experience, I’ve definitely enjoyed the freedom to examine what my team should work on. Also I sense the importance of having a clear aligned vision across product areas more, which is great insight for future leadership role.