The price of assumptions

Conjoints and consequences series: Part 1

Imagine a business at a crossroads, determined to redefine its value proposition. Nine months before I joined the organization, this was exactly the scenario they faced. The goal was ambitious: change the way outcomes were monetized and establish a new north star. To achieve this, they conducted preliminary quantitative research using conjoint analysis to understand customer preferences and willingness to pay.

Accustomed to a different approach for innovation and optimization, their unfamiliarity with the Conjoint method led them to incorporate it at the wrong time, putting too much weight on early signals, and skipping crucial steps along the way. Enthusiastic about an exciting idea (read: $$$), they charged forward without fact-checking their assumptions. This oversight resulted in missed validation steps, leading to a series of missteps and costly pivots later.

Follow along in this three-part series as we explore the power of the Conjoint method, dissect what went wrong, and uncover strategies to ensure you’re using these powerful methods correctly and effectively.

Recap: Weird survey methods and you

In my previous post “Weird survey methods and you”, we covered that Conjoint is a decision-based survey methodology that asks participants to make choices from a set of product options, simulating real-world purchase decisions.

By analyzing these choices, we can determine the relative value customers place on different features and outcomes.

This helps us build ideal product configurations and model market reactions to changes in configurations, prices, and competitor options.

Mirroring real-world decision making

Participants are presented with various combinations of features and asked to choose their preferred option, much like selecting a product or service in real life. Each choice reflects trade-offs they are willing to make, revealing the features they value most and their willingness to pay for them.

For example, when choosing their next productivity software, a participant might evaluate options based on company, cost, and outcomes related to “efficiency”.

The super-smart assistant behind the curtain

We then analyze the choice data using a statistical technique called multinomial logit. It's like having a super-smart assistant who can see patterns in the data that aren't immediately obvious to us.

Kiki with Curtis

“Kiki with Curtis,” is where our conjoint monster, Curtis Combo will break down statistical techniques in a way we can all understand.

Imagine you have a giant spreadsheet where each row represents a participant's choice, and each column represents a feature like price or outcome. Multinomial logit analyzes how each feature contributes to decisions by analyzing all choices together.

It allows us to directly compare the importance of different features, understand how changes in one feature affect overall choices, and prioritize features with the greatest impact in product development.

Doubling-down on rigor: Refine, predict, and simulate

In this section, we'll dive into additional advanced techniques that give conjoint analysis its power: hierarchical Bayesian analysis, and simulated share of choice. These methods refine our understanding of customer preferences and estimate market reactions to product concepts.

Hierarchical Bayesian (HB) analysis is a statistical method used to refine conjoint results. It borrows strength from the entire dataset to make better estimates for individual preferences. HB analysis recognizes that while each participant’s needs are unique, there are commonalities in how groups of individuals make decisions.

It improves the accuracy of predicting individual choices by considering both individual-level and group-level data.

Multinomial logit models come into play when predicting the likelihood of a particular choice being made. They analyze the chances of a participant selecting a specific combination of features, using the concept of utility. Each choice has a utility value based on the participant's preferences—the higher the utility, the more likely it is that the choice will be made.

It's like weighing the pros and cons of each option and predicting the choice that offers the most overall benefit.

Simulated share of choice uses these models to simulate market behavior. By inputting different combinations of features and prices into the market simulator, we can estimate how much market share each product will capture.

It tells us how our product stacks up compared to competitors without launching expensive real-world tests.

Creating the perfect Conjoint

When done right, Conjoint analysis can provide deep insights into customer preferences, guiding decisions and allowing you to take bigger and better bets with more confidence.

I've witnessed firsthand the incredible power of Conjoint analysis when executed correctly. In fact, one of the studies we’ll explore next, predicted the same implications to market share that took over 80 live A/B experiments to reach the same end result. It can deliver reliable insights without the extensive time and resources required by other traditional methods.

There is no “silver bullet”

Given its power, who wouldn't want to use the Conjoint method to guide their product development? However, just using the right method isn't enough. It’s crucial to apply it correctly, at the right time, and alongside other key elements.

When you're unfamiliar with approaches like Conjoint and hear about their “magic” from various sources, it can be tough to know what information to trust and how to weigh it in your decisions. Understanding how it can enhance product development and knowing when to incorporate it is essential.

There's no silver bullet or quick fix, but the effort is well worth it when done right.

Next time: A tale of two (conjoint) studies

I've often seen situations like this where the focus is on the power of Conjoint. However, the full picture is often missed, especially regarding correct application. Conjoint, like many methods, is just a tool. Its power comes from being applied with precision, consideration, and within the context of the broader ecosystem. It’s my goal to help others navigate these complexities, ensuring you have the insights and tools to succeed.

Next time, we'll dive into the two studies, highlighting what worked and where having a more complete understanding of Conjoint could have led to more informed decisions.

‘Til next time, I’m Bianca

You can find the BriteNote here