Last week we conducted a completely NON-scientific poll in which we asked people involved in “customer feedback” programs to tell us how they refer to their effort. The question was posed to readers of our blog and through LinkedIn groups that cover “customer feedback” in some way.

It’s interesting to see that most people took the time to click through and look at the 1-question poll and then didn’t respond. Also interesting to see from related research that while most people use Net Promoter they generally don’t refer to their program that way.

We won’t infer anything from this simple poll, although perhaps our industry can start rallying behind the most cited “Customer Experience” moniker… thoughts?

Blog 3 in 3 Part Series on Analysis of Bias-Filled Data

Visiting a city for three days does not give one enough information to make claims about its country’s weather. It is just as dangerous to make conclusions from customer experience feedback without treating the bias that may lie within. In the first post of this series, I discussed different types of bias and particularly the importance of self-selection bias in customer experience data. In the second post, I offered tips to pre-treat your survey to increase response propensity and identify underlying bias. Today, I will share techniques to adjust your data for this bias in order to minimize its effect on your survey results.

Most of the adjustment techniques common in customer experience surveys center around pinpointing which groups are under-represented in the data and assigning weights to these groups to adjust for their lack of response. The weight is the ratio of the representative count of one subgroup (given from a census or known population parameter) to the actual count. Say you have 100 respondents in a customer survey, but 75 of them are women and 25 of them are men. If you wanted to use this data to generalize to the larger population (or make predictions about future customers), you could multiply all the data for men and for women by the following weights:

Weight (men) = Representative Count (men) / Actual count (men) = 50 / 25 = 2

Weight (women) = Representative Count (women) / Actual count (women) = 50 / 75 = 0.67

While this is a common method (and certainly useful), it has a number of limitations, chief among them the inability to weight for multiple variables simultaneously. Logistic regression is a useful technique in this regard as it can evaluate the relative importance of a large number of independent variables to survey response (the dependent variable). Several other techniques utilize logistic regression in order to correct a predictive model for self-selection bias, including sample selection modeling and Heckman correction modeling. The idea in both of these approaches is to create two models: one predicting survey response (the response model) and one predicting some key outcome (the outcome model).  The response model’s regression coefficients are used to correct the outcome model for selection bias. These tools have been established and validated in both academic journals and industry practice.

We took this approach out for a spin with a dataset from a recent client and found some interesting trends. A response propensity (RP) score was calculated for each contact (respondent and non-respondent) in our contact base, based on logistic regression coefficients from the response model. Three segments of contacts were created: those below, at, and above the median RP. The survey data from respondents from each segment were analyzed for differences and while our results are still preliminary, we see definite distinctions for certain questions. The plot above shows that the High-RP group (the contacts who were defined statistically to be likeliest to respond) actually have a lower rating for Ease of Doing Business than the Low-RP group (those contacts defined to be least likely to respond). Without using an adjustment described above, our overall Ease rating will be pulled downwards by the fact that the Low-RP group is so under-represented. Your mileage may vary of course – the simplest way of avoiding this problem is by raising the response rate in the least-likely group.

What do you think? Do you use any other techniques for adjusting for self-selection and other biases?

Blog 2 in 3 Part Series on Analysis of Bias-Filled Data

Though most people associate the ability to predict the future with their neighborhood fortune-teller, customer experience practitioners are often in the business of forecasting customer behavior. Different flavors of regression models exist that do a great job at this, using current customers’ survey responses to gain insight into how later customers will act.

Unfortunately, self-selection bias (a form of systematic bias outlined in the first blog in this series) violates one of the classical assumptions of regression modeling – that your sample is representative of the population in question. So how does one tackle this issue before it enters the data? And when it’s there, how can the practitioner identify its presence before starting any data analysis?

The lessons learned here all center around the concept of response propensity (RP), which is a customer’s likelihood of responding to the survey. This can be based on, among other things, cultural/geographical factors and communication hindrances (whether this customer is likely to be responsive to email or inundated by their inbox, for example).

Much like pre-treating stains on laundry before tossing it in the washer, pre-treating your survey design to account for differences in RP can result in a cleaner dataset. Though RP is usually calculated after a survey has been administered, drawing insight from past surveys can tell you how this is distributed within your survey’s population. Have you found that Decision Makers are less likely than End Users to respond to surveys? Perhaps this group needs targeted reminders sent to them with language emphasizing the importance their responses hold. Or maybe past projects have shown that one region has a particularly low response rate which contributes to its members’ tiny RPs. Offering incentives personalized to this demographic could yield the response rates you need to use your responses to predict future ones.

Regardless of your steps to pre-treat your survey design, you must identify the extent to which this bias exists in your data. The most common way is to compare response rates for the different subgroups within every variable liable to influence RP. If any subgroup’s response rate is statistically significantly different from another’s, then you will need to correct for this bias before performing any predictive analytics. This method is not foolproof: it assumes that all factors that impact RP are kept for the full population of survey recipients (non-respondents and respondents). Thus, tracking any potentially relevant variables for the entire customer base can help identify self-selection bias and how exactly it impacts your data. You’ll then be ready to attack the self-selection problem head on (how? I’ll explain in the next entry) and use your data as a crystal ball for future customer behavior.

Which tools or techniques do you use to pre-treat for self-selection bias? Do you see different response rates for different groups or customer segments?

Blog 1 in 3 Part Series on Analysis of Bias-Filled Data

So you’ve designed the perfect customer feedback questionnaire, sent it out to your entire customer base and the responses are flying in. You might be getting excited about analyzing the incoming data but not so fast! In any kind of survey endeavor, especially in customer experience feedback, the analyst must be conscious of the bias present in the data collected. Before discussing techniques to identify and then correct for bias in the data (the second and third parts of this blog series, respectively), I’ll outline the different types of bias that are present in our field.

Two types of bias are a part of every data-based experiment: random bias and systematic bias. Random bias is always present when measuring customer experience or any other behavioral process; people will respond differently based on unpredictable processes such as life events, mood or even the weather! Because random biases can be assumed to fluctuate within the sample, they should not slant your data in any one way.

Systematic bias, on the other hand, skews survey results in a particular direction away from true population values. For example, if you only sent questionnaires to clients from a specific region or ethnicity, you can be sure that their answers will vary in a certain way from those of all customers. This systematic bias that is introduced in how a sample is constructed is called sampling bias.

Of course, few market researchers will intentionally omit certain groups from their survey invitations. But because we are rarely able to use probability-based sampling and instead, collect all survey responses that come in (often called convenience sampling), the sample that emerges is far from representative of the overall population. This will always invite another form of sampling bias: self-selection bias, which occurs because the people who elect to respond differ meaningfully from those who do not. Respondents tend to have more favorable opinions of the company than non-respondents and there is still debate on whether certain cultures or ethnicities are more likely to participate in surveys. Regardless, we must understand that when we analyze customer survey data, we are studying the most engaged group of customers and that, unless we utilize techniques to adjust for this bias, we may only generalize our findings to this smaller group.

Waypoint’s focus on the non-respondents is what differentiates our methodology from the rest of market research. Rather than ignoring this group and solely analyzing respondents data, we know that much insight can be found in searching for which customer traits are most significant in predicting survey response. These factors shape the group that the company most needs to energize and engage – the next step is to follow-up with typical non-respondents to see what went wrong in their experiences.

What do you think? Are you aware of different biases in your customer experience data and how do you react to them?

In our research and (often) statistics-heavy industry it’s easy to go heads-down and just focus on the work.  But Net Promoter and customer Insight work can be fun – take a look at this from our friends in Netherlands! Let’s keep an eye on it to see how this goes:

Net Promoter News: NPS Now Goes to 11 – “One More Likelier”

By the way – while you are over there on that site take a look at their very well written and backed article (including great response
from the NPS “Godfather”, Fred Reichheld) on cultural bias:  Net Promoter: Is there a “Dutch Effect”?