Passikoff: Spotting Crappy Market Research

The quality of mercy is not strained, even in the market research industry.

Things have changed over the past 30 years. Yeah, OK, I know, “Duh.” Consumers and brands and the marketplace have changed. In this instance, however, I was thinking about changes in market research. How it gets designed, fielded, and analyzed. A lot of the credit (and blame) goes to technology. Mostly technology. Hey, I’m not a Luddite, but tech has a lot to answer for.

You’ve got all those DIY research tools and Twitter outreach and online panels and social networks that all provide immediate access to respondents. Lots of respondents. And concomitant amounts of information very quickly. Data is very nearly ubiquitous. Insights, well, you be the judge. AI and machine learning means what you’ve collected can get sent out as C-level reports just as quickly. (Although generally speaking, C-level folks don’t really want more than four numbers. Fewer, if possible, which is why NPS is so popular.)

I was putting the final touches to this week’s column when I received an email. It was from one of the largest social networks in the world asking for my participation in a survey they were conducting. I won’t mention their name. The quality of mercy is not strained, even in the market research industry. But here’s a precise quote of what they sent. To me and a gabillion other members:

“We’re doing a survey about XYZ. We’d like to hear your opinion, even if you might not be familiar with the category.”

Questions abound. The first, are you kidding me? You want information from someone not familiar with the category? That’s the kind of data you want to include in your analysis? What the hell use is data from someone not familiar with the category? How accurate are insights drawn from that data going to be? And, are you kidding me?

Sure, many researchers feel these technologies have “democratized” market research. I’m not precisely sure what that means either, but it likely means you can intrude into anyone’s life at any time and don’t have to rely on shut-ins and retirees. Research has been “politicized” too. I do know what that means.

A lot of research is self-serving. Or just self-evident. Listening, thinking, and analyzing seem to have been left behind. So, talk all you want about “technology,” “Big Data” and “data mining,” and “convergence,” but none of that is a substitute for a great research design. And yes, technology in all sectors is constantly evolving. But so are consumers.

The thing is, consumers got smarter. Maybe “smarter” isn’t precisely the right word. Technology gave them access to brand and marketing information they never had before. Maybe “more agile” or “more knowledgeable” or more “conversant” are better phrases. An absolute truth about 21^st century respondents is you can’t just ask them what they think. Particularly if they’re not familiar with the category. A five-point scale just isn’t nuanced enough for highly market-acculturated consumers. Consumers are pretty much unable, and often unwilling, to articulate the kind of meaningful values that allow you to get real insights. You must be clever and get under the consumer’s conscious “radar.” More especially if they’re not familiar with the category you’re asking about (INSERT LOUD SCREAM HERE).

We use a combination of validated psychological research and some really neat higher-order statistical analyses. That fuses emotional and rational aspects of a category and is predictive of how consumers are really going to behave in the real world. They correlate at 0.80+ levels. For the wonks among you, you can find  additional information about how we do what we do here or here.

OK, what got my hackles up? It wasn’t the email. No, it was a recent report of a survey showing that sixty percent of registered Republicans plan to use TRUTH Social, Donald Trump’s not-yet-released social media platform. Its stated mission is to “create a rival to the liberal media consortium and fight back against the ‘Big Tech’ companies of Silicon Valley, which have used their unilateral power to silence opposing voices in America.” To be very clear, my raised hackles have nothing to do with any political view or affiliation or opinion of Mr. Trump. The news just happened to break right now, and it was its particular survey design and “insights” that, well, just made me crazy.

It was a quantitative survey. Nearly 2,000 registered voters. The plan-to-use scale regarding usage was apparently, “A lot,” “some,” “a little,” “not much,” or “not at all,” which is pretty basic. And no matter your political affiliation, no matter how you feel about Mr. Trump, assuming you’re a sentient political analyst – let alone an actual market researcher – would it surprise you to find Republicans were more likely to use the site than Democrats? Wasn’t that the “self-evident” answer?

Oh, the study found thirty percent of Independents (precisely the same percentage that more-often-than-not vote Republican) would use the network “a lot” or “some.” Top-2 box. The thing is, as far back as 1990, researchers knew these kinds of “intent” questions were notoriously over-stated in survey responses. Except, perhaps, where there might be some sort of sample bias. Like Republicans demonstrably indicating interest in a Trump vehicle, and Democrats, not so much.

By the way, the social networking brand, the one that didn’t care if respondents were familiar with the category or not, offered a $5 digital reward for participating. Let me ask you this, how much do you actually trust someone who’s willing to sell their opinions for $5?  Well, OK, maybe five bucks for a worthless opinion is a deal. While you reflect on that, remember what Gandhi said. “Be the change that you wish to see in the world.”

Just don’t wish for crappy research.

Robert Passikoff is founder and CEO of Brand Keys. He has received several awards for market research innovation including the prestigious Gold Ogilvy Award and is the author of 3 marketing and branding books including the best-seller, Predicting Market Success.  Robert is also a frequent contributor to TheCustomer.

Photo by Camylla Battani on Unsplash.