Focus Groups

I have to admit that I’m not much of a focus group fan. They tend to produce such wonderful ideas as New Coke and the Bob user interface. Remember those gems?

It’s been a few years, but I’ve sat through a few focus group sessions during my career. I think the general problem is that focus groups tend to describe what they think they want, not what they really need.

A former employer of mine conducted a focus group in the early ‘90’s, with the goal of discovering more information about the typical computer book user. One of the things we were trying to determine is whether or not customers would pay more for a disk in the back of a book. The focus group results were unanimous: no way would these people pay more for a book with a 3-1/2” disk in the back. According to the feedback, it didn’t matter whether that disk included source code, utilities or the meaning of life itself – it just wasn’t worth a penny.

Interestingly, one of the imprints at that former employer was conducting the same test, only they were doing it with real products in real stores. They produced two versions of the same book: one came with no disk while the other was $5 more and included the disk. Guess what? The one with the disk typically outsold the other version by a factor of 2 or 3. It wasn’t just one title, by the way. They conducted the same test with no less than 10 titles over the course of a year. So much for the focus group results.

If focus groups aren’t that effective, what’s the best way to figure out what the market needs? Rather than trying to determine this in a simulated environment, why not just go directly to the customers at work or wherever they’re likely to use the product? If you’re working on a book about Excel, that means talking with people while they’re working with spreadsheets at the office. If the topic is Java, spend some time with a developer while they’re actually designing and writing the code. This isn’t as glamorous as a full-fledged focus group, but I’m willing to bet you’ll get better information in the long run.

Comments

[shel]

How about using this blog as a focus group. Participation can be ongoing and voluntary.

[Duane]

I gotta admit one thing that I did like about the Apprentice this year was when one team was tasked to create a new office product, they headed on down to Staples and did an impromptu focus group. They asked people and took notes, but they also just rooted through carriages to see what people were actually buying and not just what they said they think they might possibly consider buying.

I've been on both sides of the focus group two way mirror. When a participant, I hated the fact that they'd clearly come with an agenda of "Which of these 4 models do you like best?" even if it was clear that we wanted to talk about features that none of them had considered. Don't come into the focus group with two narrow of a view. You end up asking your audience not what they think, but which they like better, X or Y? And they may still hate both.

Behind the mirror, as a product developer, I realized that if you listen to the focus group long enough everybody will eventually contradict each other. "Give me every feature you've got!" cries one woman. "Just make sure that I have a switch on each one to turn it off. Then, I'll decide which combination I like." In theory I love that woman, but in practice we know that'll never happen, because the rest of the group says "Just give me the features I want, don't clutter it up and confuse me. And, by the way, I don't really know what I want until I see it, so read my mind."

[Jack Krupansky]

Great topic to consider. I'll oversimplify my take on focus groups: they tend to mistakenly focus on the higher-order intellectual processes of the human mind and completely avoid the "inner reptile" that kicks in when we "go shopping". Shopping is *not* an intellectual pursuit.

Intellectually we all *hate* the idea of paying more for something that we probably won't need, but the "reptile within us" thinks differently and seems to recognize at a very primitive level that we need to "consume ahead" (the reptilian analog to "planning") for eventualities that we can't really rationally plan for.

-- Jack Krupansky

[Joe Wikert]

Shel, sure, a blog can be used as a focus group, but I think it offers the same challenges as any other focus group conducted outside the real environment. IOW, let's say you ask blog readers what features they'd like to see added to a new car design. I think you're likely to get different answers than if you asked those same people while they're sitting in traffic, on their way to work, to the grocery store, etc. My point is you're better off conducting this sort of research in the environment where the product will be used, not some artificial environment where loaded questions don't seem to lead to much.

Duane, great points about (a) going to the source and (b) contradictions in focus groups. In fact, I've seen first hand how the more outspoken focus group attendees can sway the opinions of the rest of the room. Not good.

Jack, I totally agree with your point about the "inner reptile" or gut reaction, as I might call it. There's definitely something about the sterile feel of a focus group that seems to strip much of this away.

[John Neidhart]

Joe wrote:
"Interestingly, one of the imprints at that former employer was conducting the same test, only they were doing it with real products in real stores. They produced two versions of the same book: one came with no disk while the other was $5 more and included the disk. Guess what? The one with the disk typically outsold the other version by a factor of 2 or 3. It wasn’t just one title, by the way. They conducted the same test with no less than 10 titles over the course of a year. So much for the focus group results."

It would be instructive (and essential) to know whether both versions of the book were stocked at similar levels to arrive at these results. I can think of *one* former buyer who would've eschewed the lower priced, disk-less version of the book and only stocked the version with the disk...

[Joe Wikert]

Hi John. It's been several years now, and I don't recall the stocking specifics. I can tell you that both versions were at least stocked in a similar fashion at several key accounts (back then). Let's put it this way: Stocking on both was significant enough to confirm what I noted in my original post. It also helps that it wasn't just one pair of titles at one time of the year. The number of pairs and copies sold may not have been "statistically significant", for any purists out there, but the results were compelling.

[Michael Miller]

Joe: I don't remember the focus group in question, we actually did very little research back in the day, which is a shame. Not that I'm a big fan of focus groups (all you get is a bunch of opinions), I do think research in general can tell us more about who our customers are and what they'll buy. The problem comes when management doesn't like the results, or just ignores them. I recall some very detailed research we did in preparation for launching a series back in the mid-1990s. The research said that customers like the title "User Friendly" with a white cover. Management other than me didn't like the results, so we called the series "Using" (this was the first split of the Using brand) and gave it a black cover. It didn't do well, surprise surprise. If you do bother to listen to your customers, you have to accept what you hear!

[kathy Sierra]

Yes!! Getting close to the user in the wild -- in his "native" environment -- is crucial.

Two techniques I really like are the "five whys" and haunting online discussion boards.

The "five whys" means asking the user what he wants, and then when he answers, ask, "Why?" and keep asking why or who cares or so what to each thing he says. Chances are, when you get to the FIFTH one -- you've finally got to the soul of the issue. For so many people, and especially in focus groups, they answer the "what would you want?" question by imagining what would solve the first or second "why" response, when what they REALLY want/need is something completely different. But until they get to the heart of why they're really trying to get feature X...

For me, lurking (or participating) in online discussion boards related to the product is great -- especially for topics where people are likely to look online for answers. I do a statistical analysis of the types of questions that people keep asking, and use that as the best indication of the things people are having the most trouble with. But when a vendor (or instructor) asks the questions, often people assume that they must be stupid to be struggling with that thing, so they don't bring it up. But in online forums, and especially anonymous ones, the REAL struggles are right out there. I would suggest that any author spend a great deal of time finding the places where people (especially beginners) are really looking for answers.
If I find enough pleas for help and understanding related to a particular topic, that tells me that this topic is probably not well-covered in books. Or at least not in a way that people are getting it.
I cringe the most when I find an online forum where people are looking for help in making sense of something *I* wrote ; )