Focus groups · Concept development

Traditional focus group (offline) to evaluate an online, digital concept

Yaniv Levi

October 25th, 2015

We consider running a focus group to review a new online service concept in order to get a qualitative reaction to it. Do you have any experience how valuable this can be? The costs are high but we feel that it can give us valuable insights that are worth the investment.  

Sharon McCarthy Chief Marketing Officer

October 25th, 2015

Do not do them.
I have 20 years of experience in consumer research. Focus groups are bad predictors of purchase/interest. They are great brainstorming opportunities if that's what you need.

If you have a small budget, here's what I would do instead. (Assuming this is a consumer based business).
  • Define exactly who your target is. Exactly -- demographic, attitudes about the category, how they use the category/use of other similar products, aspirations and how this service might fit.
  • Hire a freelance market research person with 7+ years experience doing qualitative research with some big consumer brands. Have them develop a questionnaire.
  • Find (recruit) 15 people that exactly meet your target. I would have you and your team find your subjects. I've posted on social media, etc. Professional recruiters are expensive and probably won't take this small a gig.
  • Pay each participant $50 to do a 45-1 hour one-on-one interview with your marketresearcher.
  • Conduct the interviews at a starbucks. (much cheaper than renting a focus group facility!)
  • Do not administer the survey yourself unless you are an experienced consumer researcher. You need context for what you are hearing and an unbiased view. It's hard to be unbiased if you developed the product; it's hard to have context without having experience doing market research for a lot of brands/businesses.
  • I would not recommend an academic b/c they often lack the real world context (benchmarks) or state of the art survey techniques. Yes, I'm generalizing and there are exceptions, but they are few and far between.
  • It should take 2.5 days for this person's time, all in, including writing up the results and taking you through them. (no powerpoint).
  • At $150/hour, that's about $3k. Just 1 focus group these days will cost you about $15k, and they will be less useful. 
  • If you want some reco's, I'm happy to provide names.
This will give you good-enough information. You will need to make sure you have defined the target correctly, though. I applaud you for wanting to do this. As an entrepreneur you will need to know your customer and their needs and aspirations better than anyone else. This will help.

Leo Frishberg UX Strategist, Product Manager, Phase II

October 25th, 2015

I am a huge believer in qualitative research, but I am extremely skeptical of "focus groups." There are literally hundreds of different methods you and your team could employ, most of which are going to get you the information you seek and none of which are focus groups.

I'd love to discuss your needs with you in greater depth, if you're interested in learning about cost/benefits of various approaches.

Craig Hajduk Chief Strategy Officer at Adonit

October 25th, 2015

Hey Yaniv, There are lots of tradeoffs to focus groups - in this case if you’re just doing one (or a very small number) you run the risk of not having the right participants in the room, or getting poor results because of a few strong personalities, or other group dynamics. The quality of the moderator matters a lot. There are other techniques you might want to look at instead. Feel free to ping me directly and I’d be happy to chat or set you up with an experienced researcher that could offer some advice. Craig

Peter Johnston Businesses are composed of pixels, bytes & atoms. All 3 change constantly. I make that change +ve.

October 25th, 2015

Stop right there. Take the word digital off your job title.

Digital is your dad's tech. Most of us discovered digital back in 1983, when they launched CDs. Since then we've had the PC (1984), internet, social network, datascience, mobile... 

We now live in a world made up of pixels and bytes. Constantly changing and fluid. One we can influence not by big programmes, defined products and campaigns, but constantly, with little touches here and there.

If you still think digital is new and something to put on your job title, you don't understand it. As you have proved here.

But there's another mindset change. The world has gone two way.
We grew up in a world of broadcast media. We could talk, because we owned a broadcasting or publishing company. We could sell our megaphone to people who didn't have one of their own to other people who wanted to use it - we called this advertising and charged for it.

But the people on the receiving end of all this noise couldn't talk back. Shouting at the television or radio doesn't work. Neither does letters to the editor. So media owners and companies got separated from the people they were talking with - because they were no longer talking with, they were talking to. So they invented odd things like customer surveys and focus groups as lip service to the idea of keeping in touch. They couldn't afford to talk to all tens of millions of course, so they decided a hundred would do - after all everyone was the same really!

In the online world this is fundamentally flawed. Here we have a true democracy - everyone has a voice and if you won't listen, they'll talk with their friends instead. Tell them not to buy from you because you don't listen or respond to their complaints, needs and wishes.

Don't waste your money on an offline survey. Finish the build of your online product - build the bit that helps them design it for you, promote it for and debug it for you - the bit where they can talk to you. 

Then you will know what they want you to build. Their online actions will tell you. You can AB test concepts and see which gets the best response. Action the suggestions, sift through hundreds of good ideas and build the best ones.

Online you can really be in touch with all of your customers, influencers and friends. Not just the few you can talk to in a survey. But not if you see digital as just another channel for broadcasting and industrial age selling.

Henry Chou Product & Marketing Leader

October 25th, 2015

Agree with the comments thus far. I am not a big fan of focus groups.  I think that there are much cheaper and better ways to get you the qualitative feedback on your concept.

Happy to chat more

Ellen Henson Engage and align product teams for customer satisfaction and business success

October 26th, 2015


When it comes to gaining insight from your target audience I like to say "test early and test often". :-)

I agree with much of what Peter says though I do find it useful to test some concepts with surveys. I wouldn't limit the testing to surveys.

In addition to the user profiles suggested by Joanna, take a look at an approach called "jobs to be done" and develop a clear value proposition using a "value proposition canvas". Before you test product or prototypes, you can validate whether your target use (jobs to be done) and value proposition resonate with your audience. This validation can be reasonably achieved through well-structured and brief surveys. Both terms are searchable for more information or feel free to contact me and I can send you some links.

Every time you test your prototypes and products, you need to test in context of both jobs-to-be-done and how well the products delivers on the value proposition.Be open to insights that change or challenge your views on jobs-to-be-done and value proposition.

I completely agree with Sharon and Mike re: focus groups -- focus groups are expensive and also subject to taint or sway by loudest voice and strongest opinions. Sharon's advice about  1:1 interviews and some direct observation of use and/or response to prototypes is spot on once you have mock-ups or prototypes.  While the videos of users per Mike's suggestion of are helpful, there isn't opportunity for open-ended questions. On the flip side, if you do any direct observation, you need to be careful not to prompt the user or guide them in any way that skews test results. (See Sharon's notes about hiring a professional researcher or get some coaching from one) Perhaps some videos first and then some direct observation? And, of course, the direct online testing suggested by Peter.

You've been presented with a lot of well-thought out options. Come back and let us all know how it works out.


October 25th, 2015

As a researcher of long standing, I strongly suggest testing the concept using survey researcher with a sufficient sample rather than amall gathering of respondents who are usually influenced by the loudest mouth among them and will net you nothing for your efforts.

Gabriel Berner    

Mike Robinson

October 25th, 2015

I agree with Peter. Any single survey or focus group is just a snapshot in time and if your product is online it needs to be changing and evolving constantly. 

If you just want to test an idea for user interface before investing in a full-blown version, create a prototype or mock-up of a portion of your idea, then use a service like to get real-world feedback on it. You get videos of people in your specified demographic using your prototype and completing certain tasks that you assign.

If you just want to ask a bunch of people what they think of the idea, create some static mockups and use a tool like surveymonkey to do your own online survey. They can find volunteers in your desired demographic for something like $5 a pop.  At those rates, you can afford to do a small survey, learn from it, adjust your survey to answer what you still want to know, and do another larger survey. This can all happen at Internet speed and for very little cash.

Jindra PhD Case Study Research Lead, Ex-post Evaluation in USAID E3 Basic Education at Management Systems International

October 25th, 2015

I have done qualitative research for 25 years and see great value but Sharon and Mike are right. 

Jindra PhD Case Study Research Lead, Ex-post Evaluation in USAID E3 Basic Education at Management Systems International

October 25th, 2015

Wanted to ask; what is it about qualitative open-ended feedback that you think you need for your product or service? Why won't a close ended survey do?