HOME > Chowhound > Food Media & News >

Discussion

America's Test Kitchen reviews are meaningless

I know this might ruffle a few feathers here but I dont get why so many people take the Taste Test results from this show as the word of god or in any way accurate.

From what I can tell, when they taste test a food product, its limited to their studio audience of about 30 people? That's way too small to get a real sample. In the normal world of opinion polls done by researchers, a minimum of around 1500 people is need before anything could be considered accurate. Less then that creates false results. Certainly 30, 60 or even 100 people would be way off.

They also dont account for regional differences. Lets take BBQ for example. People from the North East are going to have completely different opinions then people from the Midwest or Texas.

Even if ATK is using more then 30 people, Do any of them reside outside the city or state where it is filmed? Why not pic say 30 volunteers from five different cites? It would not be that hard or expensive. They are not even filming so very low cost.

Their results vary wildly from other test sites. Lets take chicken stock for example. ATK rates the brand Kitchen Basics as not recommended, burnt tasting etc. Yet Consumer Reports rates it as very good and one of the top two. Their taste testers love the roasted flavor, ATK tasters called it burnt.

So if you want, prove me wrong, but back it up with some facts. From from what I can tell, ATK taste testing is wildly inaccurate or meaningless unless you just want the opinion of people who live outside Boston.

  1. Click to Upload a photo (10 MB limit)
Delete
  1. Like most other TV shows with an audience, anybody can sign up to get tickets, so the audience may be more than just those who live near Brookline (their taping location). Still, this is a niche show with a certain demographic target, so the diversity is certainly badly skewed. Your most important point is dead-on: totally inadequate sample size, taken at one point in time.

    It's entertaining, especially when Kimball and the audience are at odds in their choices, but certainly nothing to rely on.

    1. I find complaining about the validity of speciality cooking shows to be meaningless. Don't concern yourself about something you can't change. <shrug>

      CI is a (seemingly) profitable venture and in no way are they claiming to be the last word on the reviews. Like any review its about knowing the source. You don't like the source then stick with Consumer Reports if you feel your tastes and theirs line up.

      Lots of cooking shows offers reviews of foods and products, just look at The Chew, The Kitchen, etc. So does all the morning shows-Today, Good Morning America. Most of those don't even have studio audiences weighing in.

      Don't sweat the small stuff, life is way too short.

      1. As far as the taste tests, half the time I have never even heard of the brands that they show. I've fully disagreed with many of their results.

        3 Replies
        1. re: rasputina

          Same here... what brand is that?
          I don't know and couldn't care less is the audience or Chris Kimball agrees or disagrees with me.

          1. re: rasputina

            I'm not sure I'd agree with the results of a 1500 person sample, or even a million. Taste is pretty personal.

            1. re: Savour

              Exactly- it's personal, it's regional, it's a lot of things, so why anyone gets exorcised over ATK reporting what their audience thought is beyond me.

              Take it or leave it.
              It's not like scientific research where you can extrapolate.

          2. Wow, take it for what it's worth!
            It's a tv show, not a submission to The Lancet.

            I can't prove you wrong and you can't prove you right.

            It's OPINION, not the gospel, the bible, the word of God or written in stone.

            I can't believe anyone gets upset about something they volunteer to do.

            Don't watch!

            I swear, people just look for reasons to bitch about this show/magazine/cast.
            It must release endorphins or something.

            3 Replies
            1. re: monavano

              I am not worked up about it, just pointing out the obvious. Seems like a lot of people who watch that show live and die by what they say.

              Yes it is just a show, however, they take great pride in letting everyone know how rigorously they test things. So much so that you have to pay to see some of the results because they dont want advertisers to influence the results.

              1. re: kjonyou

                It's a show.
                They go to certain lengths to attempt to ensure objectivity.
                Their business structure is what it is-been bitched about here until the cows come home.
                What else am I missing.

                1. re: kjonyou

                  Their taste tests are purely entertainment, their claims of rigorous testing apply to equipment, which audience members don't weigh in on.

                  That said, I wouldn't necessarily rely on those exclusively; I look at Consumer Reports, Chowhound, and reviews on Amazon and weight them accordingly.

              2. Have you ever seen a person of color in that audience? It's ridiculous.

                6 Replies
                    1. re: monavano

                      I am not implying it, the person above me is implying that ATK is all white in case you have not noticed.

                      Since America is quite a mix of races, its odd that even in a big city they only seem to have white audience members with limited tastes.

                      You may say food is food and it doesn't matter what your ethnic make up is, but not according to the food industry. They actually cater to ethnic difference but very small changes in the same products. So for example, Norrs Chicken base will have a slightly different formula on the exact same product depending on weather the box is written in English or in Spanish even if they are both sold in the same store.

                      1. re: kjonyou

                        Monavano's question wasn't aimed at you; it was asked of "the person above" you. Do try to keep up.

                        Also, in regard to "you may say...", well let me just say putting words in someone else's mouth is no way to prove a point.

                        1. re: kjonyou

                          Just to try to help.......

                          "I am not implying it, the person above me is implying that ATK is all white in case you have not noticed."

                          Each post that replies to anyone other than the OP shows the specific poster's name in its upper right corner. kjonyu's post says "RE: sandiasingh" at its upper right.

                          Hope thus helps you avoid unnecessary angst.

                    2. I don't think they use the studio audience as the tasters for the ATK or CI taste tests. It really seems that they use professional tasters for their panel. I don't know if they are strict as say a Consumer Reports tasting panel, but as Type A as Kimball seems to be, there is no way he would let the audience be the judge. Also Jack Bishop drops hints during their banter that it is a professional panel. That being said, taste tests are still subjective to a certain extent and their may be a New England bias. It all comes down to, if you like it, buy it.

                      5 Replies
                      1. re: littlestevie

                        I may be wrong, but I am pretty sure Jack has referenced the audience as the taste testers. There might be additional family and friends but I doubt they are paying for professional taste testing.

                        1. re: kjonyou

                          There's the show, and there are the magazines.
                          Each have "taste tests", and I highly doubt that what's published in their magazines, as "recommended" and "not recommended", comes from random audience members which you obviously see on t.v.

                          Look, that whole audience taste test is just a foil for Kimball to be his persnickety old self.
                          It's a show segment.
                          It's entertainment.

                          1. re: kjonyou

                            I have not subscribed in several years, but in the back issues I have of CI, they used to identify the test panel members and almost always, they were food professionals.

                            1. re: kjonyou

                              They pretty distinctly differentiate between their actual testers, the audience and kimball

                            2. re: littlestevie

                              Correct. The published taste tests are conducted by the Test Kitchen staff. The studio audience is just for the Cook's Country TV show. It has no input into their published results.

                              With that said, all taste tests are subjective and America's Test Kitchen is just one resource. ATK does a good job explaining food science and why some product taste better than others. For instance, the ATK did a great job debunking all the hype surround San Marzano tomatoes which is consistent with my palate. And yet, they did taste tests on the "best" hot sauce which was silly given the variety of styles.

                            3. I know some people are saying "whatever float your boat" but that is not the question.

                              Most people, myself included dont have the time or money to taste 12 different versions of every product they buy. That is why these lists exist.

                              How many times have you heard a TV chef say "just buy a good one"? Well, what brad is a good one? So we turn to reviews.

                              So yes, while I might disagree with the results, it would be nice to know what the rest of America thinks about that product in a more accurate sampling.

                              Sort of like an election poll, it will not change my mind who or what I end up voting for, but it's helpful to know which why the country is going.

                              3 Replies
                              1. re: kjonyou

                                OK, if ATK's taste tests aren't going to convince you, please, tell us what does.
                                I don't know what you want to hear here...

                                Don't complain about the problem, tell us your solution!

                                As to wanting to know what America thinks, why do you care when you state in your OP that regional tastes matter, in fact, that's a bone you have to pick with ATK.

                                "They also dont account for regional differences. Lets take BBQ for example. People from the North East are going to have completely different opinions then people from the Midwest or Texas."

                                America is VAST, and regionally, opinions vary.

                                Go find statistics that show what brands are the national leaders/best sellers, and there's your answer.
                                Don't try to hold ATK accountable for something they never set out to do in the first place.
                                They are a niche.

                                1. re: monavano

                                  Well, it's called America's Test Kitchen, not Boston's Test Kitchen, Massachusetts test kitchen or even East Coast Test kitchen. Its AMERICA'S Test Kitchen. So there is that little fact that blows your argument out of the water.

                                  Also, they pick recipes and items from VAST regions across the country. Yet their regional opinion is OK since its a TV Show? Cant have it both ways.

                                  1. re: monavano

                                    "Don't complain about the problem, tell us your solution!"

                                    I did this already, they need to sample from several different cities, then combine the results.

                                    "Go find statistics that show what brands are the national leaders/best sellers, and there's your answer."

                                    This is faulty logic. Best selling does not mean best tasting. Best selling could be lowest price, or best marketing. In fact, the best tasting is usually not the best selling because it's usually higher priced.

                                    Prime beef is better then Choice, but Choice is the best selling because it's cheaper. Get it.

                                2. It's pretty simple. Anyone considering a review of a product needs to be able to put that review in the context of its source. I don't pay very much attention to any review unless I'm able to calibrate to the source. I don't care if it's a small group, a large group, or an individual. I don't use it to any great extent until I've had a significant number of chances to determine if our 'tastes' are similar.

                                  1 Reply
                                  1. re: Midlife

                                    ATK is one source for reviews, not THE source for reviews.
                                    I have never had the impression they implied that they are.

                                  2. You are way over thinking this. No survey of people's opinion on any topic is "accurate". Otherwise we would only have polls and not bother with elections. Its an effen survey of opinions. Doesn't matter whether its 15, 1500, 15,000,000 participants. If you don't like something they do or vice versa, no biggie.

                                    2 Replies
                                    1. re: Bkeats

                                      Hu? that dose not make any sense. If you look at the right polls, some of them are VERY accurate. Google Nate Silver poll of polls.

                                      The only people that were blind sided recent by false polls where the Fox News viewers on the presidential election. The polls outside of that conservative media bubble were accurate.

                                      Your assertion that people would not vote if polls were accurate is faulty logic. Voting is not just about opinion, its also about who comes out to vote.

                                      1. re: kjonyou

                                        Nate Silver exhibits a lot of bias in his observations. Tried to read his book but couldn't finish as it was filed with it. He doesn't look for meaning from the numbers, he was finding confirmation of his beliefs in them.

                                        Polls are opinions. Nothing more. If you rely upon them for more than that, you might as well go to Delphi and asks the oracle. It would be as useful at a personal level.

                                    2. Here's an idea-write a letter to Chris Kimball.
                                      Let us know what he says.

                                      1 Reply
                                      1. re: monavano

                                        After you pay money for invoking his name, ATK, any of its affiliates and/or underwriters.

                                      2. I often do taste tests of my own, and found one linked in another thread on fish sauces which I thought was excellent:

                                        http://ourdailybrine.com/fish-sauce-t...

                                        This is the type of taste test I like to see, and hope the site takes off and does more like this one. The follow-up from the author added a lot to the review.

                                        I doubt if ATK can compare, or should be compared to this kind of detail. ATK's tests are just a segment on a time-limited TV broadcast, and should be taken as such.

                                        1 Reply
                                        1. re: RelishPDX

                                          ATK's taste tests have an entertainment component- Chris Kimball doing the "icky lima bean" face ;-)

                                        2. They're reviews like any other review. Almost all food are based on subjective opinions, be it a single person's or a group of tasters, "expert" or otherwise. They're simply giving you a majority view of their own small sampling.

                                          There's a fairly compelling reality show in the UK called "Breaking into Tesco" where people compete to get their prepared food item featured in the giant supermarket chain. The first step involves setting up a table in a random area and offering samples to passerby who toss their cup/plate into a yes bin or a no bin. Same idea, it just tells you that among that random group something was favored or it was not.

                                          And you can't look for statistical reliability in such opinions in any case. Just consider how many poorly reviewed movies are popular at the box-office. As you point out, one man's burnt is another's smoky or roasted. I can't begin to tell you how many people complain about the saltiness of products that I find appealing and not oversalted. Ten reviewers or a thousand isn't going to change that.

                                          One last point, there are countless knuckleheads here who bash dishes they've had and when you read further it's because they hate an ingredient in the dish like mayo or salt or eggs or cilantro. They're zooming into a single point. So 2,000 random people doesn't guarantee reliability, you'd need to cherry-pick them as to their generally likes/dislikes, knowledgeability and breadth of tasting experience.

                                          1 Reply
                                          1. re: ferret

                                            One of our local news stations does the public opinion taste test poll a lot.
                                            Each type of taste test has meaning, however, applicability varies.

                                          2. One other thing, it's not a "word of god" reverence, it's a common reference point. Most people here know of America's Test Kitchen, so when you state that "ATK voted it a favorite" you can consider whether it's reliable for you or not. Just like when Ebert gave a movie a thumbs up/down. People familiar with the reviewer can then act accordingly.

                                            1. When I get a chance to watch them I enjoy them. Especially when Chris's opinion is way off what the audience thinks. It's entertainment, pure and simple.

                                              1. I get home just in time to turn it on when i get home from work. But I watch it to see what kitchenware they are testing like pots, pans, poultry scissors etc. Very interesting discussion...and anyone know who is that smiling, white haired guy with the bow tie usually sitting in front??

                                                1. Good God, man/woman, it's entertainment. If you're not entertained, you owe it to yourself to watch or do something else.

                                                  Personally, I'm OK with most of what Kimball/ATK does. The equipment reviews are largely a joke, but they're no worse than the ones done by CH.

                                                  1. When lived in China, I was a recipe tester for them.
                                                    The evaluation process during and after each is rigorous, and I was definitely asked a lot of questions about my demographics.

                                                    .

                                                    Kris now in DC/NoVA

                                                    5 Replies
                                                    1. re: Kris in Beijing

                                                      How in the world do you become a recipe tester for CI when you're living in China ?????

                                                      1. re: Midlife

                                                        Yes, more info please! That sounds like a fun job and/or hobby. Was it paid? Did you have to apply or did you know someone? Did you run into problems obtaining ingredients in China? Please tell us all about it. Or tell me anyway.

                                                        1. re: NonnieMuss

                                                          Both the magazines and website contain info on how to become a tester.

                                                          1. re: NonnieMuss

                                                            Threaded answers:
                                                            >That sounds like a fun job and/or hobby.
                                                            Tons o'fun.

                                                            >Was it paid? Did you have to apply or did you know someone?
                                                            No, no $. Ad in the back of one of Mom's mag.s that she sent me
                                                            (see here: http://chowhound.chow.com/topics/592732 for my desperation for cooks reading material).

                                                            >Did you run into problems obtaining ingredients in China?
                                                            Not really. In Beijing, there are six[?] or more expat-focused grocery CHAINS and a few embassies have stores, too. PRICE was the issue. A 2K block of cream cheese at the Restaurateurs' Market can really set you back, but that's cheaper than the 8oz imported foil sleeves-in-silver-boxes at an expat grocer.

                                                            I got an email every, oh, 3 months. Usually for a summer dish or a dessert. Often with a "please focus on this part" of the prep or ratios.
                                                            Until I bought myself an oven, some were pretty hard to do!
                                                            My "signature/ most requested" brownie recipe came from a test. The published version wasn't quite the same [never is] and the way I had to solve some of the issues with the beta made it mine.
                                                            Even if the food to be made wasn't really interesting to me, I was always SO THRILLED to see their email!

                                                            Acronym soup: FYI/BTW/FWIW - I don't think my tastes are very similar to Kimball's.

                                                            Kris , now in DC/NoVA

                                                            1. re: Kris in Beijing

                                                              Thank you! And thanks Greygarious for the info too. I'm probably going to apply once I finish my next two blog projects.