HOME > Chowhound > Food Media & News >

Discussion

America's Test Kitchen reviews are meaningless

I know this might ruffle a few feathers here but I dont get why so many people take the Taste Test results from this show as the word of god or in any way accurate.

From what I can tell, when they taste test a food product, its limited to their studio audience of about 30 people? That's way too small to get a real sample. In the normal world of opinion polls done by researchers, a minimum of around 1500 people is need before anything could be considered accurate. Less then that creates false results. Certainly 30, 60 or even 100 people would be way off.

They also dont account for regional differences. Lets take BBQ for example. People from the North East are going to have completely different opinions then people from the Midwest or Texas.

Even if ATK is using more then 30 people, Do any of them reside outside the city or state where it is filmed? Why not pic say 30 volunteers from five different cites? It would not be that hard or expensive. They are not even filming so very low cost.

Their results vary wildly from other test sites. Lets take chicken stock for example. ATK rates the brand Kitchen Basics as not recommended, burnt tasting etc. Yet Consumer Reports rates it as very good and one of the top two. Their taste testers love the roasted flavor, ATK tasters called it burnt.

So if you want, prove me wrong, but back it up with some facts. From from what I can tell, ATK taste testing is wildly inaccurate or meaningless unless you just want the opinion of people who live outside Boston.

  1. Click to Upload a photo (10 MB limit)
Delete
  1. Like most other TV shows with an audience, anybody can sign up to get tickets, so the audience may be more than just those who live near Brookline (their taping location). Still, this is a niche show with a certain demographic target, so the diversity is certainly badly skewed. Your most important point is dead-on: totally inadequate sample size, taken at one point in time.

    It's entertaining, especially when Kimball and the audience are at odds in their choices, but certainly nothing to rely on.

    1. I find complaining about the validity of speciality cooking shows to be meaningless. Don't concern yourself about something you can't change. <shrug>

      CI is a (seemingly) profitable venture and in no way are they claiming to be the last word on the reviews. Like any review its about knowing the source. You don't like the source then stick with Consumer Reports if you feel your tastes and theirs line up.

      Lots of cooking shows offers reviews of foods and products, just look at The Chew, The Kitchen, etc. So does all the morning shows-Today, Good Morning America. Most of those don't even have studio audiences weighing in.

      Don't sweat the small stuff, life is way too short.

      1. As far as the taste tests, half the time I have never even heard of the brands that they show. I've fully disagreed with many of their results.

        3 Replies
        1. re: rasputina

          Same here... what brand is that?
          I don't know and couldn't care less is the audience or Chris Kimball agrees or disagrees with me.

          1. re: rasputina

            I'm not sure I'd agree with the results of a 1500 person sample, or even a million. Taste is pretty personal.

            1. re: Savour

              Exactly- it's personal, it's regional, it's a lot of things, so why anyone gets exorcised over ATK reporting what their audience thought is beyond me.

              Take it or leave it.
              It's not like scientific research where you can extrapolate.

          2. Wow, take it for what it's worth!
            It's a tv show, not a submission to The Lancet.

            I can't prove you wrong and you can't prove you right.

            It's OPINION, not the gospel, the bible, the word of God or written in stone.

            I can't believe anyone gets upset about something they volunteer to do.

            Don't watch!

            I swear, people just look for reasons to bitch about this show/magazine/cast.
            It must release endorphins or something.

            3 Replies
            1. re: monavano

              I am not worked up about it, just pointing out the obvious. Seems like a lot of people who watch that show live and die by what they say.

              Yes it is just a show, however, they take great pride in letting everyone know how rigorously they test things. So much so that you have to pay to see some of the results because they dont want advertisers to influence the results.

              1. re: kjonyou

                It's a show.
                They go to certain lengths to attempt to ensure objectivity.
                Their business structure is what it is-been bitched about here until the cows come home.
                What else am I missing.

                1. re: kjonyou

                  Their taste tests are purely entertainment, their claims of rigorous testing apply to equipment, which audience members don't weigh in on.

                  That said, I wouldn't necessarily rely on those exclusively; I look at Consumer Reports, Chowhound, and reviews on Amazon and weight them accordingly.

              2. Have you ever seen a person of color in that audience? It's ridiculous.

                6 Replies
                    1. re: monavano

                      I am not implying it, the person above me is implying that ATK is all white in case you have not noticed.

                      Since America is quite a mix of races, its odd that even in a big city they only seem to have white audience members with limited tastes.

                      You may say food is food and it doesn't matter what your ethnic make up is, but not according to the food industry. They actually cater to ethnic difference but very small changes in the same products. So for example, Norrs Chicken base will have a slightly different formula on the exact same product depending on weather the box is written in English or in Spanish even if they are both sold in the same store.

                      1. re: kjonyou

                        Monavano's question wasn't aimed at you; it was asked of "the person above" you. Do try to keep up.

                        Also, in regard to "you may say...", well let me just say putting words in someone else's mouth is no way to prove a point.

                        1. re: kjonyou

                          Just to try to help.......

                          "I am not implying it, the person above me is implying that ATK is all white in case you have not noticed."

                          Each post that replies to anyone other than the OP shows the specific poster's name in its upper right corner. kjonyu's post says "RE: sandiasingh" at its upper right.

                          Hope thus helps you avoid unnecessary angst.