💡
Robert Hodgson, the owner of a small California winery, got in touch with the head judge of the annual wine competition at the California State Fair, in which thousands of wines are entered each year, and suggested an experiment.17 The competition is set up so that each judge tastes a flight of thirty wines at a time. The wines are not identified, so the judge cannot be influenced by reputation or other factors. Hodgson suggested that in a number of those flights, the judges should be given three samples of the same wine. Would they give these identical samples the same rating, or would their ratings vary? The head judge agreed, and Hodgson ran this experiment at four consecutive state fairs from 2005 to 2008. He found that very few judges rated the three identical samples similarly. It was common for a judge to give scores that varied by plus or minus four points -- that is, to give one sample a 91, a second sample of the same wine an 87, and the third an 83. This is a significant difference: a 91 wine is a good wine that will fetch a premium price, while an 83 is nothing special. Some judges determined one of the three samples to be worthy of a gold medal and another of the three to be worth just a bronze medal -- or no medal at all. And while in any given year some judges were more consistent than others, when Hodgson compared them year to year, he found that judges who were consistent one year were inconsistent the next. None of the judges -- and these were sommeliers, wine critics, winemakers, wine consultants, and wine buyers -- proved to be consistent all the time.