I am a naive optimist about life. I tend to believe that the truth will eventually win which means I believe in Falling Apples in the long run. Yet, I remain gobsmacked over the continuing success of those who sell persuasion for science. Almost every time I analyze one of those papers or examples on the Persuasion Blog, while I’m pulling the curtain back on the Wizard of Oz, I sit at my computer, flabbergasted that the Wizard got so far with so little. So, too, today with another example of persuasive science, and again with a master of the form we’ve already evaluated, Nate at 538.
Nate Silver, you’ll recall, got famous doing his scientific journalism at the New York Times to call the 2012 Presidential election in advance, both at the national level and with all 50 states. He employed a method guaranteed to win in most cases because he kept adjusting his prediction every day. His success led to a bidding war for his scientific journalism and the New York Times lost him when they brought a Pretty Girl to a lawyer fight and Nate decamped to ESPN of all places, bringing his method to sports – which you’d expect at ESPN – and politics, science, health, economics, and pop culture – which you wouldn’t expect at ESPN.
Nate has issued a searing appraisal of his failure to predict and explain the success of Donald Trump in the Presidential primaries. You can read his analytic takedown of himself, but you need to think about what Nate and other scientific journalists are doing.
Nate believes that he can apply the scientific method to journalism. Truly.
Since Donald Trump effectively wrapped up the Republican nomination this month, I’ve seen a lot of critical self-assessments from empirically minded journalists — FiveThirtyEight included, twice over — about what they got wrong on Trump. This instinct to be accountable for one’s predictions is good since the conceit of “data journalism,” at least as I see it, is to apply the scientific method to the news. That means observing the world, formulating hypotheses about it, and making those hypotheses falsifiable. (Falsifiability is one of the big reasons we make predictions.1) When those hypotheses fail, you should re-evaluate the evidence before moving on to the next subject. The distinguishing feature of the scientific method is not that it always gets the answer right, but that it fails forward by learning from its mistakes.
Simply because you can employ the principles of the scientific method to something doesn’t mean that you are making science. For example, serious people did what Nate is doing with alchemy, phrenology, phlogiston, racial genetics, and astrology. They did use the scientific method, they were scientists, but alas they made no science because the thing they were studying was beyond explanation. There was no scientific there, there. They failed as scientists because they could not create a valid theory.
I’d argue that all journalism by its nature is beyond science. Nate, for example, seems to think that if he uses science he can predict social outcomes far in advance. But, the social outcomes he wants to predict are unique, N of 1, single cases, which is why we do journalism and not science on who’s going to win the Iowa Primary six weeks in advance. At best Nate and scientific or Big Data journalism is just Tooth Fairy Tales as with Nate’s prediction models for elections.
A key feature with anything scientific is that is produces theories, not just data or predictions or even hypotheses. If you look at Nate and his ilksters math, you find no serious theory that generates testable hypotheses and explains stuff – generalizable knowledge. This approach just adds and subtracts data points until it produces output that is statistically significant and accounts for some variance; a Tooth Fairy Tale just like the guys at the Harvard School of Public Health do. They invent fabulous stories about why X and Y are related . . . after they have run thousands of X and Y correlations and found the biggest. Telling stories after the fact is not science. It’s persuasion.
I think Nate is pretty Sincere in his beliefs, although with the kind of money and attention he’s getting, experience alone is shaping him into a guy at least trying to do persuasion, if not becoming a panther. For now, Nate doesn’t realize that he is studying astrology and believing that the stars, the sun, and the moon control outcomes in sports, politics, economics, health, and culture. And, he’s got data and regression models and competitors and publication outlets. How can he be wrong?
You see how the principles of persuasion can explain both Nate’s delusions and his success. Nate clearly misunderstands the work of science because he’s not trained and experienced as a scientist. He’s just never done the business. Nate can count, but counting is not science either. Nate fails because he is not in the theory business but in the Tooth Fairy Tale business and he doesn’t understand the difference. He will never create generalizable knowledge even if he can guess winners of all kinds of horse races. Think about it. If Nate had a true theory and was doing valid science, anyone could take his predictions and make millions of dollars and rarely lose. Good grief, Nate himself could do this. But, apparently he doesn’t.
Why not? If you had the kind of knowledge that Nate thinks he’s got or is inventing: 1) Why would you tell anyone else? and 2) You’d be worth a lot more than what ESPN is paying. You see the Queen of Tomorrow problem.
Again, I think Nate is pretty Sincere which attracts like minded Sinceres who think Sincerity plus the scientific method makes them special. That’s how ridiculous science has become in the public mind. In part because of the predations of Tooth Fairies in lab coats telling their tales in the scientific literature, Other Guys are thinking that they can do the same thing and be scientists, too, but without the grind and boredom and detail.