I extol the virtues of science for those who work the perimeter beyond good and evil, panthering persuasion on Other Guys for greatest effect. Any persuasion without science is just luck and as sports gambling proves, your luck always runs out. Science never does.

As long as it is science. Consider the mess you can make with both science and persuasion while appearing to clean house. I’ll summarize.

The TACT is aimed at persuading Other Guys to avoid tanning beds. Recruit 475 Other Guys on MTurk. Have them complete a battery of self report survey items related to their use of tanning beds (past frequency, intentions, risk, etc). Now randomly assign the 475 Other Guys to first either do a self affirmation play (rate values and write about your most important value) or not. Then the Other Guys read a persuasion message about tanning with either (get ready) a gain or loss frame. After reading the persuasive message, the Other Guys do more self reports on future intentions, fear response, message acceptance, and message derogation.

This is a randomized controlled experiment. Other Guys get randomly assigned to one of four conditions: 1) Affirm and Gain; 2) Affirm and Loss; 3) No Affirm and Gain; and 4) No Affirm and Loss. The affirmation manipulation is a widely used and successful operation. The gain and loss messages are pretested and conform to Prospect Theory and the research lit. The outcome measures of intention are standard as are the measures of fear, message acceptance and rejection. We’ve got a good experiment here.

Then the researchers destroy that experiment with randomization and control and adjust the outcome scores, especially with intentions, with those pretest self reports on past habit, intention, and risk. This is a standard Tooth Fairy Tale tactic, adjusting outcomes for prior variables as a means of trying to debias a biased sample of data. With a true experiment, as we have here, there is no need for these adjustments because they break the randomization. Now, we are not observing the effect of affirmation and framing on intentions, but the effect of affirmation and framing on adjusted intentions.

There are at least two reasons one might do this and neither are good. You might be dumber than your credential would indicate and you truly don’t understand the difference between randomized experiments and observational Tooth Fairy Tales. Worse, you might understand the difference exactly, but when you did the unadjusted analysis, the experiment failed. So. Take all those pre-experiment survey items and diddle with them. And, don’t report that in the paper.

I’m harsh in my criticism of the results and we haven’t even considered the data. Let’s look at them and you’ll understand my evaluation. Remember, all the analyses are conducted not on the original raw data, but data adjusted for the pre-experiment variables (intention, risk, etc.). Here’s the main table of results.

Tanning Table of Results

What’s going on here?

1. The self affirmation manipulation does change the outcome of future intention to use tanning beds. It’s a Small+ Windowpane. Better still: IT’S IN THE WRONG DIRECTION. Other Guys who did that well established and proven affirmation manipulation were LESS LIKELY to reduce tanning. They also found either persuasive message (gain or loss frame) to be weak and they hated it at Small Windowpanes.

This is simply astonishing. That affirmation manipulation typically works and strongly suggests something went wrong in the experiment. The authors never consider this, but remember this thing ran on MTurk with Other Guys sitting at home in front of a compute alone in the house sitting in their underwear. On average the Other Guys completed this work in 8 minutes. Do you think it possible they were paying no attention to this task?

2. The framing play makes a difference at Small to Medium Windowpanes on the two future intention items. Hey, for once framing works! And again: IN THE WRONG DIRECTION. Gain frames should work on protection behaviors as with quitting sun tanning while loss frames shouldn’t because they work on detection behavior. But in this analysis the loss frame worked better than the gain frame despite what Prospect Theory asserts. Why?

Remember they asked for self reported fear after getting the persuasive messages. Other Guys who got the loss frame persuasion reported more fear about the consequences of tanning than did the gain frame people and at a Large Windowpane difference. Why would either frame produce any fear and more particularly why the loss frame over the gain frame? Well, maybe this had something to do with it.

Based on the findings, the loss-framed message used for this experiment displayed an image of a young woman’s face after surgery to remove a cancerous lesion and included text conveying the risks of IT. The gain-framed message displayed an image of the face of a woman with healthy skin and used text conveying the benefits of avoiding IT. Other message features were consistent across conditions.

Except for the images, the two frames were identical. Here are the pictures. I’ll go out on a limb and bet that you can tell which was the loss frame and which was the gain frame.

Frame Loss Gain Images

Gee. Except for the pictures, both frames contain the same persuasion. And there’s no way those pictures would make any difference in MTurk Other Guys.

Clearly, this is not an experiment or even an observational Tooth Fairy Tale on affirmation and framing. This is a fear appeals MTurk fantasy. Blast Other Guys with that horrible disfigured face of a pretty young girl and you think they remember the affirmation manipulation? You think they read any of the text in the framed messages? All they see is that surgical scar.

Of course, I’m an idiot with this perspective. The researchers note.

This study is among the first to examine the impact of gain- and loss-framed IT prevention messages among young adult women and to investigate the potential moderating effect of self-affirmation. Compared with gain-framed messages, loss-framed messages produced significantly lower intentions to IT and significantly greater intentions to quit IT, and the effect of loss-framed messages occurred via a stronger emotional response. Self-affirmation did not moderate the message framing effect, but it increased young adult women’s intentions to tan and may have done so by producing greater message derogation and decreasing perceived strength of IT prevention messages, independent of their framing. These findings have implications for developing persuasive public health communications to prevent IT and reduce skin cancer risks among young women, and suggest important directions for future research.

You could say that. And apparently funding sources are buying it because they paid for it.

This research was supported in part through funding from the Harry J. Lloyd Charitable Trust. This work was also supported in part by the Georgetown Lombardi Comprehensive Cancer Center Support Grant #P30CA051008.

That P-grant is from the National Institutes of Health and more specifically the National Cancer Institute. Hey, kids, your tax and charitable dollars at work.

See the mess of science and persuasion in this. Design and execute a genuine experiment with random assignment to controlled conditions. Then break the experiment and turn it into a Tooth Fairy Tale by “adjusting” the outcome variables – most notably the future intention scores – with prior variable like habit and risk. With over 100 people per cell, you don’t need to “adjust” anything; randomization does all that which is why you randomize in the first place. And the researchers appear to have some awareness of this. They note.

Participant characteristics and descriptive statistics for all measures are shown in Table 1. Participants averaged 24.7 years of age (SD = 3.2), 40.8% were college/university students, and 66.1% reported an annual income ≤$50,000. Nearly 40% of participants were frequent indoor tanners. There were no significant differences in participant characteristics by experimental conditions, indicating successful randomization.

Yet, despite the successful randomization, the researchers busted it when they adjusted variables with the covariates when they should not have do so. The wording of that quote strongly suggests to me that the researchers did indeed first run the analysis based only upon the unadjusted experimental results and got nothing. They then busted the randomization (and provided no valid explanation in the text for this) and found absurd and contradictory results that were at least statistically significant.

Contrary to Prospect Theory and the weak and inconsistent results from meta analytic studies on framing, this study produced a more persuasive effect for loss frames when gain frames should have worked as proven by theory and dogma. And, with only a moment’s reflection and analysis, you know why they got the results they did. Again.

Frame Loss Gain Images

You’ve got Low WATT MTurkers blasting through this task when half of them get hit with that pretty young woman, now disfigured because she used tanning beds and got cancer. But, call this an experiment on affirmation and framing and you sound like you are doing science. Read the paper, however, and you see nothing but persuasion. This is a bad fear appeals study that shocked lazy Other Guys WATtapping for dollars at MTurk.

Pretty easy to get by the reviewers and editors at Health Psychology, isn’t it? So too at NIH, NCI, and the Harry Lloyd Charitable Trust.

Never gamble on persuasion without science. Your luck always runs out.

Mays, D., & Zhao, X. (2016). The influence of framed messages and self-affirmation on indoor tanning behavioral intentions in 18- to 30-year-old women. Health Psychology, 35(2), 123-130.

doi:10.1037/hea0000253.