Preregistrations for all research can be found at https://osf.io/p6u8k/. In all survey experiments, we don’t exclude contributors for inattentiveness or straightlining to keep away from choice results that can undermine causal inference. The researchers weren’t blinded to the hypotheses when finishing up the analyses. All experiments have been randomized apart from examine 2, which was not randomized. No statistical strategies have been used to predetermine pattern measurement.
Study 1
In examine 1, contributors have been offered with a pretested set of false and true headlines (in ‘Facebook format’) and have been requested to point out both whether or not they thought they have been correct or not, or whether or not they would take into account sharing them on social media or not. Our prediction was that the distinction in ‘yes’ responses between false and true information (that’s, discernment) shall be higher when people are requested about accuracy than when they’re requested about sharing, whereas the distinction between politically discordant and concordant information shall be higher when they’re requested about sharing than when they’re requested about accuracy.
Participants
We preregistered a goal pattern of 1,000 full responses, utilizing contributors recruited from Amazon’s MTurk however famous that we’d retain people who accomplished the examine above the 1,000-participant quota. In complete, 1,825 contributors started the survey. However, an preliminary (pre-treatment) screener solely allowed American contributors who indicated having a Facebook or Twitter account (when proven an inventory of various social media platforms) and indicated that they might take into account sharing political content material (when proven an inventory of various content material varieties) to proceed and full the survey. The objective of those screening standards was to focus our investigation on the related subpopulation—those that share political information. The accuracy judgments of people that by no means share political information on social media aren’t related right here, given our curiosity within the sharing of political misinformation. Of the contributors who entered the survey, 153 indicated that they’d neither a Facebook nor a Twitter account, and 651 indicated that they did have both a Facebook or Twitter account however wouldn’t take into account sharing political content material. An extra 16 contributors handed the screener however didn’t end the survey and thus have been faraway from the dataset. The full pattern (imply age = 36.7) included 475 males, 516 females, and 14 contributors who chosen one other gender possibility. This examine was run on 13–14 August 2019.
Materials
We offered contributors with 18 false and 18 true information headlines in a random order for every participant. The false information headlines have been initially chosen from a third-party fact-checking web site, www.Snopes.com, and have been due to this fact verified as being fabricated and unfaithful. The true information headlines have been all correct and chosen from mainstream information retailers to be roughly up to date with the false information headlines. Moreover, the headlines have been chosen to be both pro-Democratic or pro-Republican (and equally so). This was achieved utilizing a pretest, which confirmed that the headlines have been equally partisan throughout the classes (related approaches have been described beforehand11,19,20). See Supplementary Information section 1 for particulars in regards to the pretest.
Participants in examine 1 have been additionally requested: ‘How important is it to you that you only share news articles on social media (such as Facebook and Twitter) if they are accurate?’, to which they responded on a five-point scale from ‘not at all important’ to ‘extremely important’. We additionally requested contributors about their frequency of social media use, together with a number of exploratory questions on media belief. At the tip of the survey, contributors have been requested whether or not they responded randomly at any level through the survey or looked for any of the headlines online (for instance, through Google). As famous in our preregistration, we didn’t intend to exclude these people. Participants additionally accomplished a number of further measures as a part of separate investigations (this was additionally famous within the preregistration); specifically, the seven-item cognitive reflection take a look at19, a political data questionnaire, and the optimistic and destructive affective schedule35. In addition, contributors have been requested a number of demographic questions (age, gender, schooling, revenue, and quite a lot of political and spiritual questions). The most central political partisanship query was ‘Which of the following best describes your political preference?’ {followed} by the next response choices: strongly Democratic; Democratic; lean Democratic; lean Republican; Republican; and strongly Republican. For functions of information evaluation, this was transformed to a Democratic or Republican binary variable. The full survey is accessible online in each textual content format and as a Qualtrics file, together with all knowledge (https://osf.io/p6u8k/).
Procedure
Participants within the accuracy situation got the next directions: ‘You will be presented with a series of news headlines from 2017 to 2019 (36 in total). We are interested in whether you think these headlines describe an event that actually happened in an accurate and unbiased way. Note: the images may take a moment to load.’ In the sharing situation, the center sentence was changed with ‘We are interested in whether you would consider sharing these stories on social media (such as Facebook or Twitter)’. We then offered contributors with the total set of headlines in a random order. In the accuracy situation, contributors have been requested ‘To the best of your knowledge, is this claim in the above headline accurate?’ In the sharing situation, contributors have been requested ‘Would you consider sharing this story online (for example, through Facebook or Twitter)?’ Although these sharing selections are hypothetical, headline-level analyses recommend that self-report sharing selections of reports articles equivalent to these utilized in our examine correlate strongly with precise sharing on social media36.
In each situations, the response choices have been merely ‘no’ and ‘yes’. Moreover, contributors noticed the response choices listed as both sure/no or no/sure (randomized throughout contributors—that’s, a person participant solely ever noticed ‘yes’ first or ‘no’ first).
This examine was authorised by the University of Regina Research Ethics Board (Protocol 2018-116).
Analysis plan
Our preregistration specified that every one analyses can be carried out on the stage of the person merchandise (that’s, one knowledge level per merchandise per participant; 0 = no, 1 = sure) utilizing linear regression with strong customary errors clustered on participant. However, we subsequently realized that we must also be clustering customary errors on headline (as a number of scores of the identical headline are non-independent in the same approach to a number of scores from the identical participant), and thus deviated from the preregistrations on this minor approach (all key outcomes are qualitatively equal if solely clustering customary errors on participant). The linear regression was preregistered to have the next impartial variables: a situation dummy (−0.5 = accuracy, 0.5 = sharing), a information kind dummy (−0.5 = false, 0.5 = true), a political concordance dummy (−0.5 = discordant, 0.5 = concordant), and all two-way and three-way interactions. (Political concordance is outlined based mostly on the match between content material and beliefs; particularly, political concordant = pro-Democratic [pro-Republican] information (based mostly on a pretest) for American people preferring the Democratic [Republican] social gathering over the Republican [Democratic]. Politically discordant is the other.) Our key prediction was that there can be a destructive interplay between situation and information kind, such that the distinction between false and true is smaller within the sharing situation than the accuracy situation. A secondary prediction was that there can be a optimistic interplay between situation and concordance, such that the distinction between concordant and discordant is bigger within the sharing situation than the accuracy situation. We additionally mentioned we’d examine for a three-way interplay, and use a Wald take a look at of the related internet coefficients to take a look at how sharing probability of false concordant headlines compares to true discordant headlines. Finally, as robustness checks, we mentioned we’d repeat the primary evaluation utilizing logistic regression as a substitute of linear regression, and utilizing scores which might be z-scored inside situation.
Study 2
Study 2 prolonged the remark of examine 1 that most individuals self-report that it is crucial to not share accuracy info on social media. First, examine 2 assesses the relative, as well as to absolute, significance positioned on accuracy by additionally asking in regards to the significance of varied different elements. Second, examine 2 examined whether or not the outcomes of examine 1 would generalize past MTurk by recruiting contributors from Lucid for Academics, delivering a pattern that matches the distribution of American residents on age, gender, ethnicity and geographical area. Third, examine 2 averted the potential spillover results from examine 1 situation project instructed in Extended Data Fig. 1 by not having contributors full a job associated to social media beforehand.
In complete, 401 contributors (imply age of 43.7) accomplished the survey on 9–12 January 2020, together with 209 males and 184 females, and eight indicating different gender identities. Participants have been requested ‘When deciding whether to share a piece of content on social media, how important is it to you that the content is…’ after which got a response grid the place the columns have been labelled ‘not at all’, ‘slightly’, ‘moderately’, ‘very’, and ‘extremely’, and the rows have been labelled ‘accurate’, ‘surprising’, ‘interesting’, ‘aligned with your politics’ and ‘funny’.
This examine was authorised by the MIT COUHES (protocol 1806400195).
Studies 3, Four and 5
In research 3, 4, and 5 we examine whether or not subtly shifting attention to accuracy will increase the veracity of the information persons are keen to share. In specific, contributors have been requested to choose the accuracy of a single (politically impartial) information headline at first of the examine, ostensibly as a part of a pretest for one more examine. We then examined whether or not this accuracy-cue impacts the tendency of people to discern between false and true information when making subsequent judgments about social media sharing. The principal benefit of this design is that the manipulation is refined and never explicitly linked to the primary job. Thus, though social desirability bias might lead folks to underreport their probability of sharing misinformation total, it’s unlikely that any between-condition distinction is pushed by contributors believing that the accuracy query at first of the remedy situation was designed to make them take accuracy into consideration when making sharing selections throughout the primary experiment. It is due to this fact comparatively unlikely that any remedy impact on sharing can be due to demand traits or social desirability.
The solely distinction between research Three and Four was the set of headlines used, to exhibit the generalizability of those findings. Study 5 used a extra consultant pattern and included an energetic management situation and a second remedy situation that primed accuracy considerations another way. Studies Three and Four have been authorised by the Yale University Committee for the Use of Human Subjects (IRB protocol 1307012383). Study 5 was authorised by the University of Regina Research Ethics Board (protocol 2018-116).
Participants
In examine 3, we preregistered a goal pattern of 1,200 contributors from MTurk. In complete, 1,254 contributors started the survey between 4–6 October 2017. However, 21 contributors reporting not having a Facebook profile on the outset of the examine and, as per our preregistration, weren’t allowed to proceed; and 71 contributors didn’t full the survey. The full pattern (imply age of 33.7) included 453 males, 703 females, and a couple of who didn’t reply the query. Following the primary job, contributors have been requested whether or not they ‘would ever consider sharing something political on Facebook’ and got the next response choices: ‘yes’, ‘no’, and ‘I don’t use social media’. As per our preregistration, solely contributors who chosen ‘yes’ to this query have been included in our major evaluation. This excluded 431 folks and the pattern of contributors who would take into account sharing political content material (imply age of 34.5) included 274 males, 451 females, and a couple of who didn’t reply the gender query.
In examine 4, we preregistered a goal pattern of 1,200 contributors from MTurk. In complete, 1,328 contributors started the survey between 28–30 November 2017. However, Eight contributors didn’t report having a Facebook profile and 72 contributors didn’t end the survey. The full pattern (imply age of 33.3) included 490 males, 757 females, and 1 who didn’t reply the query. Restricting to contributors have been responded ‘Yes’ when requested whether or not they ‘would ever consider sharing something political on Facebook’ excluded 468 folks, such that the pattern of contributors who would take into account sharing political content material (imply age of 33.6) included 282 males, 497 females, and 1 who didn’t reply the gender query.
In examine 5, we preregistered a goal pattern of 1,200 contributors from Lucid. In complete, 1,628 contributors started the survey between 30 April and 1 May 2019. However, 236 contributors reported not having a Facebook profile (and thus weren’t allowed to full the survey) and 105 contributors didn’t end the survey. The full pattern (imply age of 45.5) included 626 males and 661 females. Restricting to contributors have been responded ‘yes’ when requested whether or not they ‘would ever consider sharing something political on Facebook’ excluded 616 folks, such that the pattern of contributors who would take into account sharing political content material (imply age of 44.3) included 333 males and 338 females.
Unlike in examine 1, as a result of the query about ever sharing political content material was requested after the experimental manipulation (reasonably than on the outset of the examine), there’s the likelihood that excluding contributors who responded ‘no’ might introduce choice results and undermine causal inference37. Although there was no vital distinction in responses to this political sharing query between situations in any of the three accuracy priming experiments (χ2 take a look at; examine 3: χ2 (1, n = 1,158) = 0.156, P = 0.69; examine 4: χ2 (1, n = 1,248) = 0.988, P = 0.32; examine 5, χ2 (3, n = 1,287) = 2.320, P = 0.51), for completeness we present that the outcomes are strong to together with all contributors (see Supplementary Information section 2).
Materials
In examine 3, we offered contributors with 24 information headlines from ref. 20; in examine 4, we offered contributors with a unique set of 24 information headlines chosen through pretest; and in examine 5, we offered contributors with one more set of 20 information headlines chosen through pretest. In all research, half of the headlines have been false (chosen from a third-party fact-checking web site, www.Snopes.com, and due to this fact verified as being fabricated and unfaithful) and the opposite half have been true (correct and chosen from mainstream information retailers to be roughly up to date with the false information headlines). Moreover, half of the headlines have been pro-Democratic or anti-Republican and the opposite half have been pro-Republican or anti-Democrat (as decided by the pretests). See Supplementary Information section 1 for additional particulars on the pretests.
As in examine 1, after the primary job, contributors in research 3–5 have been requested in regards to the significance of sharing solely correct information articles on social media (examine Four additionally requested in regards to the significance contributors’ mates positioned on sharing solely correct information on social media). Participants then accomplished varied exploratory measures and demographics. The demographics included the query ‘If you absolutely had to choose between only the Democratic and Republican party, which would do you prefer?’ {followed} by the next response choices: Democratic Party or Republican Party. We use this query to classify contributors as Democrats versus Republicans.
Procedure
In all three research, contributors have been first requested whether or not they had a Facebook account, and people who didn’t weren’t permitted to full the examine. Participants have been then randomly assigned to certainly one of two situations in research Three and 4, and certainly one of 4 situations in examine 5.
In the ‘treatment’ situation of all three research, contributors got the next directions: ‘First, we would like to pretest an actual news headline for future studies. We are interested in whether people think it is accurate or not. We only need you to give your opinion about the accuracy of a single headline. We will then continue on to the primary task. Note: the image may take a moment to load.’ Participants have been then proven a politically impartial headline and have been requested: ‘To the best of your knowledge, how accurate is the claim in the above headline?’ and got the next response scale: ‘not at all accurate’, ‘not very accurate’, ‘somewhat accurate’, ‘very accurate’. One of two politically impartial headlines (1 true, 1 false) was randomly chosen in research Three and 4; certainly one of 4 politically impartial headlines (2 true, 2 false) was randomly chosen in examine 5.
In the ‘active control’ situation of examine 5, contributors have been as a substitute given the next directions: ‘First, we would like to pretest an actual news headline for future studies. We are interested in whether people think it is funny or not. We only need you to give your opinion about the funniness of a single headline. We will then continue on to the primary task. Note: the image may take a moment to load.’ They have been then offered with one of many similar 4 impartial information headlines used within the remedy situation and requested: ‘In your opinion, is the above headline funny, amusing, or entertaining?’. (Response choices: extraordinarily unfunny; reasonably unfunny; barely unfunny; barely humorous; reasonably humorous; extraordinarily humorous.)
In the ‘importance treatment’ situation of examine 5, contributors have been as a substitute requested the next query on the outset of the examine: ‘Do you agree or disagree that ‘it is important to only share news content on social media that is accurate and unbiased’?’. (Response choices: strongly agree to strongly disagree.)
In the ‘control’ situation of all three research, contributors obtained no preliminary directions and proceeded immediately to the following step.
Participants in all situations have been then instructed: ‘You will be presented with a series of news headlines from 2016 and 2017 (24 in total) [2017 and 2018 (20 in total) for study 5]. We are interested in whether you would be willing to share the story on Facebook. Note: The images may take a moment to load.’ They then proceeded to the primary job during which they have been offered with the true and false headlines and for every have been requested ‘If you were to see the above article on Facebook, how likely would you be to share it’ and given the next response scale: ‘extremely unlikely, moderately unlikely, slightly unlikely, slightly likely, moderately likely, extremely likely’. We used a steady scale, as a substitute of the binary scale utilized in examine 1, to improve the sensitivity of the measure.
Analysis plan
Our preregistrations specified that every one analyses can be carried out on the stage of the person merchandise (that’s, one knowledge level per merchandise per participant, with the six-point sharing Likert scale rescaled to the interval [0, 1]) utilizing linear regression with strong customary errors clustered on participant. However, we subsequently realized that we must also be clustering customary errors on headline (as a number of scores of the identical headline are non-independent in the same approach to a number of scores from the identical participant), and thus deviated from the preregistrations on this minor approach (all key outcomes are qualitatively equal if solely clustering customary errors on participant).
In research Three and 4, the important thing preregistered take a look at was an interplay between a situation dummy (0 = management, 1 = remedy) and a information veracity dummy (0 = false, 1 = true). This was to be followed-up by checks for easy results of reports veracity in every of the 2 situations; and, particularly, the impact was predicted to be bigger within the remedy situation. We additionally deliberate to take a look at for easy results of situation for every of the 2 forms of information; and, particularly, the impact was predicted to be bigger for false relative to true information. We additionally performed a submit hoc evaluation utilizing a linear regression with strong customary errors clustered on participant and headline to look at the potential moderating position of a dummy for the participant’s partisanship (choice for the Democratic versus Republican social gathering) and a dummy for the headline’s political concordance (pro-Democratic [pro-Republican] headlines scored as concordant for contributors who most popular the Democratic [Republican] social gathering; pro-Republican [pro-Democratic] headlines scored as discordant for contributors who most popular the Democratic [Republican] social gathering). For ease of interpretation, we z-scored the partisanship and concordance dummies, after which included all attainable interactions within the regression mannequin. To maximize statistical energy for these moderation analyses, we pooled the information from research Three and 4.
In examine 5, the primary preregistered take a look at was to evaluate whether or not the energetic and passive management situations differed, by testing for vital a major impact of situation (0 = passive, 1 = energetic), or vital interplay between situation and information veracity (0 = false, 1 = true). If these didn’t differ, we preregistered that we’d mix the 2 management situations for subsequent analyses. We would then take a look at whether or not the 2 remedy situations differ from the management situation(s) by testing for an interplay between dummies for every remedy (0 = passive or energetic management, 1 = remedy being examined) and information veracity. This was to be followed-up by checks for easy results of reports veracity in every of the situations; and, particularly, the impact was predicted to be bigger within the remedy situations. We additionally deliberate to take a look at for easy results of situation for every of the 2 forms of information; and, particularly, the impact was predicted to be bigger for false relative to true information.
Study 6
Studies 3, Four and 5 discovered {that a} refined reminder of the idea of accuracy decreased sharing of false (however not true) information. In examine 6, we as a substitute use a full-attention remedy that immediately forces contributors to take into account the accuracy of every headline earlier than deciding whether or not to share it. This permits us to decide, inside this specific context, the utmost impact that can be obtained by focusing attention on accuracy. Furthermore, utilizing the accuracy scores elicited within the full-attention remedy, we can decide what fraction of shared content material was believed to be correct versus inaccurate by the sharer. Together, these analyses permit us to infer the fraction of sharing of false content material that’s attributable to inattention, confusion about veracity, and purposeful sharing of falsehood.
This examine was authorised by the Yale University Committee for the Use of Human Subjects (IRB protocol 1307012383).
Participants
We mix two rounds of information assortment on MTurk, the primary of which had 218 contributors start the examine on 11 August 2017, and the second of which had 542 contributors start the examine on 24 August 2017, for a complete of 760 contributors. However, 14 contributors didn’t report having a Facebook profile and 33 contributors didn’t end the survey. The full pattern (imply age of 34.0) included 331 males, 376 females, and Four who didn’t reply the query. Participants have been requested whether or not they ‘would ever consider sharing something political on Facebook’ and got the next response choices: ‘yes’, ‘no’, ‘I don’t use social media’. Only contributors who chosen ‘yes’ to this query have been included in our major evaluation, as in our different research (there was no vital distinction in responses between situations, χ2(2) = 1.07, P = 0.585). This excluded 313 folks and the ultimate pattern (imply age of 35.2) included 181 males, 213 females, and Four who didn’t reply the gender query. For robustness, we additionally report analyses together with all contributors; see Extended Data Table 2.
Materials
We offered contributors with the identical 24 headlines utilized in examine 3.
Procedure
Participants have been first requested if they’ve a Facebook account and people who didn’t weren’t permitted to full the examine. Participants have been then randomly assigned to certainly one of two situations. In the full-attention remedy situation, contributors got the next directions: ‘You will be presented with a series of news headlines from 2016 and 2017 (24 in total). We are interested in two things: (i) Whether you think the headlines are accurate or not. (ii) Whether you would be willing to share the story on Facebook. Note: the images may take a moment to load.’ In the management situation, contributors have been instructed: ‘You will be presented with a series of news headlines from 2016 and 2017 (24 in total). We are interested in whether you would be willing to share the story on Facebook. Note: the images may take a moment to load.’ Participants in each situations have been requested ‘If you were to see the above article on Facebook, how likely would you be to share it’ and given the next response scale: ‘extremely unlikely’, ‘moderately unlikely’, ‘slightly unlikely’, ‘slightly likely’, ‘moderately likely’, ‘extremely likely’. Crucially, within the remedy situation, earlier than being requested the social media sharing query, contributors have been requested: ‘To the best of your knowledge, how accurate is the claim in the above headline?’ and given the next response scale: ‘not at all accurate’, ‘not very accurate’, ‘somewhat accurate’, ‘very accurate’.
Analysis
The objective of our analyses is to decide what fraction of the sharing of false headlines is attributable to confusion (incorrectly believing the headlines are correct), inattention (forgetting to take into account the accuracy of the headlines; as per the inattention-based account), and purposeful sharing of false content material (as per the preference-based account). We can accomplish that through the use of the sharing intentions in each situations, and the accuracy judgments within the ‘full-attention’ remedy (no accuracy judgments have been collected within the management). Because contributors within the full-attention remedy are pressured to take into account the accuracy of every headline earlier than deciding whether or not they would share it, inattention to accuracy is solely eradicated within the full-attention remedy. Thus, the distinction in sharing of false headlines between management and full-attention remedy signifies the fraction of sharing in management that was attributable to inattention. We can then use the accuracy judgments to decide how a lot of the sharing of false headlines within the full-attention remedy was attributable to confusion (indicated by the fraction of shared headlines that contributors rated as correct) versus purposeful sharing (indicated by the fraction of shared headlines that contributors rated as inaccurate).
Concretely, we do the evaluation as follows. First, we dichotomize responses, classifying sharing intentions of ‘extremely unlikely’, ‘moderately unlikely’, and ‘slightly unlikely’ as ‘unlikely to share’ and ‘slightly likely’, ‘moderately likely’, and ‘extremely likely’ as ‘likely to share’; and classifying accuracy scores of ‘not at all accurate’ and ‘not very accurate’ as ‘not accurate’ and ‘somewhat accurate’ and ‘very accurate’ as ‘accurate’. Then we outline the fraction of sharing of false content material due to every issue as follows:
$${f}_{{rm{Inattention}}}=frac{{F}_{{rm{cont}}}-{F}_{{rm{deal with}}}}{{F}_{{rm{cont}}}}$$
$${f}_{{rm{Confusion}}}=frac{{N}_{{rm{deal with}}}^{{rm{acc}}}}{{N}_{{rm{deal with}}}}frac{{F}_{{rm{deal with}}}}{{F}_{{rm{cont}}}}$$
$${f}_{{rm{Purposeful}}}=frac{{N}_{{rm{deal with}}}^{{rm{inacc}}}}{{N}_{{rm{deal with}}}}frac{{F}_{{rm{deal with}}}}{{F}_{{rm{cont}}}}$$
In which, Fcont denotes the fraction of false headlines shared within the management; Fdeal with denotes the fraction of false headlines shared within the remedy group; Ndeal with denotes the variety of false headlines shared within the remedy group, ({N}_{{rm{deal with}}}^{{rm{acc}}}) denotes the variety of false headlines shared and rated correct within the remedy group, and ({N}_{{rm{deal with}}}^{{rm{inacc}}}) denotes the variety of false headlines shared and rated inaccurate within the remedy group.
For an intuitive visualization of those expressions, see Fig. 2d.
To calculate confidence intervals on our estimates of the relative impact of inattention, confusion, and purposeful sharing, we use bootstrapping simulations. We create 10,000 bootstrap samples by sampling with substitute on the stage of the topic. For every pattern, we calculate the distinction in fraction of sharing of false info defined by every of the three elements (that’s, the three pairwise comparisons). We then decide a two-tailed P worth for every comparability by doubling the fraction of samples during which the issue that explains much less of the sharing within the precise knowledge are discovered to clarify extra of the sharing.
Preregistration
Although we did full a preregistration in reference to this experiment, we don’t observe it right here. The analyses we preregistered merely examined for an impact of the manipulation on sharing discernment, as in research 3–5. After conducting the experiment, we realized that we might analyse the information in another approach to achieve perception into the related impact of the three causes for sharing misinformation described on this Article. It is these (submit hoc) analyses that we deal with. Notably, Extended Data Table 2 exhibits that equal outcomes are obtained when analysing the 2 samples individually (the primary being a pilot for the pre-registered experiment, and the second being the pre-registered experiment), serving to to handle the submit hoc nature of those analyses.
Study 7
In examine 7, we set out to take a look at whether or not the outcomes of the survey experiments in research 3–5 would generalize to actual sharing selections ‘in the wild’, and to deceptive however not blatantly false information. Thus, we performed a digital subject experiment on Twitter during which we delivered the identical intervention from the ‘treatment’ situation of the survey experiments to customers who had beforehand shared hyperlinks to unreliable information websites. We then examined the impact of receiving the intervention on the standard of the information that they subsequently shared. The experiment was authorised by Yale University Committee of the Use of Human Subjects IRB protocol 2000022539 and MIT COUHES Protocol 1806393160. Although all evaluation code is posted online, we didn’t publicly submit the information owing to privateness considerations (even with de-identified knowledge, it might be attainable to determine lots of the customers within the dataset by matching their tweet histories with publicly out there knowledge from Twitter). Researchers keen on accessing the information are requested to contact the corresponding authors.
Study 7 is an aggregation of three totally different waves of information assortment, the small print of that are summarized in Extended Data Table 3. (These are all the knowledge that we collected, and the choice to conclude the information assortment was made earlier than working any of the analyses reported on this Article.)
Participants
The primary experimental design concerned sending a non-public direct message to customers asking them to charge the accuracy of a headline (as within the ‘treatment’ situation of the survey experiments). Twitter solely permits direct messages to be despatched from account X to account Y if account Y follows account X. Thus, our first job was to assemble a set of accounts with a considerable variety of followers (who we might then ship direct messages to). In specific, we would have liked followers who have been possible to share misinformation. Our method was as follows.
First, we created an inventory of tweets with hyperlinks to certainly one of two information websites that skilled fact-checkers rated as extraordinarily untrustworthy27 however which might be nonetheless pretty standard: www.Breitbart.com and www.infowars.com. We recognized these tweets by (i) retrieving the timeline of the Breitbart Twitter account utilizing the Twitter REST API (Infowars had been banned from Twitter once we have been conducting our experiment and thus had no Twitter account), and (ii) trying to find tweets that comprise a hyperlink to the corresponding area utilizing the Twitter superior search function and amassing the tweet IDs both manually (wave 1) or through scraping (waves 2 and three). Next, we used the Twitter API to retrieve lists of customers who retweeted every of these tweets (we periodically fetched the listing of ‘retweeters’ as a result of the Twitter API solely gives the final 100 customers ‘retweeters’ of a given tweet). As proven in Extended Data Table 3, throughout the three waves this course of yielded a possible participant listing of 136,379 complete Twitter customers with some historical past of retweeting hyperlinks to deceptive information websites.
Next, we created a sequence of accounts with innocuous names (for instance, ‘CookingBot’); we created new accounts for every experimental wave. Each of the customers within the potential participant listing was then randomly assigned to be {followed} by certainly one of our accounts. We relied on the tendency of Twitter customers to reciprocally follow-back to create our set of followers. Indeed, 8.3% of the customers that have been {followed} by certainly one of our accounts selected to observe our account again. This yielded a complete of 11,364 followers throughout the three waves. (After the completion of our experiments, Twitter has made it considerably more durable to observe giant numbers of accounts with out getting suspended, which creates a problem for utilizing this method in future work; an answer is to use the focused promoting on Twitter to goal adverts whose objective is the accruing of followers because the set of customers one would really like to have in a single’s topic pool.)
To decide eligibility and to permit blocked randomization, we then recognized (i) customers’ political ideology utilizing the algorithm from Barberá et al.38; (ii) the likelihood of them being a bot, utilizing the bot-or-not algorithm39; (iii) the variety of tweets to one of many 60 web sites with fact-checker scores that may type our high quality measure; and (iv) the common fact-checker score (high quality rating) throughout these tweets.
For waves 1 and a couple of, we excluded customers who tweeted no hyperlinks to any of the 60 websites in our listing within the two weeks earlier than the experiment; who couldn’t be given an ideology rating; who couldn’t be given a bot rating; or who had a bot rating above 0.5 (in wave 1, we additionally excluded a small variety of very high-frequency tweeters for whom we have been unable to retrieve all related tweets due to the three,200-tweet restrict of the Twitter API). In wave 3, we took a unique method to avoiding bots, specifically avoiding high-frequency tweeters. Specifically, we excluded contributors who tweeted greater than 30 hyperlinks to one of many 60 websites in our listing within the two weeks earlier than the experiment, in addition to excluding those that tweeted fewer than 5 hyperlinks to one of many 60 websites (to keep away from lack of sign). This resulted in a complete of 5,379 distinctive Twitter customers throughout the three waves. (Note that these exclusions have been utilized ex ante, and excluded customers weren’t included within the experiment, reasonably than implementing submit hoc exclusions.)
One could be involved about systematic variations between the customers we included in our experiments versus those that we {followed} however didn’t observe us again. To achieve some perception into this query, we in contrast the traits of the 5,379 customers in our experiment to a random pattern of 10,000 customers that we {followed} however did observe us again (sampled proportional to the variety of customers in every wave). For every consumer we retrieved variety of followers, variety of accounts {followed}, variety of favourites, and variety of tweets. We additionally estimated political ideology as per Barberá et al.38, likelihood of being a bot39, and age and gender utilizing based mostly on profile photos utilizing the Face Plus Plus algorithm40,41,42. Finally, we checked whether or not the account had been suspended or deleted. As proven in Extended Data Fig. 5, relative to customers who didn’t observe us again, the customers that took half in our experiment {followed} extra accounts, had extra followers, chosen extra favorite tweets, have been extra conservative, have been older, and have been extra possible to be bots (P < 0.001 for all); and have been additionally extra possible to have had their accounts suspended or deleted (P = 0.012). These observations recommend that to the extent that our recruitment course of induced choice, it’s in a course that works towards the effectiveness of our remedy: the customers in our experiment are possible to be much less receptive to the intervention than customers extra typically, and due to this fact our impact measurement is probably going to be an underestimate of the impact we’d have noticed within the full pattern.
Materials and process
The remedy in examine 7 was very related to the survey experiments. Users have been despatched a direct message asking them to charge the accuracy of a single non-political headline (Fig. 4b). An benefit of our design is that this direct message is coming from an account that the consumer has themselves opted in to following, reasonably than from a very unknown account. Furthermore, the direct message begins by saying ‘Thanks for following me!’ and sending such thank-you direct messages is a typical observe on Twitter. These elements ought to considerably mitigate any chance of the customers feeling suspicious or that they’re being surveilled by our account, and as a substitute make the direct message seem extra a typical interplay on Twitter.
We didn’t count on customers to reply to our message. Instead, our intervention was based mostly on the concept that merely studying the opening line (‘How accurate is this headline?’) would make the idea of accuracy extra salient. Because we couldn’t reliably observe whether or not (or when) customers learn the message (as a result of many customers’ privateness settings stop the sending of read-receipts), we carried out intent-to-treat analyses that included all topics and assumed that remedy started as quickly because the message was despatched. Furthermore, to keep away from demand results, customers weren’t knowledgeable that the message was being despatched as a part of a analysis examine, and the accounts from which we despatched the messages had innocuous descriptions (equivalent to ‘Cooking Bot’). Not informing customers in regards to the examine was important for ecological validity, and we felt that the scientific and sensible advantages justified this method provided that the potential hurt to contributors was minimal, and the tweet knowledge have been all publicly out there. See Supplementary Information section 4 for extra dialogue on the ethics of digital subject experimentation.
Because of the speed limits of direct message imposed by Twitter, we might solely ship direct message to roughly 20 customers per account per day. Thus, we performed every wave in a sequence of 24-h blocks during which a small subset of customers was despatched a direct message every day. All tweets and retweets posted by all customers within the experiment have been collected on every day of the experiment. All hyperlinks in these tweets have been extracted (together with increasing shortened URLs). The dataset was thus composed of the subset of those hyperlinks that linked to certainly one of 60 websites whose trustworthiness had been rated by skilled fact-checkers in earlier work27 (with the information entry for a given remark being the belief rating of the linked web site).
To permit for causal inference, we used a stepped-wedge (additionally referred to as randomized roll-out) design during which customers have been randomly assigned to a remedy date. This permits us to analyse tweets made throughout every of the 24-h remedy blocks, evaluating tweets from customers who obtained the direct message initially of a given block (‘treated’) to tweets from customers who had not but been despatched a direct message (‘control). Because the treatment date is randomly assigned, it can be inferred that any systematic difference revealed by this comparison was caused by the treatment. (Wave 2 also included a subset of users who were randomly assigned to never receive the direct message.) To improve the precision of our estimate, random assignment to treatment date was approximately balanced across bot accounts in all waves, and across political ideology, number of tweets to rated sites in the two weeks before the experiment, and average quality of those tweets across treatment dates in waves 2 and 3.
Because our treatment was delivered via the Twitter API, we were vulnerable to unpredictable changes to, and unstated rules of, the API. These gave rise to several deviations from our planned procedure. On day 2 of wave 1, fewer than planned direct messages were sent as our accounts were blocked part way through the day; and no direct messages were sent on day 3 of wave 1 (hence, that day is not included in the experimental dataset). On day 2 of wave 2, Twitter disabled the direct message feature of the API for the day, so we were unable to send the direct messages in an automated fashion as planned. Instead, all 370 direct messages sent on that day were sent manually over the course of several hours (rather than simultaneously). On day 3 of wave 2, the API was once again functional, but partway through sending the direct messages, the credentials for our accounts were revoked and no further direct messages were sent. As a result, only 184 of the planned 369 direct messages were sent on that day. Furthermore, because we did not randomize the order of users across stratification blocks, the users on day 3 who were not sent a direct message were systematically different from those who were sent a direct message. (As discussed in detail below, we consider analyses that use an intent-to-treat approach for wave 2 day 3—treating the data as if all 369 direct messages had indeed been sent—as well as analyses that exclude the data from wave 2 day 3.)
Analysis plan
As the experimental design and the data were substantially more complex than the survey experiment studies and we lacked well-established models to follow, it was not straightforward to determine the optimal way to analyse the data in study 7. This is reflected, for example, in the fact that wave 1 was not preregistered, two different preregistrations were submitted for wave 2 (one before data collection and one following data collection but before analysing the data), and one preregistration was submitted for wave 3, and each of the preregistrations stipulated a different analysis plan. Moreover, after completing all three waves, we realized that all of the analyses proposed in the preregistrations do not actually yield valid causal inferences because of issues involving missing data (as discussed in more detail below in the ‘Dependent variable’ part). Therefore, as a substitute of conducting a selected preregistered evaluation, we take into account the sample of outcomes throughout a spread of cheap analyses.
All analyses are performed on the consumer–day stage utilizing linear regression with heteroscedasticity-robust customary errors clustered on consumer. All analyses embody all customers on a given day who haven’t but obtained the direct message in addition to customers who obtained the direct message on that day (customers who obtained the direct message greater than 24 h earlier than the given day aren’t included). All analyses use a post-treatment dummy (0 = consumer has not but been despatched a direct message, 1 = consumer obtained the direct message that day) as the important thing impartial variable. We be aware that that is an intent-to-treat method that assumes that every one direct messages on a given day are despatched at precisely the identical time, and counts all tweets within the subsequent 24-h block as post-treatment. Thus, to the extent that technical points prompted tweets on a given day to be despatched earlier or later than the desired time, this method might underestimate the remedy impact.
The analyses we take into account differ within the following methods: dependent variable, mannequin specification, kind of tweet thought of, method to dealing with randomization failure, and method to figuring out statistical significance. We now talk about every of those dimensions in additional element.
1. Dependent variable
We take into account three alternative ways of quantifying tweet high quality. Across approaches, a key subject is how to take care of lacking knowledge. Specifically, on days when a given consumer doesn’t tweet any hyperlinks to rated websites, the standard of their tweeted hyperlinks is undefined. The method implied in our preregistrations was to merely omit lacking consumer–days (or to conduct analyses on the stage of the tweet). Because the remedy is anticipated to affect the likelihood of tweeting, nonetheless, omitting lacking consumer–days has the potential to create choice and thus undermine causal inference (and tweet-level analyses are much more problematic). For instance, if a consumer tweets because of being handled however wouldn’t have tweeted had they been within the management (or doesn’t tweet because of remedy however would have tweeted have they been within the management), then omitting the lacking consumer–days breaks the independence between remedy and potential outcomes ensured by random project. Given that solely 47.0% of user-days contained not less than one tweeted hyperlink to a rated web site, such points are probably fairly problematic. We due to this fact take into account three approaches to tweet high quality that keep away from this lacking knowledge downside.
The first measure is the common relative high quality rating. This measure assigns every tweeted hyperlink a relative high quality rating by taking the beforehand described fact-checker belief score27 (high quality rating, [0, 1], out there for 60 information websites) of the area being linked to, and subtracting the baseline high quality rating (the common high quality rating of all pre-treatment tweets throughout all customers in all the experimental days). Each consumer–day is then assigned a mean relative high quality rating by averaging the relative high quality rating of all tweets made by the consumer in query on the day in query; and customers who didn’t tweet on a given day are assigned a mean relative high quality rating of 0 (thus avoiding the lacking knowledge downside). Importantly, this measure is sort of conservative as a result of the (roughly half of) post-treatment consumer–days during which knowledge are lacking are scored as ‘0’. Thus, this measure assumes that the remedy had no impact on customers who didn’t tweet on the remedy day. If, as a substitute, non-tweeting customers would have proven the identical impact had they really tweeted, the estimated impact measurement can be roughly twice as giant as what we noticed right here. We be aware that this measure is equal to utilizing common high quality scores (reasonably than relative high quality rating) and imputing the baseline high quality rating to fill lacking knowledge (so assuming that on lacking days, the consumer’s behaviour matches the pre-treatment common).
The second measure is the summed relative high quality rating. This measure assigns every tweeted hyperlink a relative high quality rating in the identical method described above. A summed relative high quality rating of the consumer–day is then Zero plus the sum of the relative high quality scores of every hyperlink tweeted by that consumer on that day. Thus, the summed relative high quality rating will increase as a consumer tweets extra and better high quality hyperlinks, and reduces because the consumer tweets extra and decrease high quality hyperlinks; and, as for the common relative high quality rating, customers who tweet no rated hyperlinks obtained a rating of 0. As this measure is unbounded in each the optimistic and destructive instructions, and the distribution accommodates excessive values in each instructions, we winsorize summed relative high quality scores by changing values above the 95th percentile with the 95th percentile, and changing values beneath the fifth percentile with values beneath the fifth percentile (our outcomes are qualitatively strong to different selections of threshold at which to winsorize).
The third measure is discernment, or the distinction within the variety of hyperlinks to mainstream websites versus misinformation websites shared on a given consumer–day. This measure is generally intently analogous to the analytic method taken in research 2-4. To assess the impact of the intervention on discernment, we remodel the information into lengthy format such that there are two observations per consumer–day, one indicating the variety of tweets to mainstream websites and the opposite indicating the variety of tweets to misinformation websites (as beforehand outlined27). We then embody a supply kind dummy (0 = misinformation, 1 = mainstream) within the regression, and work together this dummy with every impartial variable. The remedy will increase discernment if there’s a vital optimistic interplay between the post-treatment dummy and the supply kind dummy. As these depend measures are unbounded within the optimistic course, and the distributions comprise excessive values, we winsorize by changing values above the 95th percentile of all values with the 95th percentile of all values (our outcomes are qualitatively strong to different selections of threshold at which to winsorize).
Finally, as a management evaluation, we additionally take into account the remedy impact on the variety of tweets in every consumer–day that didn’t comprise hyperlinks to any of the 60 rated information websites. As this depend measure is unbounded within the optimistic course, and the distribution accommodates excessive values, we winsorize by changing values above the 95th percentile of all values with the 95th percentile of all values (our outcomes are qualitatively strong to different selections of threshold at which to winsorize).
2. Determining statistical significance
We take into account the outcomes of two totally different strategies for computing P values for every mannequin. The first is the usual method, during which regression is used along with asymptotic inference utilizing Huber–White cluster-robust sandwich customary errors clustered on consumer to calculate P values. The second makes use of Fisherian randomization inference (FRI) to compute a precise P worth (that’s, has not more than the nominal kind I error charge) in finite samples28,43,44,45. FRI is non-parametric and thus doesn’t require any modelling assumptions about potential outcomes. Instead, the stochastic project mechanism decided by redrawing the remedy schedule, precisely as achieved within the authentic experiment, determines the distribution of the take a look at statistic below the null speculation45. On the idea of our stepped-wedge design, our remedy corresponds to the day on which the consumer receives the direct message. Thus, to carry out FRI, we create 10,000 permutations of the assigned remedy day for every consumer by re-running the random project process utilized in every wave, and recompute the t-statistic for the coefficient of curiosity in every mannequin in every permutation. We then decide P values for every mannequin by computing the fraction of permutations that yielded t-statistics with absolute worth bigger than the t-statistic noticed within the precise knowledge. Note that due to this fact, FRI takes into consideration the small print of the randomization process that roughly balanced remedy date throughout bots in all waves, and throughout ideology, tweet frequency, and tweet high quality in waves 2 and three.
3. Model specification
We take into account 4 totally different mannequin specs. The first contains wave dummies. The second post-stratifies on wave by interacting centred wave dummies with the post-treatment dummy. This specification additionally permits us to assess whether or not any noticed remedy impact considerably differs throughout waves by performing a joint significance take a look at on the interplay phrases. The third contains date dummies. The fourth post-stratifies on date by interacting centred date dummies with the post-treatment dummy. (We be aware that the estimates produced by the primary two specs could also be problematic if there are secular tendencies in high quality and they’re used along with linear regression reasonably than FRI, however we embody them for completeness and since they’re closest to the analyses we pre-registered; excluding them doesn’t qualitatively change our conclusions.)
4. Tweet kind
The evaluation can embody all tweets, or can focus solely on circumstances during which the consumer retweets the tweet containing the hyperlink with out including any remark. The former method is extra inclusive, however might comprise circumstances during which the consumer isn’t endorsing the shared hyperlink (for instance, somebody debunking an incorrect story should still hyperlink to the unique story). Thus, the latter case would possibly extra clearly determine tweets which might be uncritically sharing the hyperlink in query. More importantly, retweeting with out remark (low-engagement sharing) exemplifies the sort of quick, low-attention motion that’s our focus (during which we argue that individuals share misinformation regardless of a need to solely share correct info—as a result of the attentional highlight is targeted on different content material dimensions). Primary tweets are way more deliberate actions, ones during which it’s extra possible that the consumer did take into account their motion earlier than posting (and thus the place our accuracy nudge can be anticipated to be ineffective).
5. Article kind
The evaluation can embody all hyperlinks, or can exclude (as a lot as attainable) hyperlinks to opinion articles. Although the hyperpartisan and pretend information websites in our listing don’t sometimes demarcate opinion items, practically all the mainstream websites embody ‘opinion’ within the URL of opinion items. Thus, for our analyses that reduce opinion articles, we exclude the three.5% of hyperlinks (6.8% of hyperlinks to mainstream sources) that contained ‘/opinion/’ or ‘/opinions/’ within the URL.
6. Approach to randomization failure
As described above, owing to points with the Twitter API on day Three of wave 2, there was a partial randomization failure on that day (lots of the customers assigned to remedy didn’t obtain a direct message). We take into account two alternative ways of coping with this randomization failure. In the intent-to-treat method, we embody all customers from the randomization-failure day (with the post-treatment dummy taking up the worth 1 for all customers who have been assigned to be despatched a direct message on that day, no matter whether or not they truly obtained a direct message). In the exclusion method, we as a substitute drop all knowledge from that day.
In the primary textual content, we current the outcomes of the specification during which we analyse retweets with out remark, embody hyperlinks to each opinion and non-opinion articles, embody wave mounted results, calculate P values utilizing FRI, and exclude knowledge from the day on which a technical subject led to a randomization failure. Extended Data Table 4 presents the outcomes of all specs.
The major checks of results of the remedy evaluate variations in tweet high quality for all eligible consumer–days. However, this contains many consumer–days for which there are not any tweets to rated websites, which can happen, for instance, as a result of that consumer doesn’t even go surfing to Twitter on that day. To quantify impact sizes on a extra related subpopulation, we make use of the principal stratification framework whereby every unit belongs to certainly one of 4 latent kind29,30: never-taker consumer–days (which might not have any rated tweets in both remedy or management), always-taker consumer–days (consumer–days the place the consumer tweets rated hyperlinks that day in each remedy and management), complier consumer–days (during which the remedy causes tweeting of rated hyperlinks that day, which might not have occurred in any other case), and defier consumer–days (during which remedy prevents tweeting of rated hyperlinks). Because the estimated remedy results on whether or not a consumer tweets on a given day are largely optimistic (though not statistically vital; see Supplementary Table 9), we assume the absence of defier consumer–days. Under this assumption, we can estimate the fraction of consumer–days that aren’t never-taker consumer–days (that’s, are complier or always-taker consumer–days). This is then the one inhabitants on which remedy results on rated tweet high quality can happen, because the never-taker consumer–days are by definition unaffected by remedy with respect to rated tweets. We can then estimate remedy results on high quality and discernment on this presumably affected subpopulation by rescaling the estimates for the total inhabitants by dividing by the estimated fraction of non-never-taker consumer–days. These estimates are then bigger in magnitude as a result of they account for the dilution due to the presence of models that aren’t affected by remedy as a result of they don’t produce tweets whether or not in remedy or management.
Moreover, it is crucial to keep in mind that our estimates of the impact measurement for our refined, one-off remedy are conservative. Although our intent-to-treat method essentially assumes that the message was seen instantly—and thus counts all tweets within the 24 h after the message was despatched as ‘treated’—we can not reliably inform when (or even when) any given consumer noticed our message. Thus, it’s possible that lots of the tweets we’re counting as post-treatment weren’t truly handled, and that we’re underestimating the true remedy impact consequently.
Reporting abstract
Further info on analysis design is accessible within the Nature Research Reporting Summary linked to this paper.
