Restructuring Comment Sections

Research

Download Full Report

← Journalism

Researchers

Natalie (Talia) Jomini Stroud

Director, Knight Faculty Fellow

Joshua Scacco

Joshua Scacco

Comment sections are prevalent on national and local news websites in the United States.1 Most comment sections are designed as one column of responses organized in a vertical, or cascading, manner. Yet several news organizations have been experimenting with new formats (see, for example, The New York Times and The Washington Post).

In our study, we compared a one-column comment section to a three-column section that organized comments by whether they favored, opposed, or had questions/other comments about the legalization of marijuana. We chose marijuana legalization as it was a topic covered frequently in the news media during the time of the study.

Screen Shot 2020 03 30 at 11.41.57 AM

The Problem

Comment sections are a prominent part of news websites. Over 90% of national and local newspaper and television news websites have a comment section.1 According to a recent survey conducted by the Associated Press Media Editors, 82% of newspaper managers and editors reported that they were unlikely to ban online commenting.

Comment sections can be risky for newsrooms. Editors worry that uncivil comments can tarnish the brand of a news organization.2 Indeed, one recent study found that over four in ten comments left on a television outlet’s Facebook page were uncivil.3 Moreover, the comments left also can negatively affect perceptions of news content4 and can polarize people’s attitudes.5 These potentially troubling side effects of comment sections have led news outlets to eliminate their comment section or to try new strategies for improving visitor interactions.

One strategy for managing the content of comment sections might be to change their organization or structure. For example, The New York Times categorizes its comments by “All,” “Readers’ Picks,” and “NYT Picks” in a tab-column structure. Other outlets, such as Quartz, have experimented with annotation and commenting alongside or within the text of a news story. These practices may hold promise for improving the quality and quantity of comments on news outlets’ sites. How comment sections are formatted may influence how individuals interact with a website, as well as their attitudes towards news topics.

Design affects how people navigate a site. Eye-tracking research shows that individuals favor the upper left and upper half of a webpage when viewing a site.6 Viewing is affected by the complexity of the webpage structure. Simpler webpage structures lead individuals to follow similar scan paths on a page compared to more complex page structures.7 Although some research finds that individuals choose to scan horizontally across website columns as opposed to vertically, others have found that search engine results lead individuals to scan vertically and favor the first two hyperlinked results.8 Research on presenting political information has shown that grid-based formats help people learn and process information.9

Based on this research, as well as current practices on several media outlets’ websites, we designed a working three-column comment section and a one-column comment section. The three-column section included comments categorized by “favor,” “oppose,” and “questions/other comments about” marijuana legalization. The one-column section did not categorize comments. We proposed that the three-column comment section would encourage individuals to scan each comment column to gain more information.

One question when beginning this research was whether the comment section structure would affect polarization and tolerance for different political views. Recent work by the Pew Research Center illustrates the need for efforts to combat political polarization and intolerance. Individuals not only choose media outlets based on their political ideology, but also are exposed to different news information as a result.10 Because the three-column comment section clearly categorizes comments on all sides of the marijuana legalization issue, viewing these comments could provide individuals with a more balanced perspective on the policy issue as well as toward others who hold differing opinions. As the report shows, the three-column comment section made little difference in polarization, tolerance, and knowledge of arguments about different views compared to no comment section at all.

Key Findings

Results, gathered from 727 participants, showed several benefits to the three-column comment section. More thinking is needed, however, in figuring out how to best display the columns on a page.

  • The three-column comment section was rated more favorably than the one-column comment section.
  • Participants were more likely to leave a comment in the three-column than the one-column comment section.
  • On average, 13% of commenters left comments that were nonsensical or inappropriate (including placing the comment in the wrong column in the three-column comment section). The percentage was similar whether people saw the one- or three-column comment section.
  • The three-column comment section was less familiar to participants than the one-column section.
  • On average, participants spent one-and-a-half to two minutes with the comment There was no difference in time spent with the one-column versus three-column comment section.
  • The one- and three-column comment sections did not intensify polarization or affect attitudes about marijuana.
  • In the three-column comment section, the third column displayed comments opposing marijuana. There are some indications that this was consequential. For instance, those viewing the three-column comment section pressed “Like” less frequently for comments opposing marijuana legalization than those seeing the one-column comment section.

Implications for Newsrooms

The structure of a news website’s comment section matters for how individuals interact with the content. This study shows that a three-column comment section can lead more individuals to comment on a news website. Further, the three-column comment section does not lead to more polarized opinions on marijuana legalization compared to the one-column comment section.

The findings also signal the need for additional thinking about how to structure the comment section. For instance, there were some indications that the participants interacted with the right-most “Oppose Marijuana Legislation” column less frequently in the three-column comment section. Figuring out a way to ensure that comments in the right-most column are given equal attention is an important next step.

With these caveats, the findings indicate that news outlets should pay close attention to how their comment sections are structured. Switching to a three-column format, for instance, could increase the percentage of site visitors commenting.

The Study

Three-Column Preferred to One-Column Comment Section

Study participants rated the comment section that they saw on four dimensions:

  • Enjoyable / Unenjoyable,
  • Informative / Uninformative,
  • Interesting / Uninteresting, and
  • Clear / Unclear

The ratings ranged from 1 to 5, with higher values indicating a more positive response. As these four measures were highly correlated, we calculated an average. We compared the average ratings from those viewing the three-column comment section to those viewing the one-column comment section.

On average, participants rated the three-column comment section more positively than the one-column comment section.11

We examined whether participants’ ratings of the comment sections varied based on their age or education. Regardless of participants’ age or education, the three-column comment section was rated more favorably than the one-column comment section.

Screen Shot 2020 03 30 at 11.52.26 AM

More Comments in Three-Column Than One-Column Comment Section

Study participants had the option of leaving a comment in the comment section. Participants were more likely to leave a comment in the three-column comment section than in the one-column comment section.

Approximately one in five participants who had the ability to comment in this study chose to do so. In the one-column comment section, 17% of respondents left a comment. In the three-column comment section, 26% of study participants did so.12

These results also held regardless of participants’ age or education level.

Screen Shot 2020 03 30 at 11.54.26 AM

Commenters Didn’t Always Use the Comment Section as Intended

We analyzed whether the comments left in the one- and three-column comment sections were irrelevant or inappropriate. Irrelevant comments included statements like “NA.”13

Inappropriate comments occurred in the three-column condition. Here, study participants had to choose from a drop-down box whether they:

  1. Supported Marijuana Legalization,
  2. Opposed Marijuana Legalization, or
  3. Had Questions or Other Comments about Marijuana Legalization

Inappropriate comments included comments left in the three-column comment section, but placed in the wrong column (e.g. a comment that clearly opposed the legalization of marijuana in the column favoring legalization).14

Thirteen percent of commenters left a nonsensical or inappropriate comment. There were no differences between the one-column and three-column comment sections.

Three-Column Comment Section Less Familiar than One-Column

After viewing either the one-column comment section or the three-column comment section, we asked study participants to indicate how familiar they were with the comment section that they saw on a four-point measure from 1 (very unfamiliar) to 4 (very familiar).

As expected, study participants were more familiar with the one-column comment section than the three-column comment section.15

Screen Shot 2020 03 30 at 11.57.53 AM

People Spend Similar Amount of Time with One- and Three-Column Comment Sections

One possible commercial benefit of comment sections is that they increase time on site. In our study, participants typically spent between one and a half to two minutes with the comment section accompanying the news article about marijuana legalization.16

The median amount of time spent with the three-column comment section (109.64 seconds) was slightly larger than the median amount of time spent with the one-column comment section (100.62 seconds), but the difference was not statistically significant.17

Little Evidence that Comments Affected Attitudes, Polarization

We investigated whether the comment section format would affect attitudes about marijuana. We found no evidence that the comment format affected participants’ attitudes. We looked at this in three different ways.

No differences in attitudes about marijuana.

First, we asked people to indicate how favorably or unfavorably they felt about legalizing marijuana in the United States. On average, participants displayed a slight preference for marijuana legalization (average of 6.8 on a 10 point scale). There were no differences in these attitudes whether respondents saw the one-column comment section, the three-column comment section, or no comment section at all.18

No differences in the extremity of attitudes about marijuana.

Second, we looked at how extreme people’s responses were – did they somewhat favor legalizing marijuana or did they extremely favor legalizing marijuana? The measure ranged from 0, indicating neutral attitudes, to 5, indicating very strong attitudes. The average attitude extremity among the study participants was 3.24. It did not matter whether people saw the one-column comment section or a three-column comment section, they had similarly extreme attitudes.19

No differences in tolerance for others with different views about marijuana.

Third, we analyzed what people thought about those holding views that differed from their own. The measure ranged from 1 to 5, with higher values indicating more tolerant attitudes. Participants in the study had an average tolerance of 2.47. A person’s openness to hearing and learning more from individuals with different attitudes about marijuana was similar regardless of whether people viewed a one-column comment section, a three-column comment section, or no comment section at all.20

One-Column Comment Section Affects Pressing “Like” Button

In the comment section, there was a “Like” button next to each comment. The number of “Likes” next to each comment left by previous site visitors was the same regardless of whether participants saw the one-column comment section or the three-column comment section.

Study participants were able to “Like” any of the comments in the comment section by pressing the button, which would increase the recorded number of “Likes” by one. In the case of Margaret’s comment, for instance, pressing the “Like” button would increase the number from 3 to 4. We recorded how many buttons each participant pressed. Across the 11 comments, respondents pressed “Like” an average of 1.55 times.

We analyzed how many buttons participants pressed depending on the type of comment. In the comment section, three comments posed questions, such as “Does smoking marijuana cause lung cancer?” As shown in the chart below, participants clicked an average of 0.31 buttons next to questions posed in the comments. It didn’t matter whether participants saw the one-column or three-column comment section, they pressed “Like” next to a similar number of comments containing questions.

Participants clicked an average of 0.67 “Like” buttons next to the four comments favoring the legalization of marijuana. Again, participants pressed a similar number of “Like” buttons next to comments favoring legalization regardless of whether they saw the one-column comment section or the three-column comment section.

Some differences did appear, however, in the number of “Like” buttons pressed next to the four comments opposing the legalization of marijuana. Those viewing the one-column comment section clicked “Like” an average of 0.68 times on comments opposing legalization. Those viewing the three-column comment section, however, clicked “Like” an average of 0.45 times on the same comments.21

Screen Shot 2020 03 30 at 12.25.10 PM

Although many reasons might explain why this occurred, one possibility is how the columns were ordered. In particular, the “Oppose Legalization” column was farthest from the left. We suspect that people were less likely to click on comments in this column because of how it was positioned on the page.

One-Column Comment Section Has Modest Effect on Understanding Different Views

After seeing the news article and the comment section (for those who were assigned to view it), we asked participants to list up to three arguments favoring marijuana legislation and up to three arguments opposing marijuana legislation. We asked, “Now, regardless of your own view on this issue, why do you think that someone would [oppose/favor] the legalization of marijuana?”

Most of the arguments that people left were sincere and reasonable arguments for why someone would favor or oppose marijuana legalization. In some instances, however, people wrote insincere arguments. For instance, when asked for a reason that someone would oppose marijuana legislation, one participant wrote: “Because they are misinformed.” We categorized this as an insincere argument because it demeans or undermines people holding that viewpoint. We went through all of the arguments that people listed and coded them as sincere or insincere.22 Overall, 19 percent were insincere.

For each person, we counted how many sincere arguments they mentioned favoring marijuana legalization and how many they named opposing marijuana legalization. We then analyzed whether the number of arguments left varied depending on whether people saw the one-column comment section, the three-column comment section, or no comment section at all. There were no differences among the conditions in the number of sincere arguments favoring the legalization of marijuana.23 Only one difference appeared when analyzing the number of sincere arguments opposing the legalization of marijuana: those in the one-column comment section listed more arguments opposing the legalization of marijuana than those who didn’t see a comment section.24

One possible reason for this finding is that the comments opposing marijuana legislation received less attention in the three-column comment section because of their placement on the page. The oppositional comments in the one-column comment section, however, were seen by more people and led people to write more sincere arguments opposing marijuana legalization. More research is needed to understand how the ordering of the columns may affect reactions.

Screen Shot 2020 03 30 at 12.33.56 PM

Methodology

We first pre-tested a series of marijuana policy-focused comments. In total, 24 comments were pre-tested (10 pro-legalization, 8 anti-legalization, 6 containing questions). Survey respondents were recruited via Amazon’s Mechanical Turk (mTurk), an online platform where individuals participate in tasks for a small fee. Ninety-nine individuals participated in the pre-test. Participants read an article adapted from a major national publication about the federal government’s enforcement of marijuana laws in Washington and Colorado, two states that have legalized recreational use of marijuana. After answering a series of questions about the article, participants were randomly assigned to read six comments about marijuana policy. These comments were culled from various online news articles about recent developments related to marijuana legalization. Respondents rated the extent to which the comment favored or opposed marijuana legalization and “makes a [strong/weak] argument” on a scale from 1 to 5. Each participant then answered “Do you favor or oppose the legalization of marijuana by the federal government in Washington, D.C.?” and “Do you favor or oppose individual states enacting their own laws when it comes to the legalization of marijuana?” from strongly oppose (1) to strongly support (5).

To narrow the list of comments from 24 to a manageable amount, we chose comments clearly favoring (range: 1 to 1.5 on average for the question about whether the comment favored or opposed marijuana legalization), clearly opposing (range: 4.5 to 5), and questioning (approximately 3) marijuana legalization. Then, the marginal means for whether each comment had a “strong argument” were estimated controlling for an individual’s viewpoint. Comments with marginal means closest to 1 (strong argument) were chosen to display in the comment sections that we created for our study (see chart).

Following the pre-test, we conducted a between-subjects experiment to test the effect of comment section structure on policy polarization, number of comments left on site, time on site, and the listing of arguments for and against marijuana legalization. Survey respondents were recruited via an online survey vendor, Survey Sampling International (SSI), which administered the experiment to a nationwide sample of 727 individuals. Although not randomly selected, the sample demographically matched the demographic information of Internet users according to the Pew Research Center’s most recent findings. Participants had to be U.S. residents who were at least 18-years-old.25

After entering the survey, participants answered a series of questions about their attitudes toward marijuana legalization as well as their use of online media. Each participant then read an article adapted from The Washington Post by Niraj Chokshi about the federal government’s enforcement of marijuana laws in Washington and Colorado, two states that have legalized recreational use of marijuana. A series of follow-up questions examined the retention of information from the article.26

Afterward, participants were randomly assigned to one of three conditions: the three-column comment section, the one-column comment section, or no comment section. In the three-column comment section, each column featured a series of comments related to marijuana legalization that were categorized based on “Support,” “Oppose,” or “Questions/Other Comments about” legalization. All comments included a working “Like” button with an associated number that would increase upon being clicked. Participants could leave a comment of their own and categorize it into one of the columns as well as interact with the comments by pressing the “Like” buttons. The one-column comment section was designed to mirror the structure of those currently used on news websites. The comment section included the same comments and interactive features as the three-column comment section except the comments were not categorized and individuals could not categorize comments they wrote. The final condition contained no comment section and served as the experimental control.

Participants viewing the comment section then answered follow-up questions about how enjoyable, interesting, informative, and clear the comment section was to them.

We examined the extent to which individuals may have become more tolerant toward others who did not share their beliefs by asking about interest, openness, willingness, and friendliness toward those who do not share their viewpoint on marijuana legalization.

To gauge whether seeing both pro- and against-arguments in the comment section influenced marijuana policy arguments, we asked respondents to list up to three reasons why an individual would support marijuana legalization and three reasons why someone would oppose legalization. Each participant then rated how strong or weak each argument was on a scale from one (weak) to seven (strong). We also included a series of six political knowledge questions as well as political and demographic questions at the conclusion of the survey.

Demographics for our sample were obtained from the Pew Research Center’s Internet & American Life Project August 2013 tracking survey. The Pew Research data asked a combined race/ethnicity question where we asked two separate questions. We recalculated the racial composition of the Pew data for non-Hispanic identifiers to compare with our data. Manipulation checks confirmed no significant differences on these demographic factors across experimental conditions, meaning that randomization was successful.

 

SUGGESTED CITATION:

Stroud, Natalie Jomini and Scacco, Joshua. (2015, January). Restructuring Comment Sections. Center for Media Engagement. https://mediaengagement.org/research/restructuring-comment-sections/

  1. Stroud, N. J., Scacco, J. M., & Curry, A. L. (2014, August). Differences among news websites in their use of interactive features. Paper presented at the Association for Education in Journalism and Mass Communication, Montreal, Canada. [↩][↩]
  2. Hermida, A., & Thurman, N. (2008). A clash of cultures: The integration of user-generated content within professional journalistic frameworks at British newspaper websites. Journalism Practice, 2(3), 343-356; Singer, J. B. (2006). Stepping back from the gate: Online newspaper editors and the co-production of content in campaign 2004. Journalism & Mass Communication Quarterly, 83(2), 265-280. [↩]
  3. Stroud, N. J., Scacco, J. M., Muddiman, A., & Curry, A. L. (2014). Changing deliberative norms on news organizations’ Facebook sites. Journal of Computer-Mediated Communication. doi: 10.1111/jcc4.12104 [↩]
  4. Lee, E.-J. (2012). That’s not the way it is: How user-generated comments on the news affect perceived media bias. Journal of Computer-Mediated Communication, 18(1), 32–45. doi:10.1111/j.1083-6101.2012.01597.x [↩]
  5. Anderson, A. A., Brossard, D., Scheufele, D. A., Xenos, M. A., & Ladwig, P. (2014). The “nasty effect:” Online incivility and risk perceptions of emerging technologies. Journal of Computer-Mediated Communication, 19, 373-387. doi:10.1111/jcc4.12009 [↩]
  6. Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., & Wichansky, A. M. (2002). Eye tracking in web search tasks: Design implications. In Proceedings of the 2002 Symposium on Eye Tracking Research & Applications (pp. 51-58). ACM. [↩]
  7. Pan, B., Hembrooke, H. A., Gay, G. K., Granka, L. A., Feusner, M. K., & Newman, J. K. (2004). The determinants of web page viewing behavior: An eye-tracking study. In the Proceedings of the 2004 Symposium on Eye Tracking Research & Applications (pp. 147-154). ACM. [↩]
  8. Cutrell, E., & Guan, Z. (2007). What are you looking for? An eye-tracking study of information usage in web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 407-416). ACM; Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., & Wichansky, A. M. (2002). Eye tracking in web search tasks: Design implications. In the Proceedings of the 2004 Symposium on Eye Tracking Research & Applications (pp. 51-58). ACM; Granka, L. A., Joachims, T., & Gay, G. (2004, July). Eye-tracking analysis of user behavior in WWW search. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 478-479). ACM. [↩]
  9. See, for instance, Huang, L., & Price, V. (2001). Motivations, goals, information search, and memory about political candidates. Political Psychology, 22, 665-692. [↩]
  10. Mitchell, A., Gottfried, J., Kiley, J., & Matsa, K. E. (2014, October 21). Political polarization & media habits: From Fox News to Facebook, how liberals and conservatives keep up with politics. Pew Research Center. Retrieved from https://www.journalism.org/2014/10/21/political-polarization-media-habits/ [↩]
  11. In a regression model controlling for demographic (sex, race/ethnicity, age, education, and income) and political factors (frequency of monitoring the news, using the Internet for politics, attitudes about marijuana, general political knowledge, partisanship, and ideology), being in the three-column condition predicts more positive ratings of the comment section relative to the one-column condition (B = 0.20, SE = 0.09, p < 0.01). The figure shows the average ratings of the comment section, holding all other values constant at their mean or modal value. [↩]
  12. In a logistic regression model controlling for the same variables mentioned in footnote 2, the coefficient for the three-column comment section is significant (B = .60, SE = .25, p < 0.05). Converting to probabilities and holding all controls at their mean or modal value, the probability of commenting in the one-column condition was 11.5% and the probability of commenting in the three-column condition was 19.1%. [↩]
  13. To evaluate our coding, we used Krippendorff’s alpha, which is a measure of inter-coder reliability that adjusts for chance agreement. The measure runs from 0 to 1, with values greater than 0.8 considered reliable. For the irrelevant comment measure, the reliability was 0.88. [↩]
  14. Here, we coded whether comments expressed a clear opinion or asked a clear question related to the comment heading that was selected in the three-column condition. Seventy percent of comments met this definition. Using Krippendorff’s alpha, the reliability was 0.75. We reviewed the remaining 30 percent (n=19) to determine whether they (a) could be appropriate for the column, even though a clear sentiment or opinion wasn’t expressed, (b) were definitely in the wrong column, or (c) were irrelevant. We determined that three of the comments were included in the wrong column. [↩]
  15. A t-test indicates significant differences in how familiar respondents were with the one-column and three-column comment section (t(466)=4.68, p<0.01). After controlling for the demographic and political factors mentioned in footnote 2, the effect of the three-column comment section is significant in an OLS regression analysis (B=-0.35, SE=0.08, p<0.01). [↩]
  16. The median time with the comment section was 104.88 seconds. [↩]
  17. The nonparametric Mann-Whitney test was not significant (Mann-Whitney U = 25,386m p = 0.11). [↩]
  18. In a regression analysis controlling for the same variables mentioned in footnote 2, the coefficients for the one-column comment section (B = .29, SE = .21, n.s.) and the three-column comment section (B = -0.22 SE = 0.20, n.s.) were not significant (the control condition was excluded). [↩]
  19. We used a folded measure of marijuana attitudes where those answering a 5 were considered least extreme and those answering either a 0 or a 10 were considered most extreme. In a regression analysis controlling for the same variables mentioned in footnote 2, the coefficient for the one-column comment section was marginally significant (B = – 0.29, SE = 0.17, p=0.08) and the three-column comment section (B = -0.17 SE = 0.16, n.s.) was not significant (the control condition was excluded). The coefficient for the one-column comment section indicates less, rather than more, extreme attitudes. Although the one-column comment section was marginally significant relative to the control condition, there were no differences between the one-column and three-column comment sections. [↩]
  20. We measured tolerance toward the out-party using four questions: (1) I would be interested in hearing more from those who favor/oppose legalizing marijuana, (2) I would be open to learning more about why those who favor/oppose legalizing marijuana believe what they do, (3) I would be willing to listen to those who strongly favor/oppose legalizing marijuana in the media, and (4) I could be friends with someone who strongly favors/opposes legalizing marijuana. Those opposing legalizing marijuana saw questions about those favoring marijuana (and vice versa for those favoring the legalization of marijuana). Those with neutral opinions on legalizing marijuana were randomly assigned to see either questions about those favoring legalization or those opposing legalization. Each item’s response options ranged from 1 to 5. The items were reliable (Cronbach’s alpha = 0.85) and were averaged to create a measure of tolerance (M=2.47, SD=0.92). In a regression analysis controlling for the same variables mentioned in footnote 2, the coefficients for the one-column comment section (B = -.08, SE = .08, n.s.) and the three-column comment section (B = -0.03 SE = 0.08, n.s.) were not significant (the control condition was the reference category). [↩]
  21. A t-test indicates significant differences between the one-column and three-column conditions in the number of comments opposing marijuana legalization that were “Liked” by the respondent (t(451)=2.14, p<0.05). After controlling for the demographic and political factors mentioned in footnote 2, the effect of the three-column comment section is only marginally significant in both an OLS regression analysis (B=-0.20, SE=0.10, p=0.06) and a negative binomial analysis (B=-0.35, SE=0.21, p=0.09). [↩]
  22. Three coders coded the sincerity of 20% of the arguments. To evaluate how much they agreed, we used Krippendorff’s alpha, which is a measure of inter-coder reliability that adjusts for chance agreement. The measure runs from 0 to 1, with values greater than 0.8 considered reliable. For the sincerity measure, the reliability was 0.83. [↩]
  23. An ANOVA for the number of sincere arguments favoring marijuana legislation was not significant (F(2, 683)=.03, n.s.). Similarly, a regression controlling for the demographic and political variables described in footnote 2 did not yield any significant effects of the condition (three-column B=.04, SE=.07, n.s.; one-column B=.09, SE=.07, n.s.). [↩]
  24. An ANOVA for the number of sincere arguments opposing marijuana legislation was significant (F(2, 683)=3.50, p<0.05). Similarly, a regression controlling for the demographic and political variables described in footnote 2 yielded a significant effect of condition (three-column B=.12, SE=.09, n.s.; one-column B=.25, SE=.97, p<.01). The results held using a dichotomized version of number of arguments (either 3 arguments or less than 3) and logistic regression. [↩]
  25. Complete data were gathered from 898 participants. We excluded data from those who participated in the study more than once (1%), who reported that they could not see the article (8%), who reported that they did not see the comment section (3%), and who incorrectly responded to a question assessing whether respondents were paying attention (7%). [↩]
  26. On average, people correctly answered 2.13 of the three questions correctly (SD=0.88). We analyzed our data looking only at those who answered two or three of the items correctly. In a few instances, a statistically significant result became marginally significant; this is attributable at least in part to the lower sample size. In general, the results were similar. [↩]