News Comments: What Happens When They’re Gone or When Newsrooms Switch Platforms

Research

Download Full Report

← Journalism

Researchers

Natalie (Talia) Jomini Stroud

Director, Knight Faculty Fellow

Caroline Murray

Caroline Murray

Yujin Kim

Yujin Kim

 

SUMMARY

In a first-of-its-kind study, the Center for Media Engagement put comment sections to the test. In partnership with 24 Gannett-owned newsrooms and Coral by Vox Media, we assigned newsrooms to:

  • Turn off their comments,
  • Continue using the Facebook commenting system on their site,
  • Switch to Coral’s commenting system, which requires registration, or
  • Switch to Coral’s commenting system and only allow subscribers to comment.

The study revealed that turning off comments reduced the average time users spent on the site compared to sites that continued using Facebook commenting. Although most people did not notice that the comments were turned off, many of those who did felt that the change made the experience worse.

For newsrooms that retained comments, comments were generally less toxic on sites that switched to Coral’s commenting system. Commenters at sites that switched to Coral, however, reported feeling more disconnected from other commenters, possibly due to people having new screen names. For newsrooms that switched to Coral for Registrants, engagement with the comment section increased.

ACKNOWLEDGMENTS

We are grateful to our funders, the John S. and James L. Knight Foundation, Democracy Fund, and the William and Flora Hewlett Foundation, with special thanks to Ashley Zohn of the Knight Foundation and Olivier Kamanda, formerly of the Knight Foundation. We deeply appreciate the tireless work of Anjanette Delgado, Tom Davidson, Pallavi Razdan, Brian Smith, and Kate Walters from Gannett in making this project a reality. Josh Awtry of Gannett and Daniel Carp, formerly of Gannett, also played pivotal roles in this project, as did Keira Nothaft of Gannett, who first contacted the Center for Media Engagement asking for advice on what to do with the comment section. Her question and advocacy set the project in motion. Thanks also to Andrew Losowsky and Kim Gardner from Coral by Vox Media. We also thank Jordan Carney for his help with accessing the comments.

THE PROBLEM

Much has been written about the value, or lack thereof, of news comment sections. These spaces, where people can leave comments in response to new stories, have been removed by some news organizations, whereas other newsrooms have invested in them heavily. Yet the contribution of news comments has not been put to rigorous testing. This study aimed to change that. We worked with 24 Gannett-owned newsrooms and Coral by Vox Media to engage in the most ambitious test to-date of what comment sections mean to news sites. For a month and a half, newsrooms were randomly assigned one of four commenting options:

  • Comments Removed: Sites removed all comments,
  • Facebook Comments: Sites continued using the Facebook commenting system,
  • Coral for Registrants: Sites switched to Coral’s commenting system, which required registration, or
  • Coral for Subscribers: Sites switched to Coral’s commenting system and only allowed subscribers to comment

We monitored the content of the comments, site traffic analytics, audience perceptions, and newsroom reactions to these changes.

KEY FINDINGS

Turning Off the Comments

  • Average time spent on the site was lower when the comments were removed compared to when Facebook comments were retained.
  • Among those who had commented on a news site before, around three-fourths did not notice that comments had been removed. Many of those who did notice said that it made the experience worse.
  • The number of comments on a newsroom’s Facebook page, rather than on their website, did not change for sites that turned off comments compared to sites that retained Facebook commenting.

Switching Commenting Platforms

  • On average, comments were less toxic on sites using Coral, although this wasn’t the case for every site involved in the study.
  • Journalists at newsrooms that switched to Coral thought that the comments were slightly more civil than journalists at newsrooms that retained Facebook commenting.
  • Audiences tended not to notice changes to the comment section, but were more likely to do so when the comments switched to Coral for Registrants, particularly at the Detroit Free Press.
  • Engaging with comments increased for people using sites that switched to Coral for Registrants.
  • Commenters at sites that switched to Coral reported feeling more disconnected to the other commenters compared to commenters at sites that retained Facebook comments, possibly due to users having new screen names.
  • Journalists at sites that switched to Coral for Registrants rated the comments as more valuable after the switch than when the site had Facebook comments. Journalists at sites that stayed with Facebook comments or switched to Coral for Subscribers rated the comments as similarly valuable before and after the intervention.

THE EXPERIMENT

In December of 2019, 24 Gannett-owned newsrooms’ comment sections were randomly assigned, to one of four different conditions: (1) comments removed, (2) Facebook comments retained, (3) switched to Coral for Registrants, or (4) switched to Coral for Subscribers.1 We first describe what happened when the comments were turned off and then share what happened when newsrooms switched to a new commenting system.

Turning off the comments

In recent years, several news organizations have turned off their comment sections to much fanfare. In July of 2020, for instance, Yahoo began including the following note on articles, “Our goal is to create a safe and engaging place for users to connect over interests and passions. In order to improve our community experience, we are temporarily suspending article commenting.” But what effect does turning off comments have for newsrooms and users?

To find the answer, we analyzed Adobe Analytics web traffic metrics from Gannett and started with commenter behavior. Commenters are engaged site visitors. Across all of the sites before the study began, the average daily page views per visit for commenters was 4.3 and for everyone was 3.0. The average daily time on site in minutes, again averaged across all sites, was 16.8 for commenters and 7.4 for everyone.

We did not detect a meaningful change in traffic metrics like page views depending on whether the site turned off comments or retained Facebook comments. We did, however, see a statistically meaningful effect on the average time people spent on the site: Visitors to sites that removed comments spent less time on the site than visitors to sites that retained the Facebook commenting system.2

Average time on site declined for all sites participating in this study. Gannett attributes the decline to users shifting to mobile platforms, including AMP and Native apps, which were not included in this study. The decline was more pronounced, however, for sites that turned off the comments than for sites that retained the comments. For the sites that retained Facebook comments, average time on site declined from 7.59 to 6.87 minutes per day, a decrease of 0.72 minutes. For sites that removed comments, average time on site declined from 7.11 to 6.04 minutes per day, a decrease of 1.08 minutes.

The following chart shows changes in the average time on site across the study period. Before the intervention began, sites that would have the comments removed tended to have higher, or similar, average time on site compared to sites that would retain Facebook commenting. After the intervention, however, sites with the comments removed had a lower average time on site compared to sites with Facebook commenting.screen shot 2020 10 20 at 11.52.42 am

One theory about eliminating comment sections on a news site is that it will drive people to social media. Using CrowdTangle, a public insights tool owned and operated by Facebook, we analyzed the total number of comments left on the news organizations’ Facebook pages each day. There was no evidence that people commented more frequently on the news organization’s social media page when comments were turned off on the website. The number of comments left each day on the sites’ Facebook pages after the intervention was similar whether the actual news website continued to use the Facebook commenting tool or had no commenting available.3

In our survey, most people didn’t notice that the comments had been removed. Among respondents from sites that turned off the comments, only 10% correctly reported that the comments had been removed. Looking only at those who had commented on the news site at least once before our study began, only 24% reported noticing the elimination of the comments. This may be because of the layout of the comments on Gannett sites, which required people to click in order to see comments, as shown below.

screen shot 2020 10 20 at 11.53.40 am

We asked those that noticed the change to tell us whether they thought it made their experience better, worse, or had no effect. Former commenters were more likely to think that it made the experience worse than to think it made the experience better.screen shot 2020 10 20 at 11.54.39 am

We also analyzed journalists’ responses to the elimination of the comment section. Journalists’ job satisfaction and thoughts about how the newsroom served the community did not vary depending on whether comments were eliminated or Facebook comments were maintained.

Switching Platforms

One reason that news sites turn off comments is due to concerns about incivility. Previous research shows that comment sections containing incivility can negatively affect attitudes about the journalism and the news organization, and can polarize people’s attitudes about the topic featured in an article. Turning off the comments, however, is only one possible strategy for dealing with incivility. Other options, such as having journalists engage in the comment section, can affect the tone of the comments. Here, we evaluated whether the platform itself can influence commenting perceptions and behaviors. In particular, we analyzed what happened when people switched from Facebook commenting to Coral’s commenting platform. We also looked at two different implementations of Coral: one for anyone who registered and a second only for subscribers. We wondered whether limiting the commenting section to subscribers would have a different effect than keeping comments open to everyone.

We first examined the comments using a toxicity algorithm from Google, where Google defines toxicity as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion.” The algorithm returns a probability that the comment is toxic from 0, meaning that there is no chance that the comment is toxic according to the algorithm, to 1, meaning that the comment is toxic according to the algorithm.4

Before the intervention (February 19 to November 8, 2019), when all 24 sites used Facebook commenting, the average toxicity was not particularly high. The average pre-intervention toxicity was 0.28, and varied between 0.24 to 0.33 across the sites.

To measure the effect of the intervention, we subtracted the average toxicity of the comments before the intervention from the average toxicity of the comments after the intervention (December 3, 2019 to January 23, 2020). Positive numbers indicate that the comments became more toxic, whereas negative numbers indicate that the comments became less toxic.

A few results stand out. First, the comments became less toxic on most of the news sites, regardless of what change was made to the comment section. This may be because the intervention took place over the winter holidays and, in general, people may write less toxic comments during this time of year.

Second, the changes in toxicity were small. The largest change from before to after the intervention was at the Ventura County Star, where comments became, on average, 0.13 less toxic.

Third, and most important for this study, the commenting platform that was used did affect the level of toxicity. Overall, sites that adopted Coral had less toxic comments than sites that retained Facebook comments. Averaging by site, sites that switched to Coral for Registrants had 0.05 less toxic comments, sites that switched to Coral for Subscribers had 0.04 less toxic comments, and sites that remained with Facebook had 0.004 less toxic comments. This is not to say, however, that all sites that switched to Coral had less toxic comments (e.g., the Detroit Free Press had slightly more toxic comments). Rather, the aggregate effect across sites is that Coral resulted in fewer toxic comments than Facebook.5

screen shot 2020 10 21 at 10.30.38 am

This reduction in toxicity was noticed by journalists at participating news organizations. Journalists at the sites that switched to Coral for Subscribers or Coral for Registrants perceived that the comments were slightly more civil than journalists at sites that retained Facebook comments.screen shot 2020 10 21 at 10.31.56 am

We analyzed whether people noticed the changes to the site. As occurred when the comments were removed, audience members tended to report that they did not notice any changes. Those at sites that switched to Coral for Registrants were more likely to notice changes compared to those at sites that retained Facebook commenting or that switched to Coral for Subscribers. Overall, however, fewer than three in ten noticed any changes.screen shot 2020 10 21 at 10.32.33 am

Those at sites that switched to Coral for Registrants were more likely to notice that a new commenting system was put into place. Those at sites that switched to Coral for Subscribers, however, were not more likely to notice the new commenting system, nor to report that the comments were available only to subscribers.screen shot 2020 10 21 at 10.33.34 am

We also considered whether noticing the changes was driven by a few of the participating sites more than others.

Those at sites that retained Facebook comments – where no change actually occurred – tended to perceive that there was not any change. The exception was Asbury Park Press, where over 30% of respondents reported noticing a change. This is likely because Asbury Park Press also conducted a trial of Coral’s Ask product during a similar time period (see endnote 1).

Among the sites that switched to Coral for Subscribers, North Jersey audiences were most likely to notice a change. Among those that switched to Coral for Registrants, those at the Detroit Free Press were most likely to notice a change.screen shot 2020 10 21 at 10.34.08 am

The commenting platform chosen also affected reported commenting behavior. More people reported commenting when the sites switched to Coral for Registrants than when sites switched to Coral for Subscribers or retained Facebook. This pattern appeared both for those who had commented prior to the intervention and those who had not.screen shot 2020 10 21 at 10.35.40 am

Commenters at sites that switched to Coral for Registrants also were more likely to report that articles regularly received more than 20 comments compared to those at sites that switched to Coral for Subscribers or retained Facebook comments.screen shot 2020 10 21 at 10.36.26 am

Although commenters reported commenting more, and reported noticing more comments, we wanted to find out whether there was evidence in the site traffic data that people accessed the comments more frequently on sites that switched to Coral for Registrants. On the Gannett sites, people had to click a button in order to view the comments. This allowed us to examine whether opening the comment section varied by newsroom. The results were supportive: more people opened the comment section on sites that switched to Coral for Registrants compared to sites that retained Facebook comments.6screen shot 2020 10 21 at 10.38.08 am

We also analyzed whether the commenting platform affected subscription rates. There were no differences in the number of subscriptions depending on whether the site had Coral for Registrants, Coral for Subscribers, or Facebook comments. For most of the news organizations in this study, however, comments came from subscribers to the site’s digital (site + e-edition) or print edition. For the sites with paywalls, 94% of the comments came from subscribers. At the Detroit Free Press, the one site in our study that did not have a paywall in place during the intervention, only 6% of comments came from subscribers to the print or e-edition.

We asked respondents to tell us how connected they felt to their fellow commenters. Before the intervention, commenters felt similarly connected. After the intervention, those at sites that retained Facebook commenting felt more connected than those at sites that switched to Coral. This suggests that Coral may have reduced feelings of connection among commenters. It is possible that this is because people saw different screen names for their fellow commenters after the switch to Coral.screen shot 2020 10 21 at 10.39.12 am

In general, there were few differences in what journalists thought about the comments based on whether their newsroom switched platforms, turned off comments, or retained Facebook comments. Perceptions of commenters and job satisfaction, for instance, were similar regardless of the newsrooms’ commenting system or lack thereof. They did perceive changes in civility, as previously described. Journalists at the sites that switched to Coral for Registrants also rated the comments as more valuable after the intervention than they did when they had Facebook comments, although their average response hovers around “neither agree nor disagree” that the comments are valuable. Journalists at sites that stayed with Facebook comments or switched to Coral for Subscribers rated the comments as similarly valuable before and after the intervention.screen shot 2020 10 21 at 10.40.21 am

METHODOLOGY

Twenty-four news sites were randomly assigned to one of four conditions:

  1. Comments Removed: Sites removed all comments,
  2. Facebook Comments: Sites continued using the Facebook commenting system,
  3. Coral for Registrants: Sites switched to Coral’s commenting system, which required registration, or
  4. Coral for Subscribers: Sites switched to Coral’s commenting system and only allowed subscribers to comment

Block randomization was used based on the size of the site (large, medium, or small). The categorization was made based on the number of page views and in consultation with Gannett. On November 21, 2019, the commenting change went into effect for three of the sites (one site in each condition 2-4) as a soft launch. On December 3, 2020, all sites converted to their assigned condition.

Site Traffic

Gannett provided us with daily site analytics for unique visitors, registrations, orders, adjusted orders, average time spent, average page views per visit, return visits, and comment opens from January 1, 2019 through January 31, 2020.

Comment Quality

To evaluate the quality of the comments, we compiled comments from all sites with Facebook comments between February 19, 2019 and November 8, 2019. In total, we recorded 96,579 comments. To access these comments, we worked with Gannett to obtain

URLs of articles published between these dates and then accessed the corresponding Facebook comments.7

After the intervention began, we again compiled comments for those sites that retained Facebook commenting between December 3, 2019 and January 23, 2020 based on URLs provided by Gannett. Coral provided the comments for the sites that switched to the Coral platform.

We then used Google’s Perspective API to assess the toxicity of each comment. Toxicity is scored from 0 to 1, with 0 indicating a prediction of no toxicity and 1 indicating a prediction of a highly toxic comment. We also conducted a follow-up test using Amazon.com’s Mechanical Turk (see endnote 4).

Audience Survey

To analyze audience responses, surveys were sent to news subscribers and advertised by the partner newsrooms. The pre-wave was fielded between October 24, 2019 and December 2, 2019. In total, 8,029 responses were received. We re-contacted 6,067 people who left a valid email address on the survey. The post-wave survey was fielded between January 24, 2020 and February 11, 2020.

In total, we analyzed the responses of 2,206 people who shared their views before and after the intervention. Of those who completed the surveys, 62% are male and 91% are white. The respondents are an average of 65 years old (SD =11) and are highly educated (78% had a Bachelor’s Degree or more). Twenty-five percent of those who completed both waves of the survey had commented on the site and 55% read the comments at least a few times per month prior to the study.

Newsroom Survey

In collaboration with Gannett, we sent a survey to journalists at each of the newsrooms that participated in this study. We received 422 valid responses between October 16, 2019 and December 2, 2019 and 367 respondents provided an email address. In the post-wave, 127 responses were received between January 27, 2019 and February 19, 2019, excluding data from 10 additional individuals who switched newspapers between the pre- and post-wave of the survey.

 

SUGGESTED CITATION:
Stroud, Natalie Jomini, Caroline Murray, & Yujin Kim. (2020, October). News comments: What happens when they’re gone or when newsrooms switch platforms. Center for Media Engagement. https://mediaengagement.org/research/comment-changes.

  1. After randomization and the pre-wave surveys had gone out, we were notified that Asbury Park Press and Cincinnati Enquirer would also be participating in a trial of Coral’s Ask tool. Cincinnati was able to delay their trial, but Asbury was not. We then added one more site, the Milwaukee Journal Sentinel to the trial as a site comparable in size (both Asbury and Milwaukee were assigned to retain Facebook comments). The data were analyzed with Asbury or with Milwaukee separately. There were no differences in the findings. We therefore retain Asbury for the purposes of reporting since it was part of the original randomization. [↩]
  2. The data are the daily average time on site for each news organization from January 1, 2019 through January 31, 2020. We used time series analysis with seven lags of the dependent variable to take into account auto- correlation in the data and fixed effects for the news organization and day. We find that the daily average time on site differs for those sites that turned off comments beginning on the date that they did so compared to those sites retaining Facebook comments and the days prior to the comments being turned off on the other sites. This difference is statistically significant. [↩]
  3. The data are the daily number of comments left on the newsroom’s Facebook page from January 1, 2019 through January 23, 2020. We used time series analysis with seven lags of the dependent variable to take into account auto-correlation in the data and fixed effects for the news organization and day. There were no significant differences between sites that turned off the comments and sites that retained Facebook commenting. [↩]
  4. Note that Coral also makes use of this algorithm – comments that are above a toxicity threshold set by the newsroom first get a warning, and then if the commenter submits it anyway or writes something that is still above the threshold, it gets held back for moderator approval. We exclude rejected comments from Coral to be comparable with Facebook, where we do not have access to rejected comments. We note that it is possible that incivility still exists in the comments in ways that evade the algorithm. To validate our conclusions, we also took a sample of 100 comments before and 100 comments after the intervention for each site that retained comments. In three instances, there were less than 100 available, but the results are the same if we weight the data so that each newsroom has 100. In total, we analyzed 3,458 comments. We used mTurk and had 3 people who identified as Republicans and 3 people who identified as Democrats evaluate each comment. Each mTurker was able to evaluate at most 105 comments. mTurk workers were asked to report how toxic the comment was. The same conclusion, where the sites that switched to Coral had less toxic comments compared to the sites that retained Facebook, appeared. [↩]
  5. We examine the level of toxicity of the comments with fixed effects for the news organization and day. As the number of comments varies by newsroom, we also weighted the data so that each newsroom’s comments count for the same percentage of cases analyzed. Statistically significant results show that toxicity is lower among those sites using Coral compared to those retaining Facebook comments. [↩]
  6. The data are the daily number of comment section opens for each news organization from January 1, 2019 through January 31, 2020. We used time series analysis with seven lags of the dependent variable to take into account auto-correlation in the data and fixed effects for the news organization and day. We find that the daily number of comment section opens are significantly higher after sites that switched to Coral for Registrants compared to sites that retained Facebook and the sites prior to the intervention.[↩]
  7. Not all comments were obtained from the sites prior to the intervention because of lags in the sites loading. We went back and obtained all comments from ten sites where it was possible because Facebook comments were still available. Comparing the subset of comments we obtained to the full set for these ten sites, the correlation was .77. There did not appear to be a specific pattern. The average difference in the mean toxicity of the comments for each newsroom was 0.01 (Range = -0.04 to 0.07). [↩]