Comment sections on news websites spark much controversy. Do they provide a space for civil discussion or are they just hotbeds of incivility? For this study, we report on what more than 12,000 people1 told us about comment sections on 20 different news sites. Participating sites include news outlets from across the U.S. and several kinds of media (print, television, radio, and online-only). This project was funded by The Coral Project, a collaboration between The New York Times, The Washington Post, and Mozilla that is supported by The John S. and James L. Knight Foundation.
The categories of questions we sought to answer include:
- Do people want journalists to interact in news comment sections?
- Do people think that news comment sections are civil?
- Do commenters feel connected to other commenters?
- How frequently do people comment?
- Do people know about comment section features, such as the ability to flag offensive comments?
- On which types of stories do people comment?
- Why do people comment?
- How easy do people think that comment sections are to navigate?
- Are there differences across news organizations depending on their size?
Results show that sometimes perceptions differ across sites, but in other instances, perceptions are remarkably consistent. The following sites participated: Alaska Dispatch News, AL.com, The Arizona Republic, The Atlantic, Civil Beat, The Dallas Morning News, Deseret News, Fort Worth Star-Telegram, KXAN, PBS NewsHour, Philly.com, The Seattle Times, Southern California Public Radio/KPCC, The State Journal-Register, The Texas Tribune, TribLIVE, Twincities.com / St. Paul Pioneer Press, Voice of San Diego, The Washington Post, and Willamette Week.
Key Findings
Study findings include:
- Preferences for news comment sections are fairly uniform across news sites. For example, around three-quarters want journalists to clarify factual questions, and experts to respond to comments.
- Perceptions of the civility of news comment sections vary widely by site. Between 14% and 78% of respondents per site rate the comments on the site very or somewhat civil.
- Fewer than half of respondents on any news site feel connected to other commenters.
- 62% think that they can report offensive comments, although it is possible on all sites.
- For two large sites, more than 50% of respondents comment on the site weekly or more frequently. For all of the other sites, fewer than half of the commenters comment weekly or more often.
- Those commenting on larger sites are more likely to be commenting on U.S. politics/domestic policy and international news. Those commenting on smaller sites are more likely to be commenting on state government or their neighborhood/local community.
- For most sites, the most common reason that people comment is to express an emotion or opinion.
- Most commenters across all sites find comment sections easy to navigate. Yet just under half per site, on average, say that it is easy to sort the comments.
Implications for Newsrooms
This report examines perceptions of comment sections by surveying people at 20 different news sites. Some aspects of these sites, such as the ease of navigating the comment sections, feelings of connectedness, and preferences for the space, are relatively consistent across the 20 sites. Although some aspects could be improved, such as the ability to sort comments, survey respondents generally find comment sections on news sites easy to navigate. Across the sites, most commenters did not report feeling connected to their fellow commenters. Further, the desires to have experts engage and journalists respond to factual questions in the space are consistently high across respondents at all of the sites.
Other aspects, such as the types of stories on which people comment and their perceptions of civility, vary greatly across the 20 sites. Unsurprisingly, those commenting on smaller sites are more likely to comment on more geographically narrow topics, like state government and local politics. Larger sites attract more commenting on national and international topics.
What is more surprising, however, is the extent to which perceptions of civility vary across the sites. Some news sites are seen as civil spaces whereas others are decidedly not. As we did not do an analysis of the comments themselves, we do not know whether the sites actually have more or less civil comments. What we do know is that there are large differences in what people think about the civility of the comments across the sites.
Our efforts to understand why people see differences in civility turned up a few answers. On average, smaller sites are seen as more civil (although this isn’t always the case). Similarly, we also find that sites receiving fewer comments are seen as more civil. Those believing that the site allowed anonymity think that the comments are less civil. Yet those using a Facebook comment section – a less anonymous platform than several of the others – see the comments as similarly civil as those using other commenting platforms. This suggests that the perception of anonymity matters more than the extent to which the platform allows anonymity.
The results show that news staff are good at detecting the civility of the comment section; their reports of how civil the space is lined up well with the survey responses. There is less of a correspondence between the survey responses and staff reports on how connected commenters are to one another, however.
Finally, this survey demonstrates that commenters are not always aware of the features available in news comment sections. Thirty-eight percent of commenters did not know that they could report offensive comments, for example.
It is our hope that this report can offer some insight to newsrooms, and to those interested in designing comment sections. By knowing what remains constant and what differs across sites, developers can better understand what to tailor and what can be more uniform. Getting these insights directly from commenters provides a nuanced understanding of how the space works.
The Study
Preferences for News Comment Sections
We asked respondents whether journalists should clarify questions, actively contribute, or direct the conversation; whether experts should respond to comments; whether newsrooms should highlight quality comments; and whether people should be allowed to post anonymously.
In general, the responses were fairly consistent across news sites.
- Around three-quarters want journalists to clarify factual questions and experts to respond to comments.
- Just over half want journalists to actively contribute and just under half want newsrooms to highlight quality comments.
- Approximately a third want sites to allow anonymous posting and a similar percentage want journalists to direct the conversation.
Journalists Clarifying Factual Questions
On average, 81% of survey respondents at each news site would like it if journalists clarified factual questions in the comment section. Across the news sites, the percentage varies between 71% and 87%.
Experts in the Comment Section
Well over a majority of respondents show a preference for experts responding to comments in the comment section. On average, 73% of survey respondents at each news site would like it if experts on the topic of the article responded. This percentage varied between 61% and 82% across the news sites.
Journalists Actively Contributing to Comment Sections
Close to a majority of survey respondents at each news site, an average of 58%, would like it if journalists actively contributed to comment sections. Variance across the news sites is between 51% and 68%.
There are statistically meaningful and notable differences across commenters, comment readers, and those who neither comment nor read comments in their preferences for journalists actively contributing to news comment sections. Sixty-one percent of commenters would like journalists to actively contribute compared to 56% of those who read comments and 50% of those who do neither.
Highlighting Strong Comments
Fewer than half of respondents at each news site, 42% on average, would like it if journalists highlighted quality comments in comment sections. Respondents at only one site exceeded 50% in wanting news organizations to highlight quality comments. At all other outlets, the percentage agreeing strongly or somewhat fell between 34% and 48%.
Anonymous Commenting
A third, 33% of survey respondents at each news site on average, would like it if journalists allowed people to comment anonymously. This percentage varies between 20% and 49% across sites.
There are statistically meaningful and notable differences across commenters, comment readers, and those who do neither in their preferences for anonymity. Thirty-six percent of commenters would like comment sections to allow anonymity compared to 27% of those who read comments and 25% of those who neither read comments nor comment.
We also examined whether preferences for anonymity varied based on the platform used for the comment section. Those commenting on a site using Facebook were more supportive of anonymous commenting compared to those using other platforms. Forty percent of those commenting on a site using Facebook agreed strongly or somewhat that they would like it if news comment sections allowed anonymity. This is compared to 30% of those using Disqus, 29% of those using Livefyre, and 29% of those using Civil Comments.2
Journalists Directing the Conversation
On average, 33% of survey respondents at each news site would like it if journalists directed the conversation in comment sections. Across the news sites, the percentage varies between 21% and 44%.
Predicting Preferences
We analyzed what predicts holding beliefs about journalists and experts getting involved in news comment sections. The one factor that consistently matters is how much survey respondents trust in the news organization.3 The more people trust a news organization, the more they want journalists to clarify questions, actively contribute, and direct the conversation. The higher the trust, the more people want experts to respond to comments and newsrooms to highlight quality comments.
In only one instance does the opposite relationship appear: the more people trust a news outlet, the less they want to comment anonymously.
Perceptions of Civility Across News Sites
We asked respondents to tell us how civil, or uncivil, they find the comments. There is a great deal of variability in how commenters rate the civility of each site. On average, 44% of commenters per news site say that the comment section is “somewhat civil” or “very civil.” This ranges from 14% to 78%.
There’s a modest relationship between site size and perceptions of civility. In general, smaller sites are rated as more civil. This relationship is not absolute, however. As shown in the chart below, some small sites are seen as equally uncivil as some medium and large sites.
Civility by Commenting Behavior
We analyzed what sort of people see the comment section as civil.
How often an individual comments relates to how much civility they perceive in the comment section. The more frequently people comment, the more civility they perceive.
Perceptions of civility are also related to whether people comment on another site, what we’re calling “exclusive commenting.” Those who comment exclusively on the news site for which they were surveyed perceive more civility than those who also comment on other sites.
Fifty-one percent of those who are not exclusive commenters see the space as somewhat or very uncivil; 42% of exclusive commenters feel the same.
Civility by Commenting Features
Perceptions of civility also vary by what features people believe are available in the comment section (note that these are people’s beliefs, not whether the features are actually available). Those who believe people can post anonymously perceive less civility than those who believe anonymity is not an option.
Those who believe articles regularly receive fewer than 20 comments see the comment section as more civil than those who believe articles on the site regularly receive more than 20 comments.
Finally, those who know that you can report offensive comments see the comment section as less civil than those who do not know that this is an option.
These are perceptions, however, and not whether the space actually allows anonymous commenting, whether the articles regularly receive more than 20 comments, or whether sites allow people to report offensive comments.
We can get a better handle on whether anonymity, or the lack thereof, corresponds with civility by looking at the platform used. Facebook comments, for instance, are less anonymous than many other platforms. Yet there are no differences between perceptions of civility for those using Facebook and those using other platforms. Forty-one percent of those commenting on a site using Facebook say that the comments are somewhat or very civil. Forty percent of those on other sites report the same.
We asked the news staff at each organization to tell us whether their articles regularly receive 20 comments per article or more. Those commenting on sites regularly receiving more than 20 comments per article perceive the space as less civil. Forty-seven percent of those commenting on sites receiving more than 20 comments per article see the space as uncivil compared to 39% of those commenting on sites receiving fewer comments. The more comments, the less civility perceived.
Finally, all of the news organizations included in this study provide a way for people to report offensive comments. Those who knew this saw the comment section as less civil. One explanation for this finding could be that those who have reported a comment as offensive are both more likely to know about the option and to see the space as uncivil.
Civility by Trust and Staff Reports
Trust relates to perceptions of civility. Those who trust the news organization more see the comment section as more civil than those who trust the organization less.
We also asked news staff from each organization to tell us how civil they thought that the comment section is. There is a strong correspondence between the two – the more civil the staff member thinks the site is, the more civil the commenters think the space is.4
Feelings of Connectedness Across News Sites
Respondents reported how connected they felt to other commenters. Twenty-six percent per site felt “somewhat connected” or “very connected” on average. This ranged from 13% to 42%.
Feelings of connectedness are only slightly higher if we look only at those leaving comments; average feelings of connectedness by site among commenters is 30% and varies between 17% and 48%.
Connectedness by Commenting Behavior
Like perceptions of civility, the frequency of commenting relates to feeling connected to other commenters on the site. The more frequently one comments, the more connected one feels to other commenters.
Whether one comments on another site also relates to how connected one feels to other commenters. Those who do not comment on another site feel slightly more connected to other commenters than those who do comment on other sites.
Connectedness by Trust and Staff Reports
Trust is related to connectedness as well. People who express greater trust in the news organization feel more connected to commenters on the site than those who express lower levels of trust.
We also asked the newsroom staff to report on how connected they believed that commenters felt. Here, the staff reports did not match the commenters’ perceptions of feeling connected.5 Sixty-five percent of newsroom staff believed that their commenters felt somewhat connected. Based on the survey data, the percentage of commenters feeling very or somewhat connected did not exceed 50% at any site.
Misperceptions About the Comment Section
Commenters were asked about the features found in the comment section. There were some differences depending on the platform used.
All of the sites had some way for commenters to report offensive comments, yet only 62% of commenters believed that this was the case. When looking at the platform, commenters from sites using Facebook were more likely than other commenters to believe that they could not report offensive comments. Forty-one percent of those commenting on a Facebook site said that they could report offensive comments, compared to 63% of those commenting on a site using another commenting platform.
News staff at 14 of the 20 sites participating in the study reported that articles on the site regularly receive more than 20 comments. Forty-two percent of commenters at these sites agreed that the articles regularly receive more than 20 comments. For the six sites reporting that their articles do not regularly receive more than 20 comments, 28% of commenters surveyed about these sites believed that they did.
We also asked whether commenters thought that they could leave anonymous comments. Twenty-one percent of those commenting on a news site using Facebook thought that they could compared to 44% of those using Disqus, 58% of those using Livefyre, and 61% of those using Civil Comments.2 This suggests that either some survey respondents are misinformed, or that there are various definitions of what counts as anonymity. We suspect that the latter may be the case.
Commenting Frequency
We asked respondents how often they comment. As shown below, there is some variability across the sites. For two large sites, more than 50% of respondents comment on the site weekly or more frequently. For all of the other sites, fewer than half commented weekly or more often.
Trust also relates to commenting frequency. Those who express greater levels of trust in the news organization also comment more frequently than those who express lower levels of trust.
Exclusive Commenting
As we’ve mentioned a few times in this report, we asked respondents whether they commented on other news sites or not. We consider those that do not comment on any other news site as “exclusive commenters.” On average, 78% of commenters on each site identify as exclusive commenters. This varies between 64% and 88% across news sites.
Commenting on How Many Articles
Respondents indicated the percentage of articles on which they leave comments. As shown in the next chart, fewer than half of survey respondents say that they comment on more than 20% of the articles that they read on any given news site.
Commenting by Story Type
Respondents reported the types of stories on which they comment. The top five categories across all respondents are: United States politics or domestic policy, state government, neighborhood or local community, international news, and science.6 For each, we look at variability by news site.
United States Politics or Domestic Policy
Leaving comments on U.S. politics and domestic policy stories varies between 26% and 81% per news site. In general, the larger the news site, the higher the percentage of respondents saying that they leave comments on articles about U.S. politics and domestic policy.
State Government
Commenting on stories about state government also varies greatly: between 11% on one news site and 80% on another. There is a slight pattern based on the size of the site; the smaller the news site, the higher the percentage of respondents saying that they leave comments on articles about state government.
Neighborhood or Local Community
There is extensive variability in commenting on neighborhood or local community stories depending on the site. The percentage indicating that they comment on neighborhood or local community stories varies from 12% to 73%. As shown in the following chart, site size matters for whether people comment on local stories. Those commenting at smaller news sites are more likely to comment on stories about their neighborhood or local community compared to those commenting on larger sites.
International News
Commenting on international news is less common overall compared to local, state, and national politics. Across all of the news sites, an average of 23% of respondents per news site say that they commented on international news. As the chart below shows, there is a slight relationship between the size of the site and commenting on international news; more people on large sites do so compared to small sites.
Science
On average, 19% of respondents per news site comment on stories about science. As the chart below shows, there is one news organization that stands out from the rest. At this outlet, 42% of respondents comment on science. For all other outlets, however, fewer than 30% say that they comment on science. There’s no clear relationship between the size of the site and commenting on science.
Reasons for Commenting
Those completing the survey were asked to indicate why they comment from a list of 16 different options; they could pick as many reasons as they wanted.7 The top five categories across all respondents are: to express an emotion or opinion, to correct inaccuracies or misinformation, to take part in the debate, to add information, and to balance the discussion.8 For each, we look at variability by news site.
Express an Emotion or Opinion
On average, 44% of commenters per news site say that they leave comments in order to express an emotion or opinion. This varies by news site between 34% and 55%.
Correct Inaccuracies or Misinformation
A slightly smaller percentage of commenters, on average 42% per news site, say that they leave comments in order to correct inaccuracies or misinformation. This varies by site between 28% and 55%.
Take Part in the Debate
For each news site, an average of 39% of commenters say that they leave comments in order to take part in the debate. This percentage varies by site between 23% and 56%.
To Add Information
A similar percentage on average, 38% of commenters per news site, say that they leave comments in order to add information. This varies by site between 22% and 66%. There is one outlier here; on one site, two-thirds of respondents say that they comment in order to add information.
To Balance the Discussion
A slightly smaller average of 35% of commenters per news site say that they leave comments in order to balance the discussion. This varies by site between 26% and 49%.
Ease of Navigating the Comment Section
Those participating in the survey were asked about how easy they find navigating the space. In general, majorities find the comment sections easy to navigate.
Locating Comments
On average, 80% of respondents from each news site think that it is easy to locate the comments. This varies between 65% and 91% across sites.
Posting Comments
Seventy-two percent per site, on average, think that it is easy to post comments. The percentage varies between 61% and 87% across sites, as shown in the following chart.
Follow a Thread
Across all of the news sites that participated in this study, an average of 70% of respondents per site think that it is easy to follow a thread in the discussion. There is some variability – between 55% and 83% per site thought that it is easy to follow a thread. In general, those from smaller sites find it easier than those from larger sites.
Liking/Disliking Comments
Two-thirds of respondents per site, 68% on average, think that it is easy to like/dislike news comments. Across the 20 news sites, this varies between 57% and 84%.
Sorting Comments
Just under half, 47%, of respondents per site believe that it is easy to sort the comments. This varies from 34% to 56%.
Methodology
We started this project by asking a variety of news organizations to distribute our survey. We sought to invite a diverse group of news organizations, ranging in geographic location, size, and medium (newspaper, television, radio, digital-only, etc.). We reached out to each organization through emails sent either by the Center for Media Engagement or The Coral Project. We continuously updated our list and invited new participants as others declined. The study took place May 25 through December 13, 2016.
Each organization had the same survey; the only difference being that we tailored each survey so it named the participating organization. We let each outlet determine how they wanted to encourage their audience to take part in the survey. Strategies included writing an article about the survey on the news site, sharing the survey link on social media or with a banner ad on the organization’s website, or distributing the link via a newsletter sent to subscribers. In total, 22 organizations promoted the survey, however this report only includes results for 20 of the organizations. One outlet eliminated comments mid-way through the survey, and a second outlet received very few responses. The participating outlets include: Alaska Dispatch News, AL.com, The Arizona Republic, The Atlantic, Civil Beat, The Dallas Morning News, Deseret News, Fort Worth Star-Telegram, KXAN, PBS NewsHour, Philly.com, The Seattle Times, Southern California Public Radio/KPCC, The State Journal-Register, The Texas Tribune, TribLIVE, Twincities.com / St. Paul Pioneer Press, Voice of San Diego, The Washington Post, and Willamette Week. Across these sites, seven used Disqus, four used Livefyre, three used Facebook, three used in-house platforms, two used Civil Comments, and one used Viafoura when the survey was fielded.
For this report, we categorized the sites based on their size according to Amazon.com’s Alexa analytics rankings. Alexa ranks sites within the U.S. based on their traffic. The site garnering the most traffic in the United States is ranked 1 and each site is then ranked with increasing numbers based on its traffic. For this study, small sites are those ranking more than 10,000 in the United States, large sites are those ranking less than 2,000, and medium sites are those that fall in the middle.
SUGGESTED CITATION:
Stroud, Natalie Jomini, Van Duyn, Emily, Alizor, Alexis, Alibhai, Alishan, and Lang, Cameron. (2017, January). Comment section survey across 20 news sites. Center for Media Engagement. https://mediaengagement.org/research/comment-section-survey-across-20-news-sites
- In total, 18,729 people clicked to begin the survey and 9,575 completed the entire survey. We use all available data for each question, reporting the sample size beneath each table. [↩]
- We exclude commenting systems for which we did not have data from more than one news site from this analysis. [↩][↩]
- We asked survey respondents to indicate how much of the time they thought they could trust the news organization to report the news fairly. Across all the organizations, 32% said just about always, 44% most of the time, 20% only some of the time, and 4% none of the time. There was variability by news organization. Between 10% and 71% per organization said that they could trust the news just about always. [↩]
- The correlation between news staff reports and commenter ratings of civility (n=20) is 0.63. [↩]
- The correlation between news staff reports and commenter ratings of connectedness (n=20) is -0.08. [↩]
- The other categories included: health and medicine, technology, sports, music and the arts, religion and spirituality, business and finance, and other. [↩]
- Question wording draws on previous research from Diakopoulos, N., & Naaman, M. (2011, March). Towards quality discourse in online news comments. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (pp. 133-142). [↩]
- The other categories included: to educate others, to ask questions, to share an experience, to note missing information, to persuade others, to discuss with others, to add humor to the discussion, to see how others react, to be part of the community, to show sympathy to others, and to applaud other people or actions. [↩]