Using Facebook Messenger to Improve Online Discussions

Research

Download Full Report

← Journalism

Researchers

Caroline Murray

Caroline Murray

Martin J. Riedl

Affiliate Fellow

Natalie (Talia) Jomini Stroud

Director, Knight Faculty Fellow

SUMMARY

The Center for Media Engagement partnered with Vox and Spaceship Media to explore whether the design of online discussion groups can affect conversation quality and if Facebook Messenger can be used as an engagement tool. We invited members of The Weeds Facebook group – a 20,000 member discussion forum dedicated to the Vox produced podcast about politics and policy – to receive training from Spaceship Media on how to have meaningful discussions and to then take part in small group chats on Facebook Messenger between March 23 and May 6, 2020. We found that:

  • Members of the group chats reported that they were less likely to self-censor, or withhold their opinions, in their small group chats than in the Facebook group.
  • Participants in the group chats asked each other more genuine questions and shared more personal experiences than did commenters in the Facebook group. Conversations in the Facebook group contained more inflammatory questions but were more likely to stay on-topic than the group chats. This could be due to the type of people motivated to sign up for small groups or the quality of the conversation inspired by these spaces, or both.
  • Of participants who said they would be willing to support Vox financially, group chat participants said they would contribute more money on average than those who wanted to participate in the group chats but weren’t able to do so based on the study design.

THE PROBLEM

Engaging in meaningful dialogue allows people to learn from others’ experiences and develop more informed attitudes and opinions. However, many Americans feel that the quality of political discourse has soured in recent years and some avoid it altogether.1 What kind of online experiences engage people and allow for respectful dialogue is an area of exploration. The Center for Media Engagement partnered with Vox and Spaceship Media to evaluate one possible approach to improving political discourse online that allows newsrooms to translate the popularity of a large Facebook group into more meaningful, smaller group conversations.

The Weeds Facebook group is a 20,000 member discussion forum dedicated to the Vox produced podcast about politics and policy. We invited members of the group to receive training conducted by Spaceship Media on how to have meaningful discussions and to then take part in small group chats on Facebook Messenger between March 23 and May 6, 2020. Of those who volunteered, we randomly gave half a chance to participate in weekly group chats. We surveyed participants – both those who were given the chance to participate in the group chats and those who were not – before and after a six-week experience. We examined whether participating in group chats, as opposed to participation in the large Facebook group alone, affected the quality of conversations, attitudes toward Vox, and connections between group members.

KEY FINDINGS

We found evidence that the discussions in the group chats were of a higher quality than those in the Facebook group:

  • Members of the group chats reported that they were less likely to self-censor, or withhold their opinions, in their small group chats than in the Facebook group.
  • Although group chat participants felt that the group chats were just as dominated by a few people as the large Facebook group was, members of the group chats rated people in their chat as more respectful and civil than commenters in the Facebook group.
  • Participants in the group chats asked each other more genuine questions and shared more personal experiences than did commenters in the Facebook group. Conversations in the Facebook group contained more inflammatory questions but were more likely to stay on-topic than the group chats. This could be due to the type of people motivated to sign up for small groups or the quality of the conversation inspired by these spaces, or both.

Those who participated in the training and group chats largely did not change their views of The Weeds Facebook group or Vox in comparison to those who did not participate. There were also mixed consequences for engagement:

  • Participants in the group chats reported a decreased sense of loyalty to the Facebook group after their small group chat experience, likely because nearly three-fourths of people found the experience to be better than the large Facebook group experience.
  • Of participants who said they would be willing to support Vox financially, group chat participants said they would contribute more money on average than those who wanted to participate in the group chats but weren’t able to do so based on the study design.

IMPLICATIONS

Receiving guidance on how to have tough conversations and then engaging in smaller, more intimate conversations on Facebook Messenger can facilitate better political dialogue and connections between people than commenting on posts in a large Facebook group. However, group chats on Facebook Messenger are not a foolproof tool to increase audience engagement with news organizations. Newsrooms should consider the time commitment involved with and sustainability of moderating these political discussions.

THE EXPERIMENT

For this study, funded by the News Integrity Initiative at the Craig Newmark Graduate School of Journalism at CUNY, members of The Weeds Facebook group were invited to participate in small group chats about politics on Facebook Messenger. We assigned those who were interested, at random, to have one of two different experiences.

A random half of those interested in taking part were invited to participate in a one-hour virtual training session about how to have meaningful conversations. The session was led by Spaceship Media, an organization that works to promote civic dialogue in divided communities. After the training, participants were placed, again at random, into one of six group chats of 8-10 people to discuss new episodes of The Weeds. Each week for six weeks, participants were asked to listen to a current episode of The Weeds. The moderator of The Weeds Facebook group then posted a relevant discussion prompt written by Spaceship Media in the group chats. Group chat participants were asked to take two surveys – one before their experience with the training and group chats and one after the six weeks had ended.

The other group of participants in the study did not take part in any of the group chats or the training. They served as a “control group” so that we could compare their experience to those who participated in the training and group chats and understand whether the training and group chats affected people’s beliefs, attitudes, and behaviors. The moderator of The Weeds Facebook group posted the same discussion prompts shared in the small groups on the main Facebook page each week. This group of participants also took the two surveys approximately six weeks apart, with their version of the second survey excluding any questions that asked specifically about the training or the group chats.

Group Environment and Connections Between Group Members

People who participated in the training and group chats largely felt similarly about the Facebook group after their experience. Group chat members didn’t develop a stronger social network in the Facebook group, nor did they rate the group environment any differently after participating. They also did not report different levels of respect and understanding toward those with opposing views on political issues, such as Medicare for All, than did respondents who did not participate in the group chats.

Looking specifically at those who participated in the group chats, there were some differences in their perceptions of the small groups compared to the large Facebook group. Overall, they viewed their group chat more positively than they viewed the Facebook group.

Group chat participants felt a stronger connection to members of their group chat than to the members of The Weeds Facebook group.2 They also rated members of their group chats as more respectful and more civil than the people in the Facebook group.3

We also examined the extent to which people felt free to express their thoughts in the group chats versus in the Facebook group. Members of the group chats reported they were less likely to self-censor, or withhold their opinions, in their group chats than in the Facebook group.4Screen Shot 2020 11 30 at 2.09.53 PM

Participants in the small group chats didn’t believe that their chats were more or less likely to be dominated by a few people than the Facebook group was.

Engagement with Vox

The experience with the training and the small group chats had mixed effects on engagement. Here, we compare those who were assigned, at random, to participate in the small group chats to those who were not given the opportunity to participate in the group chats.

Small group chat participants weren’t more or less likely than members of the Facebook group alone to report visiting Vox’s website, sharing Vox’s content on social media, receiving a Vox newsletter, or listening to another Vox podcast during the time of the study. Small group chat participants were more likely to say they had listened to The Weeds podcast, but they were reminded to do so each week as part of the study.5 Small group chat participants also reported reading comments and posts in the large group less frequently in the past month than respondents who did not participate in the group chats, but again, they were asked to prioritize conversations in their group chat as part of the study.6

Group chat participants felt less loyal to The Weeds Facebook group after participating in the small group chats.7 Loyalty likely declined because many in the small group chat found the experience superior to the large Facebook group experience. There were no changes in how proud they were to be a part of the Facebook group, however.Screen Shot 2020 11 30 at 2.12.29 PM

Many of those in the small group chats enjoyed the experience; 74.0% of participants reported that their conversations were better in the small group chats than in the Facebook group. 85.7% of the small group chat participants also said the experience was enjoyable and that they would like to participate in other small group chats led by Vox in the future.

Participation in the group chats didn’t affect overall willingness to financially support Vox; 12.2% of small group chat participants and 12.2% of the members of the Facebook group alone said they wouldn’t contribute to Vox and 87.8% of both groups responded “yes” or “maybe” when asked if they would financially support Vox.8 Among the participants who said they may be willing to support Vox financially, however, members of the small group chats would contribute more money than those who did not participate in the group chats. 42.9% of the group chat participants said they would be willing to contribute more than $10, compared to 11.1% of the people who did not participate in the group chats.9

Discussion Quality

We explored whether the small group chats or the large Facebook group yielded better political conversation. Here, we compare comments left in the small group chats to comments left by anyone in response to the same discussion prompts on the larger Facebook page. This means that the findings are more speculative because we cannot be sure whether differences between these groups are because (a) the sort of person who volunteers for a study like this differs from those who post to the page, or (b) the different experiences in the small group chats and the larger Facebook group.

We identified a few key elements of a high-quality discussion, including asking genuine questions, using evidence to support a position, sharing personal experiences, acknowledging the points of others, and staying on-topic.10

Group chat participants asked more genuine questions, or questions that sought information or an opinion from others, than did commenters in the large Facebook group.11 In contrast, non-genuine questions, or sarcastic, inflammatory, and leading questions, appeared more often in comments left in the large Facebook group.12Screen Shot 2020 11 30 at 2.13.13 PM

More of the comments on the posts in the large group stayed on-topic or were relevant to the episode or prompt than in the messages in the group chats.13 81.0% of the comments on the posts in the large Facebook group were on-topic compared to 56.2% of the messages from those in the group chats.

We did not find any differences among the messages in the group chats and comments in the Facebook group in whether evidence, such as links, statistics, or news articles, was used.14 However, there were differences in the types of evidence participants presented. Commenters in the large Facebook group employed more numbers or percentages in their discussions than group chat participants.15 Group chat participants cited other kinds of evidence, such as books, podcasts, or movies, more often than commenters in the large group.16

People in the group chats also shared more personal experiences than did the people in the large Facebook group.17 Examples of personal information included their interactions with the healthcare system, their struggles adjusting to coronavirus-related restrictions, and anecdotes about their family, job, or friends.

We did not find consistent differences between the group chats and the large Facebook group in terms of whether participants’ messages or comments were replies to another person. We defined replies as messages or comments that acknowledged someone else’s statement or answered their question.

Opinions about the training session

We also asked people who participated in the Spaceship Media training about their experience in the session. 56.0% of the participants found the experience valuable and 58.0% agreed that the training helped prepare them for discussions in the small group chats. 42.0% said they learned new things during the training session and 38.0% agreed that they had better discussions in the small group chat because of the training.

METHODOLOGY

Experiment

Participants for the project were recruited from a series of call-out posts in The Weeds Facebook group. The moderator of the Facebook group posted the call-out three times, on February 28, March 2, and March 5, 2020. The host of The Weeds podcast, Matthew Yglesias, also shared the call-out on Facebook on March 6, 2020. The posts included information about the project and a request for people to take a survey about their experiences in The Weeds Facebook group, their opinion of Vox, and their ability to participate in the project.

In total, 227 members of The Weeds Facebook group finished the survey and 184 qualified for the project, meaning that they gave their contact information and said that they were willing to participate in group chats on Facebook Messenger and a one-hour training session from Spaceship Media.

We randomized the 184 qualified people into two groups, one group that received training on how to have meaningful conversations and participate in small group chats (the treatment group) and one group that took a second survey after the project, but otherwise maintained their normal activity in the larger Facebook group (the control group).

The training sessions were led by the Spaceship Media team and framed the group chats as an opportunity for participants to learn from people with different perspectives and to build relationships. Participants were encouraged to deeply listen to one another and to express kindness, curiosity, and empathy in their conversations. Spaceship Media and Vox hosted six of these one-hour training sessions between March 10 and 14, which members of our research team also attended.

Of 95 invited participants, 61 members of The Weeds Facebook Group attended a training session. For the purposes of this report, we consider those who were invited to attend training but did not do so as part of the control group.18 The great majority of the findings shared in this report remain consistent whether those who were invited to attend a training but did not do so were included as part of the control group, as part of the treatment group, or excluded from the analysis entirely. The only finding that changed is how often members of the treatment group read comments and posts in the large Facebook group during the study. When those who were invited to attend a training but did not do so were included in the control group or excluded from the analysis, the treatment group’s engagement with the Facebook group significantly decreased after their experience in the group chats. When those who were invited to attend a training but did not do so were included as a part of the treatment group, this finding was not statistically significant.

We randomized the 61 participants into 50 different formations of six group chats of about 10 individuals. We ran ANOVA, crosstabulations, and homogeneity of variance tests to examine pre-existing differences between the group chat formations. We chose the formation out of the 50 iterations with the lowest number of differences in our key variables and major demographics between group chats. In this iteration, there were significant differences across the groups for feelings about certain political topics, including President Trump, North Korea, Medicare For All and the Republican party, but there were no differences for any of the other variables.

The moderator of The Weeds Facebook group created the group chats on March 23, 2020. A researcher and members of the Spaceship Media team were also added to each of the six group chats. Every week for six weeks, small group chat participants received an email asking them to listen to a new episode of The Weeds podcast and discuss the episode in their group chat.19 Spaceship Media crafted discussion prompts for each week that touched on the major themes from the episodes, which included the Democratic primary, racial inequality, and how coronavirus could impact healthcare and the economy. Starting on March 25, the moderator sent the prompts to each group chat every Wednesday afternoon. On the same day, the moderator also posted the prompts in the larger Facebook group.

The last prompt was posted in the group chats and the larger Facebook group on April 29, 2020. A week later, the moderator sent a message to all the group chats announcing that the project had concluded and that participants could exit the chats. Aside from sharing the discussion prompts, the moderator refrained from moderating the small-group discussions unless specifically tagged by a group chat member. When the moderator was tagged during this project, it was to answer procedural questions, such as the time the prompts would be posted.

On May 6, 2020, members of the control and treatment groups were sent a post-survey via Qualtrics. The post-survey was nearly identical to the survey participants had taken in March. The post-survey for the group chat participants also contained questions specific to their experience in the group chat and the training session. When we analyzed the survey data, we only used responses from people who took both the pre-survey and post-survey (62 people from the control group and 51 people from the treatment group).

Content Analysis

For the content analysis, researchers manually copied the text of all messages from the six group chats and all comments on the prompts in the larger Facebook group into Excel as the project was unfolding. Messages from the moderator of The Weeds Facebook group or the Spaceship Media team were removed from the dataset. This resulted in a dataset of 242 comments from the posts in the Facebook group and 2,491 messages from the group chats. Three coders examined at least 20% of the messages and comments to obtain intercoder reliability.20 The coders then evaluated the full dataset.

In our analysis, we included all comments on the posts in the Facebook group, not only the comments left by members of the control group. Therefore, the results of the content analysis could be attributed to the differences in the kind of discussion forum (Facebook group versus Facebook Messenger), differences in the kinds of people who express interest in discussion projects, or a combination of the two factors.

Some members of the group chat and commenters on the Facebook posts in the large group sent or posted several messages in a row. In order to see if this style of sharing had an outsized impact on our content analysis, we examined all of our variables with these messages separated and with these messages condensed as one message. The results of the content analysis hold in every case except for how often a message was a reply to someone else’s message or comment. When we ran our analysis with the messages in a row separated, there were no significant differences between how often comments from the Facebook group and the group chat messages were replies. When we ran our analysis with the messages in a row condensed, more of the group chat messages were replies. However, in both cases, the most active group chat had an impact on the results. When we removed the most active group chat’s messages from the analysis with the messages in a row separated, the comments in the Facebook group were more likely to be replies. There were no significant differences between the Facebook group and the group chats when we excluded the most active group’s messages and ran our analysis with the messages in a row condensed. Removing the most active group chat from the analysis did not affect any of the other findings in this report.

Demographics 1

SUGGESTED CITATION:

Murray, Caroline, Riedl, Martin J., & Stroud, Natalie Jomini. (December, 2020). Using Facebook Messenger to improve online discussions. Center for Media Engagement. https://mediaengagement. org/research/using-facebook-messenger-to-improve-online-discussions

  1. Pew Research Center. (2019, June 19). Public Highly Critical of State of Political Discourse in the U.S. https://www.pewresearch.org/ politics/2019/06/19/public-highly-critical-of-state-of-political-discourse-in-the-u-s/. [↩]
  2. Paired t(49) = 2.04, p = .047. We asked participants to respond to six statements intended to measure social connections, including “I have strong personal relationships with some people in (my small group chat/The Weeds Facebook group)” and “I feel like I am part of a community in (my small group chat/The Weeds Facebook group)” on a scale of 1 (strongly disagree) to 7 (strongly agree). We combined these responses into one variable to test for differences (Cronbach’s alpha: 0.827). [↩]
  3. Respectful: paired t(48) = 8.18, p < .001. Civil: paired t(49) = 9.82, p < .001. [↩]
  4. Paired t(49) = -5.49, p < .001. We asked participants to respond to six statements intended to measure their willingness to self-censor, including “It is safer to keep quiet than post an opinion in my small group chat that I know most others don’t share” and “When I disagree with others in my small group chat, I’d rather go along with them than argue about it” on a scale of 1 (strongly disagree) to 7 (strongly agree). We combined these responses into one variable to test for differences (Cronbach’s alpha: 0.833).[↩]
  5. χ2(1, 113) = 5.25, p = .02. [↩]
  6. We tested for differences using ANOVA. There was a significant effect of condition F(1, 113) = 7.27, p < .05. [↩]
  7. We tested for differences using ANOVA. There was a significant effect of condition F(1, 102) = 10.35, p < .05. [↩]
  8. χ2(2, 98) = .434, p > .05. [↩]
  9. χ2(1, 78) = 9.96, p = .007. [↩]
  10. Masullo Chen, G., Riedl, M. J., Shermak, J. L., Brown, J., & Tenenboim, O. (2019). Breakdown of democratic norms? Understanding the 2016 US Presidential Election through online comments. Social Media + Society, 5(2), doi: 10.1177/2056305119843637; Stromer-Galley, J. (2007). Measuring deliberation’s content: A coding scheme. Journal of public deliberation, 3(1), doi: 10.16997/jdd.50; Stroud, N. J., Scacco, J. M., Muddiman, A., & Curry, A. L. (2015). Changing deliberative norms on news organizations’ Facebook sites. Journal of Computer-Mediated Communication, 20(2), 188-203, doi: 10.1111/jcc4.12104. [↩]
  11. χ2(1, 2733) = 9.91, p = .002. [↩]
  12. χ2(1, 2733) = 54.94, p < .001.[↩]
  13. χ2(1, 2733) = 55.96, p < .001. [↩]
  14. χ2(1, 2733) = 0.01, p > .05. [↩]
  15. χ2(1, 2733) = 32.85, p < .001. [↩]
  16. χ2(1, 2733) = 13.89, p < .001. [↩]
  17. χ2(1, 2733) = 5.18, p = .02. [↩]
  18. One participant who did not attend a training session was mistakenly assigned to a group chat. We ran all tests from the survey and the content analysis with and without their responses and messages. Removing or including their responses and messages did not change any of the results shared in this report. [↩]
  19. On the week on April 8, 2020, some members of one of the group chats did not receive an email from the Vox moderator telling them the episode to which they should listen. They still received the prompt in the group chat and exchanged messages about the topic at hand (coronavirus’ impact on healthcare). [↩]
  20. Reliabilities were computed using Krippendorff ’s alpha. The metric runs from 0, indicating a lack of reliability and extensive disagreement among people examining the same content, to 1, indicating perfect agreement among people examining the content. Scores greater than 0.80 indicate strong reliability and those greater than 0.67 indicate acceptable levels of reliability. The final Krippendorff ’s alphas for the variables were: relevance (0.791), genuine questions (0.871), inflammatory questions (0.749), presentation of evidence (0.858), numbers as evidence (0.779), URLs as evidence (0.924), reports, documents, news articles or academic research as evidence (0.806), other kinds of evidence (0.773), personal experiences (0.819) and replies (0.729). [↩]