Published on November 24th, 2014

“No one has tried this kind of work before in such a way — could I, as the first attempt, be successful?”

For three months, research associate Arielle Cardona attempted to code for incivility within online news comments in new ways –using paid Workers in Amazon’s Mechanical Turk, undergraduate students in the Mechanical Turk sandbox, and by machine training the content analysis program WordStat. Using a previously coded data set as the comparison “Exemplar” coding, she compared how well these groups and program were at accurately detecting incivility.

Although we never reached perfect agreement, tweaking and editing our instructions and question format consistently made us better. This document serves as a progress report, with some thoughts on the next possible steps in research.


  • Arielle Cardona

    Arielle Cardona
    Research Associate

    Arielle Cardona (MA, University of Texas at Austin) worked for the Center for Media Engagement from 2014 to 2015. She is a recent graduate of the UT Communication Studies program, where she studied women in politics and online political discourse. Prior to her MA, Arielle received her bachelor’s degree in communications at the University of Arizona.