I'm developing a directory which reflects user ratings for news stories. Stories would rated individually, but would also be aggregated into a rating for the author/contributor and for the news site/domain.
My goal is to create ratings which tend to point to quality journalism overall, even if individual ratings often reflect emotional bias.
Who can help me understand these dynamics and can advise me on how to create user rating mechanisms?
I know a lot of academic research on how people read and evaluate news, and a lot of academic research on the sociology and motivation behind sharing. Most of that that I'm aware of points to people are more likely to share and positively rate content that they find serves homophilic impulses and/or reinforce their epistemic bubble (a pop version: the filter bubble).
Not sure how you make the connection between "people have said they rate this story highly" and "quality journalism", though. Could you point to research on that?
Signed, former journalist and academic.
Dan is right - lots of research in this area, and lots of technology to support it. Perhaps the best first step is to do a good job of researching what has been done before rushing off into this arena.
Hmmm, maybe the solution is combining a couple of key performance indicators - Daniel, in your opinion, would people more prone to impulsive behavior spend more or less time on a page/article? Or at least, less time reading an article? Do you know of any research that covers something like that?
If so, then you may be able to combine time on page and ratings -
1) Disregard ratings given before a certain amount of time (like anything given under 30 sec etc).
2) You could even just use the avg time spent on the page as a quality indicator itself.