Why Maldita.es surveyed users to find out their ‘superpower’

This case study was first published in Engagement Explained, a fortnightly newsletter that brings you in-depth and replicable engaged journalism case studies from news organisations across Europe. Sign up here to receive the next edition in your inbox.

In a nutshell

A 10-question survey that asked users to share what they were prepared to contribute to Maldita.es and its fact-checking operation.

Background

• Maldita.es is a non-profit fact-checking and data journalism platform focused on combating the spread of disinformation. Its name translates as ‘the damned’ and its tagline is ‘journalism to not be fooled’.
• It was started in 2014 by Clara Jiménez Cruz and Julio Montes, two Spanish broadcast journalists who sought to fact-check stories in a social-media friendly format. They started Maldita Hemeroteca and, in 2018, they quit their jobs and launched Maldita.es.
• The team spends most of its time debunking claims and stories brought to them by the community (the members of which are called ‘malditas’ and ‘malditos’), and sharing the results back out to them via social media to circulate as far as possible.
• The organisations also creates bots and browser extensions to enable users to combat misinformation and promote transparency in public and private institutions.
• It recently launched added two new verticals: Maldita Ciencia, a science public engagement project, and Maldito Feminismo, a platform to look specifically at hoaxes related to women and gender.
• The team recently collaborated with RTVE to help fact-check statements by candidates taking part in the general election television debate.
• In October 2018, Maldita was awarded funding from the Engaged Journalism Accelerator to build a CRM to organise its community and streamline the information submitted to Maldito Bulo, its fact-checking service.

How did they do it?

• The survey sought to gather data on what users thought about Maldita.es, how they found out about it, which projects they knew about and the skills they were prepared to contribute to the organisation. The team dubbed these skills ‘superpowers’.
• Clara Jiménez Cruz, co-founder of Maldita.es, put together 10 questions including ‘did you know Maldita.es is an independent media and a not-for-profit?’ and ‘if you had to define Maldita.es in a tweet or phrase to explain it to a friend, how would you do it?’
• She wanted to make it fun to fill in so she used simple, playful language and included emojis, a picture of Homer Simpson wearing a Maldita.es t-shirt and an explanation of why engaging the community was important.
• The survey included some light-hearted asks, for example, whether anyone in the community was able to fix the door of the organisation’s offices in Madrid, which had been off its hinges for some time (spoiler: someone came forward to fix the door).
• Those who filled in the survey were invited to a party to meet the Maldita team and other malditos.

What did they learn?

• The team didn’t expect to get more than a few hundred responses but received 2,645 answers in only four days, after which the survey was closed. Staff put this down to the fact that they rarely make asks of their community so users were ready to respond.
• There was a variety of answers to the superpower question. Some people put down their job (e.g paediatrician) and others wrote specific aptitudes or skills (e.g “I can sing in Japanese”). There were very few common themes or threads.
• The survey helped Maldita.es reach new people. Around 30% of responses (around 800) came from people who hadn’t engaged with the organisation before.
• They found that there’s a trade-off between free text inputs, where users can write what they want, and drop-down questions that provide structured data about the community. The team decided to use more free text inputs because everyone defines their superpower in a different way (as the answers proved).
• Explaining how users’ superpowers could help in the fact-checking process was very important. Clear examples of how verification and debunking would work helped people better understand a process that was new to many of them.
• There were some themes in the words people used to describe Maldita.es, including datos (data), engañen (deceive or hoax), útil (useful) and gente (people). These were turned into a word cloud.
• The team is looking at how it can add the responses to a database which can be searched by journalists looking for specific expertise for a particular debunk or fact-check.

In their own words

Clara Jiménez Cruz, co-founder, Maldita.es

"We want to engage our audience in the process of debunking since we know we need them often. Since Maldita's way of talking to its audience is already a bit cheeky, we thought the word ‘superpower’ worked just right with our mood.”

How would you improve it?

"I wouldn't do it entirely different but I think we need to do a second experiment with chosen community members that are already contributing in different ways so that they can help us sort out how to explain the processes in a better, more clear way”.

Now try it for yourself

• Pew Research Center, which collects data and produces many reports, has a guide to how it designs questionnaires (including doing pre-tests the night before the surveys go live). Harvard has a similarly useful design tip sheet.
• Hearken and GroundSource collaborated on an ethical framework for engagement to help newsrooms cultivate positive relationships with the people they serve.
• The Membership Puzzle Project has published the survey (and a bunch of other materials) that it uses when doing user-centered design thinking and testing.
• If you’re a Krautreporter member, you may know that the German organisation asks users to fill in their expertise in the ‘My account’ section of the website. It’s optional but around 30% of Krautreporter members have done so.