Dr. Rick Adams was involved in the Great Brain experiment. The Great Brain experiment is a gratis smartphone application that uses games to collect data of basic brain functions. I talked with Dr. Adams about his experience of using a gamified approach to data collection and public engagement.
Background. Gamification is the use of game-elements in real-life settings [see previous article]. The Great Brain Experiment tests people in psychological experiments, in form of games. These experiments are looking at brain functioning, such as memory, impulsiveness, perception and decision-making. The publications that resulted from this application are listed here.
 = signals a comment I made
Q: Why did you use a gamification approach to data collection?
Our project came about because the Welcome Trust wanted to do a public engagement exercise. So they had a budget which they gave to all of their centres to spend on any initiative that they could come up with related to public engagement in science. So they gave us about 7,000-10,000 pounds and then we had to think about which project we wanted to do. And the idea of putting experiments on phones just seemed liked such a natural way to get people interested in science, because people are already playing games on their phone. Its also a good way for us to collect data we thought at the time, and it turned out to be.
Q: How much more effort was it in relation to a normal study?
We had a professional game developer, who had the game engine. So we had to spend quite a bit more money, we had to spend another roughly 10,000 pounds to get the games coded as professionally as possible. Because a big problems with these kinds of approaches is that scientist aren’t used to using game engines making games look like real games. And if you want people to play your experiment, it has to be fun. If there is no fun they won’t do it.
Q: How difficult was it to collect data while also having a fun game?
The big problem is the amount of data you get, you could only have a game that last 2 minutes, maximal 4 minutes. In the lab you would get people to engage in a task for 40 minutes, whereas you had a tenth of that time, but you also have maybe a 1,000 or 10,000 more subjects. So our hope was that, although the data was much more noisy and we had less data from every person, this massive increase in numbers would lead us to find the relationships that have been published using task in the lab. And we did, so it does work. The most popular game in our whole battery was the shortest game which is between 90 seconds and 2 minutes. The longer they are, the quicker the drop-off.
Q: Despite the time constraint, did you encounter any other problems?
Ethics. So you have to get ethical permission to collect people’s data like that and then anonymize is so that you can use it. That was one problem. It is expensive, so we ended up paying this game developer 30,000 pounds altogether, because we put out even more games in the second version. Most labs don’t have that money lying around, although it is relatively small for a grant so it’s possible to get money to do something like that. With the actual data quality we didn’t have problems. We were very lucky that we could use the Welcome Trust PR machine to publicize it. I think we had more than 100,000 people playing it. It was published in The Guardian and Wallstreet journal, but most people don’t have access to that kind of PR and now a lot of games are more common and it is even harder to stand out and get yourself noticed. So we were kind of lucky. Obviously, if you can’t get that amount of people than the noise in the data can become a problem.
Q: Besides having many participants, what are the other advantages of gamification of data collection?
An additional thing is because you get the same people playing five or six different games you can look at how performance on one task correlates with the performance on another task, which is not done that often in Psychology. For example, does attention
span correlate with reward sensitivity? Things like that. You could also correlate it with clinical data, for example with people’s depression scores. We never tried to correlate it with how much people move around, we never got ethics to collect more detailed user information. It is much more difficult to get ethical agreement to that. You can look at much older people, it is very hard to recruit over 60s for cognitive experiments. We only had a few hundred people over 60, whereas thousands of younger people, but even a few hundred was enough. We could then look at task performance across the life-span.
Q: That is interesting I would expect that you mostly get younger people who are interested in video games.
We actually got an enormous number of those people compared to other people. In normal Psychology research you get so few of the other people at all, so even using this approach you still get more than you would recruit normally. So actually its better, we can look at people in different countries, you could contrast Android and IPhone user (laughs).
Q: Which research questions cannot be answered by this approach?
Any study that requires a lot of data for every individual subject to fit a model. Testing complex models can be very difficult to do by this approach, because of the short time people will play games. Any situation where you need very tight control over what kind of people are in your groups. For example, if people are only accepted into participating research with a certain score, this could lead people just lie in these questionnaires, online that is even more likely than in person.
Q: How do you think gamification will change the way we do science?
Most of psychology is built on experiments involving Psychology students [for further reading, see Heinrich et al. (2010)] , and this has to be addressed. This approach gives people a chance to validate findings in a much bigger sample. I think it is not going to be the only future for doing experiments, you sometimes want to control the environment. Also it is difficult to get our participation numbers in the future. In Psychiatry there is this big debate on how we can redraw diagnostic boundaries between disorders, and finding other kinds of structure in how people behave. So population studies involving lots of people and lots of task could be very good at finding new structure in people’s behavior, for example compulsive traits or sociability traits. They might constitute separate axis on which you can describe individual variation we haven’t noticed before. Clair Gillan [see Gillan et al.(2016)] has done a big study using Mechanical Turk, she got 1,000 people to do it online and found that their performance of psychiatric traits clustered into three categories.
Q: Will it become more common to use gamification and similar technology in the future?
The fact that you have to get someone to code your game in a games engine is probably a dis-incentive for a lot of people, because it is expensive and takes time. If you are paying people to take part in a study it might be less important to have a fun game. Whereas if you want people to participate for nothing it has to be engaging, and you have to pay someone to make a fun game. Even though we paid the developer 30,000 pounds at the end that still works out as 30 pence per subjects, which is a lot less than we would pay in the lab. So it is actually more cost efficient. Just depends if you got that amount of money, you will need a special grant for it. It would be helpful to have platforms like Mechanical Turk for people to put their games on, rather than having people putting games in different places because the former approach might get more people involved.
Q: Is it the responsibility of a scientist to increase public engagement with science?
I think there are problems in how public engagement is done, because it requires such an enormous amount of work, and a lot of people doing science are already extremely busy. Founder could give more administrative help and assistance to get larger scale projects off the ground. It turned out to be worthy for us, but we would never have come up with such a project at first place, because it would be so much work. If we had someone to help us with the administrative work, not only money, that would have made the whole thing easier. That would lead to much bigger public engagement projects in the future involving more scientists, if it would be coordinated by an extra person.
Thank you very much!