Eric Thrailkill — University of Vermont
Crowdsourced participant sampling offers addiction science a complimentary approach to lab-based studies, clinical trials, and epidemiological data. In this talk, I will describe a recent study that used the Gorilla Experiment Builder platform to conduct an experiment examining risk factors for cigarette smoking and other substance use. I will focus on what I learned along the way; the features I used for the project, the challenges that came up, and how the design functions allowed me to create and carry out a rigorous design that could be reproduced easily. I will further describe how this approach continues to facilitate projects with in-person and online samples in my laboratory.
Eric Thrailkill 0:00
I have a laser point, great. So yes, thank you so much for the invitation to talk about some of my work here. I will go through sort of like a story I tried to put together about it. So that’s gonna go somewhat like how I found Gorilla experiment builder, which is sort of changed my research life. And, and then how just just doing a little background on online research on substance abuse risk factors, which is what I’ve been doing.
And then provide a case of that drive looking at individual differences in loss aversion, and risk for cigarette smoking and other substance abuse problems. And then talk about some current directions, I’m taking this research and considerations and sort of ways I want to develop things. So thank you very much. I’ll proceed.
So during the pandemic, I was spending a lot of time, well, earlier in the pandemic, I should say, I was spending a lot of time trying to figure out how to do research online. I’ve been doing that a little bit before that, but the pandemic kick started that. And so I was doing all the programming stuff and that sort of thing and trying to figure everything out. And eventually, somehow, someway, I came across this podcast online. And it was a webinar, actually, by Joe Devlin, who was putting on a series of interviews with academics in psychology, neuroscience, behavioural sciences at University College London, on different topics, and one of the topics was Gorilla experiment builder. So this is what introduced me to this tool, which I eventually got to use.
And I’ve been using ever since. And so in, in the venue I’ve been using, it is, is trying to study risk factors associated with substance abuse. And since well, before the pandemic, people in the substance abuse research field have been interested in online research. And so this is a paper, which seems kind of old now. But it came out 2019 pre pandemic, of course, on the use of crowdsourcing addiction science, and what this paper goes over are, are a number of areas where online research has been particularly helpful for setting substance abuse problems and people and so it goes over the the use of crowd sourcing to to get gather large, large samples to basically replicate what we’ve seen in laboratory studies, looking at case control type designs, folks who are using substances versus folks who do not use substances and replicating those different scales are a good place to start.
But other users have entered develop more intervention tools, and pilot test tools before using them in person with clinical samples, and also develop new measures and validate them rapidly. And then they also are able to contact people over and over and not have them come into the laboratory for longitudinal measure measurement, which is extremely useful. So what this paper ended up concluding was that online addiction research is going to be complementary to clinical trials, human laboratory studies, epidemiological studies, and it’s going to help overall benefit the field by improving reproducibility, rigour and expanding possibilities, the study factors related to substance substance abuse, and overall health related behaviours.
And so with that sort of background, I was interested in in using this and so this is just a figure showing that if you search, the number of papers being published and number citations in most papers, it seems to be increasing exponentially and just sort of entering easy terms like Mechanical Turk, addiction
and so on. Here’s this paper that was the eventual result of me getting interested in gorilla experiment builder. And so I was interested in loss aversion. And looking at it, and cigarette smokers. And so just to give a little bit of background, as we are all pretty familiar with from all the talks today, behavioural economics is just integrating psychology into study of choice and decision making that people make. And it’s, it’s been pretty obvious for a long time for many, many people that real life behaviour is not conforming to economic predictions.
And so one example of this this was just described in the previous talk is that potential losses have a larger impact on our choices that potential gains that are otherwise equivalent. And this is what I call losses or what’s called loss aversion loss averse behaviour, we behave as if the losses are having a larger effect on our behaviour, they are on our on our valuation than gains.
And you might be able to think about loss aversion as a potential protective factor against the losses that inevitably inevitably happen in relation to our health from engaging in behaviours such as substance abuse, and there are some in person studies that actually suggest this and so with people who are drinking in excess or using cocaine problems who have problems with these types of behaviours, standard measures of loss aversion have found lower levels of loss aversion among these groups in comparisons to matched control groups are groups of people who are otherwise matched on socio demographic variables such as educational attainment, gender.
They’re showing, in comparison, lower levels of loss aversion, meaning that they are behaving as if potential losses are having less effect or a similar amount of effect on their behaviour as potential gains. And so, in addition to loss aversion, or other important decision making factors is one of them that’s particularly well studied in substance abuse research is delay discounting or the devaluation of rewards with the delay to their feature receipt.
And so it’s been documented since in the 1990s, that individuals who are using heroin or smoking cigarettes or using cocaine, so on and so forth, lots of unhealthy behaviours have steeper or higher delay discounting of the future rewards associated with them in comparison to people who are otherwise matched, but are not using these substances.
So the study that I did was was that after looking at the research on loss aversion that was out there, it was comparison, comparatively less developed research on delay discounting. Loss aversion studies, substance abuse disorders, are not had not accounted for delayed discounting, you know, because, you know, people are, these are, you didn’t know whether these factors are accounting for one another, or separate from one another, are going on independently in influencing behaviour. And none of the studies on loss aversion had examined cigarette smoking, which we know is highly comorbid with these other substance use problems, but had not been examined by itself when it is, of course, related to hundreds and hundreds of 1000s of deaths every year. And so it’s very important to understand cigarettes.
So we do this study, using real experiment building. So we set this up in a pretty straightforward way. We had some basic demographic and health questions people accepted the study on mechanical Turk. We did not tell them that it was about smoking we told them that it was about general health and choices. And so we had asked them questions about cigarette smoking, but also about drinking about drug use about whether they sleep well at night, whether they have problems with being depressed. We didn’t make it clear to them up front that this would be about smoking but separated them based on their answer to the smoking question.
And then after doing that, they completed tasks we had a simple mixed gamble task, which was a hypothetical coin flip between a potential loss or potential gain. It was not consecrated meaning that they didn’t actually get shown whether they won the gain amount or loss amount. It was just would you accept this gamble as a yes or no question.
And then we also use the standard measure to measure delay discounting this monetary choice questionnaire, which has been studied in many different settings, and many, many studies in the past. So we’re able to measure both of these factors. And we included delay discounting, because we know already that smokers have steeper delayed discounting than non smokers or never smokers, that’s well established. So this provided a positive control to tell us that we’re actually getting people who are cigarette smokers.
And then, we targeted to get 200 people in each group, Mechanical Turk, the two groups were people who are currently smoking cigarettes, or people who had never smoked cigarettes, as defined as having smoked less than 100 cigarettes in their lifetime. And they’re not currently smoking or using other tobacco products. And our current cigarette smokers were required to say that they’re also not using currently, tobacco products other than cigarettes.
Okay, and then we, we attempted to stratify the groups on gender and educational attainment. And then we included standard bot checks taken from the sample materials on Gorillas website. And we had a sort of information sheet that had to be checked that if they didn’t check it, they weren’t able to move forward in the study. And so this is another sort of point where people can be selected for.
So we use Gorilla experiment builder to do this. The basic design of the study was that we had factors right so smoking status, currently smoking, never smoked and got a task order, whether they got the delay discounting first or loss aversion first. And then we had two versions of the loss aversion task at that point to really get into but we had people complete different conditions of it, in order to provide a more rigorous measure of their loss averse behaviour, or lack thereof.
And so what this actually ended up looking like, I’m not gonna include the actual picture, and it’s even more complicated, something like this, if you want to see the actual experimental experiment on gorilla, you can go to this QR code and all these materials are available. For free, freely available on open materials, all the tasks and experiment design are available for anybody to look at.
So we had our initial questionnaires. And based on that there is separated into groups based on smoking status. And then we have three levels of educational attainment, high school or lower, some college, or college graduates. And then we had three levels of gender, male, female or other identifying. So you can see sort of complex the complexity increases. And to the point where we’ll just skip to it that we had 56 quotas over I’m sort of proud of that, because it seems like a lot, but it actually was actually it was very neatly organised and easy to work with. So it’s very cool. And out of this, we got data. So we screened lots and lots of people excluded lots and lots of people
based on our quotas, requirements, but we’re able to keep track of all that pretty easily. And we eventually got pretty close to meeting our goals in terms of the size of this groups, for smokers and never smokers.
So here’s a sort of sample demographic table on this side. We tried our best to match on gender and educational attainment. We didn’t quite get there because the prevalence of people who who report that they have a high school diploma or less on Mechanical Turk is very low. So it’s very difficult to find people who have a low level of educational attainment on Mechanical Turk, just sort of a quirk of the platform. But anyways, we got pretty close we included these variables in our analyses.
Anyways, So here are the actual results. So I’m showing here on the left the screenshots of what’s somebody would get these two different tasks. So it’s, it’s very simple. And the data are showing here that people who on this gambling task people who reported that they’re currently smoking are accepting more gambles. And that’s our simple measure of loss, of behaviour. They’re less loss averse. They’re accepting more risky gamble’s that are giving them potential for worse outcomes on their, on their gambles, so.
Never smokeds, they are when the gain the potential gain amount is twice as much potential loss and now they’re accepting about exactly half those gambles that’s consistent with loss aversion. So they’re perfectly loss averse in the never smoked when current smoking people are less loss averse. We find also in our control measures like discounting, we are seeing that we have steeper delayed discounting and higher discounting rate in the currently smoking individuals person, versus those are not or had never smoked. So this was our main result of the study.
And so what have I done since this, we’ve replicated this finding using the same sort of approach, experiment. Design, we’re including more detail on substance use other risk factors, looking into alcohol use and drug use in more detail. We’re looking at other health relevant behaviours such as sedentary lifestyle, asking questions about people’s physical activity and looking at behavioural economic risk factors and those whether those increased risk for being sedentary. Refining our methods, including other sorts of tools, such as IP hub, Qualtrics screening tools, and I’m just attempting to keep up with realising features that are coming out and trying to train other people that I work with, on how to use these tools. And going forward.
We’ve heard a lot about gamification early on, in these sessions. And I think gamification is not just for kids, I think we should definitely use it to include participant engagement, next types of studies, and just trying to integrate more of the tools and keep up with what’s going on.
And I’m in particularly just want to point out that going forward, I’m very much interested in longitudinal measurement with this sorts of thing, keeping in touch with people, because that’s also going to increase the quality of responses. You know, if you have somebody that’s coming back time and time again, to complete these questionnaires, tasks, then it’s pretty likely that their quality participants, not being a robot. So with that, I want to thank everybody who’s involved in the studies, particularly Jo, for inviting me and funding if you want to get in touch with me here’s my email address, also get in touch with me on Twitter. Thank you very much.
Jo Evershed 18:21
Eric, that was fantastic. Thank you. I think you’re absolutely right, that gamification is a way that we can build as each researcher could build themselves a library of participants that’s interested in their in their research and bring them again and again, to come back in a way that doesn’t feel so horribly demanding. Because you know, it. We need to find our tribe, both in research, but also in terms of the participants who wants to take part in our research and contribute to it. I mean, I’m sure we also need separate samples occasionally to validate it just to make sure that’s not a weird sample. But I think there’s definitely place for that there. Have you done any longitudinal research already or not?
Eric Thrailkill 19:04
Not yet, I’m sort of getting ready to do to launch study on that pretty soon. But just starting very simply, 2 measurements.
Jo Evershed 19:16
So I did have another question, because I maybe you said it in your talk. Is it that people with everybody else while I’m asking this question, if you’ve got a question for Eric, there was a highly technical one, but others welcome as well, in the q&a, please now. Do you? Is it sort of asking about causality here? Is it that people with risky decision making are more likely to smoke or is it that smokers are riskier decision making? Do we know which order that happens in and is that a question? And if not, how could we answer it?
Eric Thrailkill 19:48
So that’s one of the things I’m trying to pursue and so there are two ways or a couple of ways to do it. There’s probably more than what I’m going to mention and and the first is to start with people who’ve never had, who never had any exposure to smoking or substance use or such as, as adolescents and follow them, as they sort of develop into, you know, trying certain substances and, you know, keep track of whether they’re, you know, decision making, predicts that those transitions or predicts their their likelihood of sampling substances.
Another way to approach this experimentally, see what if you whether you have a sort of method for changing decision making, whether that translates to an immediate change in their sort of substance use behaviour. Those, again, in in the delay discounting area, those have been both pursued. And it seems like if you can influence people’s episodic future thinking, you know, they’re thinking about the future self, and things they would like to do. And you have them doing that, while they have an opportunity to smoke, those sorts of cues can result in less smoking.
And so that’s been done laboratory studies, people are trying to develop that for more sort of application in real life, to present cues to people in real life to get them to when they’re about to smoke with very high levels of craving sweets of things. It’s less well developed with loss aversion there as well, but seems like you could picture some ways to bring people’s attention to the potential losses that could be there are going to happen to them if they continue behaving a certain way, such as family, friends, and those sorts of things, longevity, that might help them inhibit certain behaviours in the moment, but it’s just less well developed.
And so one thing that early on was shown with delay discounting and smokers that’s a former smokers discount future works in the same way as never smokers. And so it seems like quitting might be related to sort of repair. And these sorts of decision making areas we’ve seen recently. It’s not published yet but the same sort of things going on with loss aversion, sort of former smokers or, or loss averse in the same way as never smokers are.
Jo Evershed 22:59
Fantastic. That was a very comprehensive answer, which was fascinating. There are two questions in the chat in the q&a for you, which I’m going to leave you to answer independently. Because we in the interest of time, we now need to move on to Casey Roark. Eric, thank you so much for joining us here today. Please stay for the rest of the session.