A Case Study of Repro­ducible Addic­tion Sci­ence Online


By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

Eric Thrailkill — Uni­ver­si­ty of Vermont


Crowd­sourced par­tic­i­pant sam­pling offers addic­tion sci­ence a com­pli­men­ta­ry approach to lab-based stud­ies, clin­i­cal tri­als, and epi­demi­o­log­i­cal data. In this talk, I will describe a recent study that used the Goril­la Exper­i­ment Builder plat­form to con­duct an exper­i­ment exam­in­ing risk fac­tors for cig­a­rette smok­ing and oth­er sub­stance use. I will focus on what I learned along the way; the fea­tures I used for the project, the chal­lenges that came up, and how the design func­tions allowed me to cre­ate and car­ry out a rig­or­ous design that could be repro­duced eas­i­ly. I will fur­ther describe how this approach con­tin­ues to facil­i­tate projects with in-per­son and online sam­ples in my laboratory.

Full Tran­script:

Eric Thrailkill 0:00
I have a laser point, great. So yes, thank you so much for the invi­ta­tion to talk about some of my work here. I will go through sort of like a sto­ry I tried to put togeth­er about it. So that’s gonna go some­what like how I found Goril­la exper­i­ment builder, which is sort of changed my research life. And, and then how just just doing a lit­tle back­ground on online research on sub­stance abuse risk fac­tors, which is what I’ve been doing.

And then pro­vide a case of that dri­ve look­ing at indi­vid­ual dif­fer­ences in loss aver­sion, and risk for cig­a­rette smok­ing and oth­er sub­stance abuse prob­lems. And then talk about some cur­rent direc­tions, I’m tak­ing this research and con­sid­er­a­tions and sort of ways I want to devel­op things. So thank you very much. I’ll proceed.

So dur­ing the pan­dem­ic, I was spend­ing a lot of time, well, ear­li­er in the pan­dem­ic, I should say, I was spend­ing a lot of time try­ing to fig­ure out how to do research online. I’ve been doing that a lit­tle bit before that, but the pan­dem­ic kick start­ed that. And so I was doing all the pro­gram­ming stuff and that sort of thing and try­ing to fig­ure every­thing out. And even­tu­al­ly, some­how, some­way, I came across this pod­cast online. And it was a webi­nar, actu­al­ly, by Joe Devlin, who was putting on a series of inter­views with aca­d­e­mics in psy­chol­o­gy, neu­ro­science, behav­iour­al sci­ences at Uni­ver­si­ty Col­lege Lon­don, on dif­fer­ent top­ics, and one of the top­ics was Goril­la exper­i­ment builder. So this is what intro­duced me to this tool, which I even­tu­al­ly got to use.

And I’ve been using ever since. And so in, in the venue I’ve been using, it is, is try­ing to study risk fac­tors asso­ci­at­ed with sub­stance abuse. And since well, before the pan­dem­ic, peo­ple in the sub­stance abuse research field have been inter­est­ed in online research. And so this is a paper, which seems kind of old now. But it came out 2019 pre pan­dem­ic, of course, on the use of crowd­sourc­ing addic­tion sci­ence, and what this paper goes over are, are a num­ber of areas where online research has been par­tic­u­lar­ly help­ful for set­ting sub­stance abuse prob­lems and peo­ple and so it goes over the the use of crowd sourc­ing to to get gath­er large, large sam­ples to basi­cal­ly repli­cate what we’ve seen in lab­o­ra­to­ry stud­ies, look­ing at case con­trol type designs, folks who are using sub­stances ver­sus folks who do not use sub­stances and repli­cat­ing those dif­fer­ent scales are a good place to start.

But oth­er users have entered devel­op more inter­ven­tion tools, and pilot test tools before using them in per­son with clin­i­cal sam­ples, and also devel­op new mea­sures and val­i­date them rapid­ly. And then they also are able to con­tact peo­ple over and over and not have them come into the lab­o­ra­to­ry for lon­gi­tu­di­nal mea­sure mea­sure­ment, which is extreme­ly use­ful. So what this paper end­ed up con­clud­ing was that online addic­tion research is going to be com­ple­men­tary to clin­i­cal tri­als, human lab­o­ra­to­ry stud­ies, epi­demi­o­log­i­cal stud­ies, and it’s going to help over­all ben­e­fit the field by improv­ing repro­ducibil­i­ty, rigour and expand­ing pos­si­bil­i­ties, the study fac­tors relat­ed to sub­stance sub­stance abuse, and over­all health relat­ed behaviours.

And so with that sort of back­ground, I was inter­est­ed in in using this and so this is just a fig­ure show­ing that if you search, the num­ber of papers being pub­lished and num­ber cita­tions in most papers, it seems to be increas­ing expo­nen­tial­ly and just sort of enter­ing easy terms like Mechan­i­cal Turk, addiction

and so on. Here’s this paper that was the even­tu­al result of me get­ting inter­est­ed in goril­la exper­i­ment builder. And so I was inter­est­ed in loss aver­sion. And look­ing at it, and cig­a­rette smok­ers. And so just to give a lit­tle bit of back­ground, as we are all pret­ty famil­iar with from all the talks today, behav­iour­al eco­nom­ics is just inte­grat­ing psy­chol­o­gy into study of choice and deci­sion mak­ing that peo­ple make. And it’s, it’s been pret­ty obvi­ous for a long time for many, many peo­ple that real life behav­iour is not con­form­ing to eco­nom­ic predictions.

And so one exam­ple of this this was just described in the pre­vi­ous talk is that poten­tial loss­es have a larg­er impact on our choic­es that poten­tial gains that are oth­er­wise equiv­a­lent. And this is what I call loss­es or what’s called loss aver­sion loss averse behav­iour, we behave as if the loss­es are hav­ing a larg­er effect on our behav­iour, they are on our on our val­u­a­tion than gains.

And you might be able to think about loss aver­sion as a poten­tial pro­tec­tive fac­tor against the loss­es that inevitably inevitably hap­pen in rela­tion to our health from engag­ing in behav­iours such as sub­stance abuse, and there are some in per­son stud­ies that actu­al­ly sug­gest this and so with peo­ple who are drink­ing in excess or using cocaine prob­lems who have prob­lems with these types of behav­iours, stan­dard mea­sures of loss aver­sion have found low­er lev­els of loss aver­sion among these groups in com­par­isons to matched con­trol groups are groups of peo­ple who are oth­er­wise matched on socio demo­graph­ic vari­ables such as edu­ca­tion­al attain­ment, gender.

They’re show­ing, in com­par­i­son, low­er lev­els of loss aver­sion, mean­ing that they are behav­ing as if poten­tial loss­es are hav­ing less effect or a sim­i­lar amount of effect on their behav­iour as poten­tial gains. And so, in addi­tion to loss aver­sion, or oth­er impor­tant deci­sion mak­ing fac­tors is one of them that’s par­tic­u­lar­ly well stud­ied in sub­stance abuse research is delay dis­count­ing or the deval­u­a­tion of rewards with the delay to their fea­ture receipt.

And so it’s been doc­u­ment­ed since in the 1990s, that indi­vid­u­als who are using hero­in or smok­ing cig­a­rettes or using cocaine, so on and so forth, lots of unhealthy behav­iours have steep­er or high­er delay dis­count­ing of the future rewards asso­ci­at­ed with them in com­par­i­son to peo­ple who are oth­er­wise matched, but are not using these substances.

So the study that I did was was that after look­ing at the research on loss aver­sion that was out there, it was com­par­i­son, com­par­a­tive­ly less devel­oped research on delay dis­count­ing. Loss aver­sion stud­ies, sub­stance abuse dis­or­ders, are not had not account­ed for delayed dis­count­ing, you know, because, you know, peo­ple are, these are, you did­n’t know whether these fac­tors are account­ing for one anoth­er, or sep­a­rate from one anoth­er, are going on inde­pen­dent­ly in influ­enc­ing behav­iour. And none of the stud­ies on loss aver­sion had exam­ined cig­a­rette smok­ing, which we know is high­ly comor­bid with these oth­er sub­stance use prob­lems, but had not been exam­ined by itself when it is, of course, relat­ed to hun­dreds and hun­dreds of 1000s of deaths every year. And so it’s very impor­tant to under­stand cigarettes.

So we do this study, using real exper­i­ment build­ing. So we set this up in a pret­ty straight­for­ward way. We had some basic demo­graph­ic and health ques­tions peo­ple accept­ed the study on mechan­i­cal Turk. We did not tell them that it was about smok­ing we told them that it was about gen­er­al health and choic­es. And so we had asked them ques­tions about cig­a­rette smok­ing, but also about drink­ing about drug use about whether they sleep well at night, whether they have prob­lems with being depressed. We did­n’t make it clear to them up front that this would be about smok­ing but sep­a­rat­ed them based on their answer to the smok­ing question.

And then after doing that, they com­plet­ed tasks we had a sim­ple mixed gam­ble task, which was a hypo­thet­i­cal coin flip between a poten­tial loss or poten­tial gain. It was not con­se­crat­ed mean­ing that they did­n’t actu­al­ly get shown whether they won the gain amount or loss amount. It was just would you accept this gam­ble as a yes or no question.

And then we also use the stan­dard mea­sure to mea­sure delay dis­count­ing this mon­e­tary choice ques­tion­naire, which has been stud­ied in many dif­fer­ent set­tings, and many, many stud­ies in the past. So we’re able to mea­sure both of these fac­tors. And we includ­ed delay dis­count­ing, because we know already that smok­ers have steep­er delayed dis­count­ing than non smok­ers or nev­er smok­ers, that’s well estab­lished. So this pro­vid­ed a pos­i­tive con­trol to tell us that we’re actu­al­ly get­ting peo­ple who are cig­a­rette smokers.

And then, we tar­get­ed to get 200 peo­ple in each group, Mechan­i­cal Turk, the two groups were peo­ple who are cur­rent­ly smok­ing cig­a­rettes, or peo­ple who had nev­er smoked cig­a­rettes, as defined as hav­ing smoked less than 100 cig­a­rettes in their life­time. And they’re not cur­rent­ly smok­ing or using oth­er tobac­co prod­ucts. And our cur­rent cig­a­rette smok­ers were required to say that they’re also not using cur­rent­ly, tobac­co prod­ucts oth­er than cigarettes.

Okay, and then we, we attempt­ed to strat­i­fy the groups on gen­der and edu­ca­tion­al attain­ment. And then we includ­ed stan­dard bot checks tak­en from the sam­ple mate­ri­als on Goril­las web­site. And we had a sort of infor­ma­tion sheet that had to be checked that if they did­n’t check it, they weren’t able to move for­ward in the study. And so this is anoth­er sort of point where peo­ple can be select­ed for.

So we use Goril­la exper­i­ment builder to do this. The basic design of the study was that we had fac­tors right so smok­ing sta­tus, cur­rent­ly smok­ing, nev­er smoked and got a task order, whether they got the delay dis­count­ing first or loss aver­sion first. And then we had two ver­sions of the loss aver­sion task at that point to real­ly get into but we had peo­ple com­plete dif­fer­ent con­di­tions of it, in order to pro­vide a more rig­or­ous mea­sure of their loss averse behav­iour, or lack thereof.

And so what this actu­al­ly end­ed up look­ing like, I’m not gonna include the actu­al pic­ture, and it’s even more com­pli­cat­ed, some­thing like this, if you want to see the actu­al exper­i­men­tal exper­i­ment on goril­la, you can go to this QR code and all these mate­ri­als are avail­able. For free, freely avail­able on open mate­ri­als, all the tasks and exper­i­ment design are avail­able for any­body to look at.

So we had our ini­tial ques­tion­naires. And based on that there is sep­a­rat­ed into groups based on smok­ing sta­tus. And then we have three lev­els of edu­ca­tion­al attain­ment, high school or low­er, some col­lege, or col­lege grad­u­ates. And then we had three lev­els of gen­der, male, female or oth­er iden­ti­fy­ing. So you can see sort of com­plex the com­plex­i­ty increas­es. And to the point where we’ll just skip to it that we had 56 quo­tas over I’m sort of proud of that, because it seems like a lot, but it actu­al­ly was actu­al­ly it was very neat­ly organ­ised and easy to work with. So it’s very cool. And out of this, we got data. So we screened lots and lots of peo­ple exclud­ed lots and lots of people

based on our quo­tas, require­ments, but we’re able to keep track of all that pret­ty eas­i­ly. And we even­tu­al­ly got pret­ty close to meet­ing our goals in terms of the size of this groups, for smok­ers and nev­er smokers.

So here’s a sort of sam­ple demo­graph­ic table on this side. We tried our best to match on gen­der and edu­ca­tion­al attain­ment. We did­n’t quite get there because the preva­lence of peo­ple who who report that they have a high school diplo­ma or less on Mechan­i­cal Turk is very low. So it’s very dif­fi­cult to find peo­ple who have a low lev­el of edu­ca­tion­al attain­ment on Mechan­i­cal Turk, just sort of a quirk of the plat­form. But any­ways, we got pret­ty close we includ­ed these vari­ables in our analyses.

Any­ways, So here are the actu­al results. So I’m show­ing here on the left the screen­shots of what’s some­body would get these two dif­fer­ent tasks. So it’s, it’s very sim­ple. And the data are show­ing here that peo­ple who on this gam­bling task peo­ple who report­ed that they’re cur­rent­ly smok­ing are accept­ing more gam­bles. And that’s our sim­ple mea­sure of loss, of behav­iour. They’re less loss averse. They’re accept­ing more risky gam­ble’s that are giv­ing them poten­tial for worse out­comes on their, on their gam­bles, so.

Nev­er smokeds, they are when the gain the poten­tial gain amount is twice as much poten­tial loss and now they’re accept­ing about exact­ly half those gam­bles that’s con­sis­tent with loss aver­sion. So they’re per­fect­ly loss averse in the nev­er smoked when cur­rent smok­ing peo­ple are less loss averse. We find also in our con­trol mea­sures like dis­count­ing, we are see­ing that we have steep­er delayed dis­count­ing and high­er dis­count­ing rate in the cur­rent­ly smok­ing indi­vid­u­als per­son, ver­sus those are not or had nev­er smoked. So this was our main result of the study.

And so what have I done since this, we’ve repli­cat­ed this find­ing using the same sort of approach, exper­i­ment. Design, we’re includ­ing more detail on sub­stance use oth­er risk fac­tors, look­ing into alco­hol use and drug use in more detail. We’re look­ing at oth­er health rel­e­vant behav­iours such as seden­tary lifestyle, ask­ing ques­tions about peo­ple’s phys­i­cal activ­i­ty and look­ing at behav­iour­al eco­nom­ic risk fac­tors and those whether those increased risk for being seden­tary. Refin­ing our meth­ods, includ­ing oth­er sorts of tools, such as IP hub, Qualtrics screen­ing tools, and I’m just attempt­ing to keep up with real­is­ing fea­tures that are com­ing out and try­ing to train oth­er peo­ple that I work with, on how to use these tools. And going forward.

We’ve heard a lot about gam­i­fi­ca­tion ear­ly on, in these ses­sions. And I think gam­i­fi­ca­tion is not just for kids, I think we should def­i­nite­ly use it to include par­tic­i­pant engage­ment, next types of stud­ies, and just try­ing to inte­grate more of the tools and keep up with what’s going on.

And I’m in par­tic­u­lar­ly just want to point out that going for­ward, I’m very much inter­est­ed in lon­gi­tu­di­nal mea­sure­ment with this sorts of thing, keep­ing in touch with peo­ple, because that’s also going to increase the qual­i­ty of respons­es. You know, if you have some­body that’s com­ing back time and time again, to com­plete these ques­tion­naires, tasks, then it’s pret­ty like­ly that their qual­i­ty par­tic­i­pants, not being a robot. So with that, I want to thank every­body who’s involved in the stud­ies, par­tic­u­lar­ly Jo, for invit­ing me and fund­ing if you want to get in touch with me here’s my email address, also get in touch with me on Twit­ter. Thank you very much.

Jo Ever­shed 18:21
Eric, that was fan­tas­tic. Thank you. I think you’re absolute­ly right, that gam­i­fi­ca­tion is a way that we can build as each researcher could build them­selves a library of par­tic­i­pants that’s inter­est­ed in their in their research and bring them again and again, to come back in a way that does­n’t feel so hor­ri­bly demand­ing. Because you know, it. We need to find our tribe, both in research, but also in terms of the par­tic­i­pants who wants to take part in our research and con­tribute to it. I mean, I’m sure we also need sep­a­rate sam­ples occa­sion­al­ly to val­i­date it just to make sure that’s not a weird sam­ple. But I think there’s def­i­nite­ly place for that there. Have you done any lon­gi­tu­di­nal research already or not?

Eric Thrailkill 19:04
Not yet, I’m sort of get­ting ready to do to launch study on that pret­ty soon. But just start­ing very sim­ply, 2 measurements.

Jo Ever­shed 19:16
So I did have anoth­er ques­tion, because I maybe you said it in your talk. Is it that peo­ple with every­body else while I’m ask­ing this ques­tion, if you’ve got a ques­tion for Eric, there was a high­ly tech­ni­cal one, but oth­ers wel­come as well, in the q&a, please now. Do you? Is it sort of ask­ing about causal­i­ty here? Is it that peo­ple with risky deci­sion mak­ing are more like­ly to smoke or is it that smok­ers are riski­er deci­sion mak­ing? Do we know which order that hap­pens in and is that a ques­tion? And if not, how could we answer it?

Eric Thrailkill 19:48
So that’s one of the things I’m try­ing to pur­sue and so there are two ways or a cou­ple of ways to do it. There’s prob­a­bly more than what I’m going to men­tion and and the first is to start with peo­ple who’ve nev­er had, who nev­er had any expo­sure to smok­ing or sub­stance use or such as, as ado­les­cents and fol­low them, as they sort of devel­op into, you know, try­ing cer­tain sub­stances and, you know, keep track of whether they’re, you know, deci­sion mak­ing, pre­dicts that those tran­si­tions or pre­dicts their their like­li­hood of sam­pling substances.

Anoth­er way to approach this exper­i­men­tal­ly, see what if you whether you have a sort of method for chang­ing deci­sion mak­ing, whether that trans­lates to an imme­di­ate change in their sort of sub­stance use behav­iour. Those, again, in in the delay dis­count­ing area, those have been both pur­sued. And it seems like if you can influ­ence peo­ple’s episod­ic future think­ing, you know, they’re think­ing about the future self, and things they would like to do. And you have them doing that, while they have an oppor­tu­ni­ty to smoke, those sorts of cues can result in less smoking.

And so that’s been done lab­o­ra­to­ry stud­ies, peo­ple are try­ing to devel­op that for more sort of appli­ca­tion in real life, to present cues to peo­ple in real life to get them to when they’re about to smoke with very high lev­els of crav­ing sweets of things. It’s less well devel­oped with loss aver­sion there as well, but seems like you could pic­ture some ways to bring peo­ple’s atten­tion to the poten­tial loss­es that could be there are going to hap­pen to them if they con­tin­ue behav­ing a cer­tain way, such as fam­i­ly, friends, and those sorts of things, longevi­ty, that might help them inhib­it cer­tain behav­iours in the moment, but it’s just less well developed.

And so one thing that ear­ly on was shown with delay dis­count­ing and smok­ers that’s a for­mer smok­ers dis­count future works in the same way as nev­er smok­ers. And so it seems like quit­ting might be relat­ed to sort of repair. And these sorts of deci­sion mak­ing areas we’ve seen recent­ly. It’s not pub­lished yet but the same sort of things going on with loss aver­sion, sort of for­mer smok­ers or, or loss averse in the same way as nev­er smok­ers are.

Jo Ever­shed 22:59
Fan­tas­tic. That was a very com­pre­hen­sive answer, which was fas­ci­nat­ing. There are two ques­tions in the chat in the q&a for you, which I’m going to leave you to answer inde­pen­dent­ly. Because we in the inter­est of time, we now need to move on to Casey Roark. Eric, thank you so much for join­ing us here today. Please stay for the rest of the session.

Get on the Registration List

BeOnline is the conference to learn all about online behavioral research. It's the ideal place to discover the challenges and benefits of online research and to learn from pioneers. If that sounds interesting to you, then click the button below to register for the 2023 conference on Thursday July 6th. You will be the first to know when we release new content and timings for BeOnline 2023.

With thanks to our sponsors!