Get­ting Research Online

Dr Joshua Bal­sters, Goril­la
@DrBalsters

YouTube

By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

Full Tran­script:

Dr. Joshua Bal­sters:
Okay. Thank you very much, Joe. Thanks for get­ting me up here. So, can you write in the chat, just tell me you can see the screen? Type, “See screen” into the chat for me, just so I know that you can see my slides. I’m not going to be just talk­ing to air. All right, bril­liant. Thank you, every­one. That’s great. So, thank you for com­ing to join us today. My name is Joshua Bal­sters. It feels like a long time ago now. I used to be a lec­tur­er in the Psy­chol­o­gy Depart­ment at Roy­al Holloway.

Dr. Joshua Bal­sters:
Well, I guess before that, I was just going from post­doc to post­doc. I spent quite a few years doing almost every kind of research real­ly. I’ve done atten­tion stud­ies, mem­o­ry, role-based learn­ing, social deci­sion mak­ing, val­ue-based deci­sion mak­ing, some brain imag­ing stuff. I think that’s why Jo want­ed me to come online to talk a bit about this, try­ing to con­dense all of that research expe­ri­ence to give you some hints and tips about tak­ing work online. Some of the things that we’ve seen, espe­cial­ly over the last year with Goril­la users, what are some of the best ways to take your research online?

Dr. Joshua Bal­sters:
So essen­tial­ly, we’re going to start off just by intro­duc­ing this ques­tion, why should you go online? What are the ben­e­fits to you? We’re going to talk a lit­tle bit about the chal­lenges and some of the rebut­tals, the things that pesky review or two is going to prob­a­bly say to you. We’re going to talk about some of the prac­ti­cal con­sid­er­a­tions, some of the real­ly use­ful tips and tricks that we’ve been pick­ing up at Goril­la and things we’ve picked up in oth­er places to try and help you out and help you cre­ate the best online exper­i­ment pos­si­ble. I’m going to go through some inspi­ra­tional sto­ries as well, some real­ly bril­liant exam­ples of some of the kinds of just astound­ing research that you can take and you can take online.

Dr. Joshua Bal­sters:
So, I think in three sim­ple words, why should you take your research online? Speed, scale, and reach. I think these are real­ly the three pil­lars of online research. The speed with which you can col­lect data when you’re work­ing online, I still find this absolute­ly fas­ci­nat­ing. It’s just incred­i­ble think­ing about the months I used to spend doing non-online research, doing lab-based research. In fact, now with online research, I grab a sand­wich. By the time I get back, my exper­i­men­t’s done. I think it’s absolute­ly incred­i­ble the speed that you can col­lect data with online. The impor­tant thing about that speed and it’s something

Dr. Joshua Bal­sters:
I think for every­one, no mat­ter what stage in your career you are, whether you decide to stay in acad­e­mia or leave, that speed allows you to buy back your time. That’s prob­a­bly one of the most impor­tant assets we have, our time. So, in my per­son­al opin­ion, you don’t real­ly learn that much after hav­ing 10 par­tic­i­pants through the lab and you’ve prob­a­bly refined your spiel, you’re bet­ter at explain­ing your exper­i­ments to people.

Dr. Joshua Bal­sters:
After that, you’re not real­ly gain­ing any­thing in terms of skills. It’s just time that’s being con­sumed. Where­as with online research, you’re buy­ing back that time, because it is just done for you in a mat­ter of hours rather than months. That means you’ve got free time on your hands and you can decide what to do. You can decide whether it’s bet­ter to learn a new analy­sis type skill, read around your sub­ject more, get around to writ­ing your paper, or just have a glass of wine and watch some­thing nice on Net­flix and take a bit of time for you. Buy­ing back your time is prob­a­bly the most impor­tant thing, I think, about mov­ing your research online.

Dr. Joshua Bal­sters:
But on top of that, we also have the scale of online research. With lab-based research, there’s this lin­ear rela­tion­ship. If I want twice as many sub­jects, it takes twice as long to col­lect. You don’t have that with online research. If I want to go 2 times, 3 times, 10 times as many par­tic­i­pants, it’s usu­al­ly a small frac­tion of time that you are adding on to your data col­lec­tion, rather than dou­bling, tripling, expo­nen­tial­ly increas­ing your time to col­lect data. With that, you also get rich. Rich works in two ways, which is quite inter­est­ing. The first of all, it allows you to find par­tic­i­pants who you would­n’t nor­mal­ly get to work with. I found this when I was in Ire­land, I did a lot of aging research.

Dr. Joshua Bal­sters:
The elder­ly indi­vid­u­als were absolute­ly won­der­ful, but I think after a while you real­ize the elder­ly indi­vid­u­als who are able to make the efforts to come into a lab­o­ra­to­ry and do research with you, these aren’t per­haps the typ­i­cal elder­ly indi­vid­u­als. You’re miss­ing out, for exam­ple, on lots of peo­ple who might have dif­fer­ent kinds of phys­i­cal pain, hip prob­lems, knee prob­lems. These peo­ple can’t come into the lab and see you. You’re miss­ing out on research­ing with them. So, online research means that you can send research into peo­ple’s homes. So, it means you get a more rep­re­sen­ta­tive sam­ple and it means in line with the idea of scale. You’re increas­ing the num­ber of par­tic­i­pants in what is some­times quite dif­fi­cult to acquire populations.

Dr. Joshua Bal­sters:
This is great, obvi­ous­ly, in terms of repro­ducibil­i­ty. The BeOnline Con­fer­ence, well, this con­fer­ence obvi­ous­ly, we’ve got a great ses­sion tomor­row after­noon, wrap­ping every­thing up with Jo and a num­ber of oth­er won­der­ful speak­ers to talk about Repro­ducibil­i­ty 2.0. The oth­er thing as well is we have, just before that, Jonathan Tsay. He’s going to be doing a sym­po­sium about motor con­trol online. In pre­vi­ous chats, he’s told us that the abil­i­ty to take his work online has opened up a whole new par­tic­i­pant pool and we’re increas­ing the sam­ple sizes in that par­tic­i­pant pool. So, I think these are some of the key ben­e­fits for online research. Online research was on the rise any­way pre-pandemic.

Dr. Joshua Bal­sters:
From what we hear from our clients, it’s been the push they need­ed to get online and start doing online research. Most like­ly, what’s going to hap­pen is that online research is here to stay. What we’re hear­ing is that most peo­ple are going to do a com­bi­na­tion of lab-based and online stud­ies now on. So, it’s impor­tant that you get famil­iar with online research as it becomes increas­ing­ly com­mon­place for all the rea­sons that I’ve just mentioned.

Dr. Joshua Bal­sters:
For all the super­vi­sors out there, this quote means a lot to me. Being able to test online means my research does­n’t grind to a halt. This was espe­cial­ly impor­tant for my PhD stu­dents, I did not want their research and men­tal health suf­fer, because they could­n’t car­ry out their stud­ies. Now, I was in the sit­u­a­tion with my own PhD stu­dent. She had a mag­nif­i­cent MRI study. The day she was about to start test­ing, we had to shut the whole MRI scan­ner down.

Dr. Joshua Bal­sters:
I think this hap­pened to a lot of peo­ple. I’m sure there’s a lot of sym­pa­thy out there for peo­ple who’ve been in this sit­u­a­tion. Online research, I think, it’s going to help peo­ple, like I said, buy back their time and it’s going to increase the well-being of a lot of peo­ple doing research. For me, that’s real­ly the most impor­tant thing, buy­ing back your time with online research.

Dr. Joshua Bal­sters:
I’m going to jump now and talk a lit­tle bit about the chal­lenges of online research, the cri­tiques that you often hear about online research. This is David Ogilvy who you might not have heard of. Appar­ent­ly, he was the inspi­ra­tion for Don Drap­er in Mad Men. He was quite a famous mar­ket researcher. He has this very famous quote that “The trou­ble with mar­ket research is that peo­ple don’t think how they feel, they don’t say what they think, and they don’t do what they say.” I think this is a great quote for all behav­ioral sci­ence, to be hon­est, but I think a lot of peo­ple have always con­sid­ered online research to just be sur­vey based.

Dr. Joshua Bal­sters:
This is their cri­tique. Well, I need to do some­thing that gets at some of the implic­it behav­ioral para­me­ters, things that go beyond what peo­ple say, go beyond your tra­di­tion­al sur­vey-based research. How can I do that online? Well, now we know there are increas­ing­ly a num­ber of plat­forms that are mak­ing that avail­able and allow­ing you to cre­ate absolute­ly won­der­ful exper­i­ments online, which are incred­i­bly com­plex and sophis­ti­cat­ed. I’m going to talk about just a cou­ple of these at the end of this talk.

Dr. Joshua Bal­sters:
Now, yeah, once we move online, you might have to think, “Oh, God, do I need to learn anoth­er pro­gram­ming lan­guage?” If you were like me, I trained in MATLAB, which is becom­ing pret­ty much a defunct lan­guage. I start­ed learn­ing Python and R. And then online research came and I just thought, “Oh, my God. Now I need to learn HTML and JavaScript. Real­ly?” Basi­cal­ly, you don’t have to any­more. There are a lot of amaz­ing tools out there, which allow you to do a lot of things just using a stan­dard graph­i­cal user inter­face, nice drag and drop interface.

Dr. Joshua Bal­sters:
In fact, if you want to find out more about that, I wrote a blog piece for Goril­la a cou­ple weeks ago. It was a very cathar­tic blog piece, I have to say. I got a lot of demons out talk­ing about those evil red lines of code. You can have a read of that. Essen­tial­ly, the point is you don’t need to wor­ry about learn­ing a new pro­gram­ming lan­guage. You can go out there and you can cre­ate won­der­ful exper­i­ments with­out need­ing a line of code.

Dr. Joshua Bal­sters:
Now, I think the biggest chal­lenge for online research is this idea that online research is inher­ent­ly flawed and that par­tic­i­pants are going to behave bad­ly. They’re not going to give you the type of data qual­i­ty that you get in the lab. I would chal­lenge this asser­tion. I don’t think it’s nec­es­sar­i­ly true. In fact, Jo Ever­shed wrote a real­ly nice blog piece pre­vi­ous­ly about the illu­sion of con­trol. We talk about these cri­tiques of online research, this idea that, “Oh, well, obvi­ous­ly, they are going to be just press­ing the space­bar with one hand while play­ing Fort­nite with the oth­er hand or watch­ing the Eng­land game lat­er on,” for example.

Dr. Joshua Bal­sters:
I don’t think that’s a fair crit­i­cism. I think when we think about lab-based test­ing, we paint it as this pic­ture of puri­ty, when it real­ly is not. For exam­ple, I’m sure every now and again, we’ve drift­ed off in the mid­dle of an exper­i­ment in a lab and our mind has gone off to wan­der­ing, espe­cial­ly if it’s a bit of a bor­ing exper­i­ment. Now, as well, you can see in terms of qual­i­ty, well, there are obvi­ous­ly dis­trac­tions in the lab. Peo­ple go in with their phones. They go in with their smart­watch­es on. They’ll get buzzers. There’ll be oth­er dis­trac­tions. So, the point that I would like to raise, I would, first of all, like to chal­lenge this idea that the lab is some­how the ground truth of behavior.

Dr. Joshua Bal­sters:
I think in a lot of ways you can argue it is an unre­al­is­tic sit­u­a­tion. Again, think­ing lat­er on, I think there’s going to be some chats about this in the Repro­ducibil­i­ty 2.0 ses­sion tomor­row. So, idea that lab-based test­ing may not be the ground truth we think it is. That said, I would also say that if you design good exper­i­ments, then you can get just as good data qual­i­ty out of an online study that you would get in the lab. I’m going to show you just a hand­ful of exam­ples that prove that is the case.

Dr. Joshua Bal­sters:
These are what we think about, our sev­en-step guide for suc­cess­ful­ly doing an online study. These are just some sug­ges­tions and a way of think­ing about build­ing your next online exper­i­ments and the things that you should con­sid­er. First of all, con­sid­er your study, what is the best study to get start­ed with, espe­cial­ly if you are new to online research? What’s the best study to take? I think, per­son­al­ly, I would say start with some­thing that you’re famil­iar with. It’s the best way to get start­ed on this.

Dr. Joshua Bal­sters:
Rather than tak­ing a whole brand new top­ic, as well as tak­ing on a whole brand new domain with online research. Why don’t you start say­ing, “Well, this is what I did in the lab. So, let’s try it online”? It’s always great to have these kinds of san­i­ty checks, lit­tle things that can be built into your study that will help you feel con­fi­dent about what you are doing. So, when you’re select­ing a study, I’ve rec­om­mend try­ing to do that.

Dr. Joshua Bal­sters:
The next thing we’re going to talk about is build­ing the study. What do you need to build your exper­i­ment? These are a cou­ple things to bear in mind, I think, some key things relat­ed to online research. First thing, I would say, is you need clear and con­cise instruc­tions. I can’t stress this enough. What I would say is that you’re not con­fined any more to the lab-based design, where you typ­i­cal­ly would have an A4 sheet of paper with your instruc­tions writ­ten all over it and you would say, “Oh, read this. If you have any ques­tions, let me know.” First of all, you’re not there to answer any ques­tions, but also, it’s a very bor­ing and for­mal way of doing things. As you’ve moved online, why not change essen­tial­ly? We can do things in any for­mat now.

Dr. Joshua Bal­sters:
One of the things I real­ly enjoy as part of my time at Goril­la is Goril­la Acad­e­my. One of the things I have in there is a social influ­ence task, a real fun thing that I made, where we see the influ­ence Rot­ten Toma­toes has on your behav­ior. With­in that, what I decid­ed to do, rather than hav­ing a lengthy block of text to explain par­a­digm, I just made a video of me doing the task and talk­ing through the tri­al. I think some­times that’s just the eas­i­est way of doing things. So, don’t feel con­strained by the way you used to do things in the lab. You can adapt and do lots of crazy new things with online research.

Dr. Joshua Bal­sters:
In fact, this YouTube link, which we’re going to make acces­si­ble lat­er on, this links to Simone’s research. Simone is actu­al­ly going to be speak­ing next after me, the first speak­er in the Buf­fet of Research sec­tion. So, def­i­nite­ly, you want to lis­ten to her. She’s got so many top tips, it’s not even real. Yeah, her videos are amaz­ing. She talks to how to take an incred­i­bly com­pli­cat­ed eye track­ing study. She has the most beau­ti­ful, detailed instruc­tions. That’s the kind of thing that you want to be going for.

Dr. Joshua Bal­sters:
So, next thing I want to talk about is keep it short. This is some­thing that we rec­om­mend with our online stud­ies. If you can, it’s not always pos­si­ble, but if you can, try to keep your stud­ies to 5 to 10 min­utes. That’s going to real­ly help. The oth­er thing as well is to use progress bars and this real­ly helps a lot with attri­tion. If a par­tic­i­pant can see how many tri­als along they are, then they’re going to be able to say, “Oh, I’ve only got a small sec­tion left. I should keep going.” Rather than some­body who’s done 95% of the study, drop­ping out are the last 5%. So, I think that’s going to be incred­i­bly help­ful. If you can, build in breaks.

Dr. Joshua Bal­sters:
For a lot of rea­sons with online stud­ies, if peo­ple know that they’re going to get a break every five min­utes, that’s incred­i­bly help­ful for them. So, that’s some­thing to bear in mind as well. If you can, include vig­i­lance checks. So, with Goril­la Acad­e­my, again, this was a dif­fer­ent one. This was the atten­tion study. It was a beau­ti­ful exper­i­ment by Pol­ly Dal­ton look­ing at dichot­ic lis­ten­ing and you hear the voic­es all around your head. It’s like you’re in the room, but it only works if you have head­phones in. So, if some­body just has their speak­ers on, it won’t work.

Dr. Joshua Bal­sters:
So, in order to get around that and make sure peo­ple were lis­ten­ing with their head­phones, I had a lit­tle lit­mus check that just said, “Can you hear this noise on the left or the right?” You would­n’t be able to tell from your speak­ers, but in the head­phones, it’s obvi­ous. So, have these lit­tle vig­i­lance checks built into your study. What you’ll see when I went to ana­lyze the data is that by exclud­ing indi­vid­u­als that failed the vig­i­lance check, I increased my effect size. I was able to actu­al­ly pre-reg­is­ter those exclu­sion cri­te­ria because of pilot­ing, which is what I’m actu­al­ly going to come to in a second.

Dr. Joshua Bal­sters:
Now, the next thing as well is this debrief ques­tion­naire. So, just ask peo­ple. I can’t rec­om­mend that high­ly enough. If you’ve got par­tic­i­pants doing your exper­i­ments, you can just have a form at the end that says, “Was there any­thing unusu­al? Was there any­thing you think could be explained bet­ter? Did this make sense? Did you enjoy it?” Ask peo­ple for their feed­back. It’s just so sim­ple and it will give you so much infor­ma­tion before you put your exper­i­ments online. Hav­ing a debrief ques­tion­naire is incred­i­bly help­ful, espe­cial­ly when you’re piloting.

Dr. Joshua Bal­sters:
Before I get into pilot­ing, I just want to briefly men­tion ethics. Prob­a­bly before, in fact, you’d get into all the effort of build­ing your exper­i­ments, when you’re think­ing about your online study, you need to think about the ethics and think­ing specif­i­cal­ly about the plat­form. Does your plat­form con­form to infor­ma­tion secu­ri­ty rules in your depart­ment? Does it con­form to things like GDPR? Now, these are hor­ri­ble, bor­ing things that nobody real­ly wants to have to deal with. So, if you can, find a plat­form that has that infor­ma­tion and says, “Don’t wor­ry, we’ve got this.” Your ethics com­mit­tee will prob­a­bly ask you about it. So, look into it definitely.

Dr. Joshua Bal­sters:
Now, I want to talk about pilot­ing. Emi­ly Breese gave the most won­der­ful talk at last year’s BeOnline all about pilot­ing and how cru­cial it is to do pilot­ing in your exper­i­ments. Now, there are a cou­ple rea­sons for this. I think one of those rea­sons is that typ­i­cal­ly, the par­tic­i­pants we get in psy­chol­o­gy depart­ments at uni­ver­si­ties, they’re psy­chol­o­gy stu­dents at the uni­ver­si­ty or they’re peo­ple who are expe­ri­enced par­tic­i­pants. So, these expe­ri­enced par­tic­i­pants, essen­tial­ly, they’re going to be able to fill in the gaps if you don’t have good enough instruc­tions. But when you move online, this prob­a­bly isn’t the case. So, you’re not around to answer their ques­tions. So, please make sure to pilot with peo­ple and fig­ure these things out.

Dr. Joshua Bal­sters:
Yeah, Emi­ly gives these won­der­ful exam­ples from the trail mak­ing test, fun­da­men­tal of neu­ropsy­chol­o­gy, where you have to either draw the line between one, two, three, four, five or the alter­nat­ing one, A, two, B, three, C. She talks about all the kinds of issues she had and some of the crazy answers peo­ple came up with that helped her refine her instruc­tions. Emi­ly goes through this much bet­ter than I do and she actu­al­ly has a won­der­ful frame­work in her video. So, I strong­ly rec­om­mend hav­ing a look at that, but don’t think about pilot­ing just for your instructions.

Dr. Joshua Bal­sters:
Pilot­ing is also good to look at your data out­put. Is your data out­put cor­rect? Do you have all the infor­ma­tion you need to ana­lyze your data? Because the speed of online test­ing is absolute­ly won­der­ful, but the flip side of that is that if you col­lect all your data so quick­ly, then unfor­tu­nate­ly, if some­thing’s wrong, you’ve just had to pay 100 or 500 peo­ple for their time when you’ve got an error. That’s not your par­tic­i­pan­t’s fault. That’s on you. So, make sure before you do your big reveal, your big release, that you check your data out­put is cor­rect and has every­thing you need.

Dr. Joshua Bal­sters:
Like I’ve men­tioned just before that, you can use this pilot­ing for exclu­sion cri­te­ria. So, you can fig­ure out, “Oh, any­one who seems to think this, they weren’t doing it right.” So, I can say that is an exclu­sion cri­te­ria. I can’t stress it enough. Pilot, pilot, pilot is prob­a­bly the most impor­tant thing you can do. You do need to pilot more with online test­ing than you would need to with lab-based testing.

Dr. Joshua Bal­sters:
I’ve already talked a lit­tle bit there about pre-reg­is­tra­tion. Obvi­ous­ly, pilot­ing is a great way to pre-reg­is­ter any exclu­sion cri­te­ria that you might have in your exper­i­ments. I’ve seen some­thing float­ing around Twit­ter recent­ly. I’m not sure if it’s a preprint or a pub­lished paper, but essen­tial­ly, they com­pared pre-reg­is­tered reports with non-reg­is­tered equiv­a­lents. They tend to find that in terms of mark­ers of cre­ativ­i­ty, every­thing is stable.

Dr. Joshua Bal­sters:
So, all those con­cerns before about pre-reg­is­tra­tion is going to ruin dis­cov­ery sci­ence, they don’t seem to be valid, but they did find that the sta­tis­ti­cal qual­i­ty and method­olog­i­cal rig­or of pre-reg­is­tered work was greater than the non-reg­is­tered con­trols. So, I think pre-reg­is­tra­tion is going to be real­ly impor­tant. Again, just to give anoth­er plug for that Repro­ducibil­i­ty 2.0 ses­sion, we’re going to get into all of these kinds of ques­tions and details by the end of BeOnline.

Dr. Joshua Bal­sters:
And then that’s it. It’s time to col­lect your data, time to ana­lyze your data. There are some won­der­ful advances com­ing up as well, things like our mark­down, which means you’re going to be able to cre­ate these beau­ti­ful, curat­ed scripts that are able to keep track of all your analy­sis and do all of your basi­cal­ly work with the raw data to pro­duce a beau­ti­ful, clear tran­script to the end of your results. There’s a lot of stuff com­ing through. A lit­tle sneak hint, there might be some­thing com­ing up in the future, long in the future with Goril­la and data, but that’s some­thing prob­a­bly for our next BeOnline Conference.

Dr. Joshua Bal­sters:
What are the things you want to con­sid­er in your exper­i­men­tal plat­form? Well, does my exper­i­men­tal plat­form do what I need it to do? That’s prob­a­bly the most cru­cial ques­tion. So, it’s worth lay­ing out. What do you need it to do? What do you need your exper­i­men­tal plat­form to do? Most peo­ple are think­ing prob­a­bly Task Builder. I need some way of pre­sent­ing images, respons­es, et cetera, but you prob­a­bly want to sur­vey as well. If you’re not doing sur­vey-based research, the abil­i­ty to cre­ate a demo­graph­ics form is incred­i­bly useful.

Dr. Joshua Bal­sters:
I found in the past that there’s some­times a mis­match between recruit­ment pan­els and cri­te­ria I set out and peo­ple I actu­al­ly end up with, because I think it’s not real­ly any­one’s pri­or­i­ty at the moment to update their recruit­ment pan­el set­tings and update their cur­rent life sit­u­a­tion. So, it’s worth hav­ing your own lit­tle sur­vey to work with that.

Dr. Joshua Bal­sters:
Now, then you have the Task Builder. Can the Task Builder do what I want? Does it have the fea­tures that I need? Are there spe­cial­ist tools that I need, eye track­ing or mouse track­ing? Is there a spe­cial kind of par­a­digm that I’m look­ing at? What did oth­er peo­ple use? I think these are all the impor­tant ques­tions you need to ask your­self. I think as well, do I need to do this in full code edi­tor or maybe you want to say, “Can I do this in the code edi­tor?” Do I have the free­dom to write some­thing from scratch or is this some­thing that I can get nice and eas­i­ly built using a sim­ple graph­i­cal user inter­face? These are the kinds of ques­tions you want to be ask­ing about your exper­i­ment before you get start­ed, before you choose an exper­i­men­tal platform.

Dr. Joshua Bal­sters:
The things we’re going to talk about today are a cou­ple of new tools. There’s the Game Builder. We’re going to talk about that today, which is absolute­ly incred­i­ble. It’s real­ly, I think, going to change a lot of the way we do research. We have a won­der­ful Game Builder pan­el, who are going to talk about the research they’ve been doing with games. And then we’re going to get a won­der­ful demon­stra­tion from Nick, our CTO. Tomor­row, we’ve got again, a great con­sumer deci­sion mak­ing pan­el talk­ing about Shop Builder and we’re going to get a Shop Builder demo. This is such a cool tech­nique. It’s such a cool tool for being able to do shop­ping-based research.

Dr. Joshua Bal­sters:
Anoth­er ques­tion, I think, you want to ask your­self is, “How well will I be sup­port­ed in what I’m doing? Are there good forums out there? Are there YouTube videos? What kinds of sup­port doc­u­men­ta­tion is there out there? What’s the com­mu­ni­ty like out there?” These are things you’ll want to be ask­ing your­self. One of the things I love about Goril­la and I work for Goril­la, this is what I’m famil­iar with. There are oth­er tools out there that have equal­ly great forms. I’d just say that out. But what I love about Goril­la from when I was a lec­tur­er, before I even joined them was the col­lab­o­rate tool, because it was anoth­er way of buy­ing back my time.

Dr. Joshua Bal­sters:
I was able to basi­cal­ly mon­i­tor my third year project stu­dents and sup­port them remote­ly, which meant at that time, I was able to go to a con­fer­ence in Paris, rather than spend­ing a cou­ple weeks in Egham. Egham’s love­ly, don’t get me wrong, but it’s no Paris. And then I could go to this amaz­ing con­fer­ence in Paris and I could look at my lap­top. I could check my emails and I could see if the stu­dent had an issue or prob­lem. I was able to sup­port them remote­ly. I found that col­lab­o­rate func­tion incred­i­bly use­ful. It bought me back time. Same with the send func­tion. The send func­tion is so use­ful, because in a lot of these dif­fer­ent exper­i­ment plat­forms, you don’t have to wor­ry any­more about mak­ing all the sys­tem diag­nos­tics match up. So, it’s like, “Here’s my task.”

Dr. Joshua Bal­sters:
Now, I am assum­ing that you’re run­ning a com­put­er that is Win­dows 7, ver­sion 2.93 bit. It’s 64 bits. It needs to be run­ning MATLAB 2014b. I think like a lot of peo­ple out there, I’ve got exper­i­ments that will only run in a very spe­cif­ic sit­u­a­tion that it’s a Tues­day and the sun’s just right. You don’t have to wor­ry about that stuff with lots of online plat­forms, because all the com­put­ing infra­struc­ture is behind the scenes. You don’t have to wor­ry. So, if you click “Send”, they’re going to get a work­ing ver­sion of your exper­i­ment. You don’t have to spend time help­ing them to debug your exper­i­ments and get it up and run­ning on their system.

Dr. Joshua Bal­sters:
Final­ly, I love our Open Mate­ri­als. It’s a great plat­form. It’s a great place for researchers to dis­sem­i­nate their projects and a great place to share stim­uli, which is so help­ful rather than hav­ing spent hours cre­at­ing stim­u­lus sets. This one is from Sarah Blake­more’s group. This was a vari­a­tion of the Raven’s matri­ces task. Now, that stim­u­lus set is online and any­one can take it. You don’t even need to ask per­mis­sion. They’re held in a repos­i­to­ry. You can just search for it and use it as you see fit. There are a num­ber of par­a­digms on Goril­la Open Mate­ri­als. Also, oth­er tools as well. There are lots of things out there, in JSI, in Python, for exam­ple. There are loads of tools out there, loads of exper­i­ments already out there ready for you to down­load and use, which is just so helpful.

Dr. Joshua Bal­sters:
Has your test­ing been val­i­dat­ed by oth­ers? What did they use to val­i­date it? Which plat­form did they use basi­cal­ly? You can also ask just about the plat­form itself. Is there a paper that val­i­dates the plat­form or is that some­thing that does­n’t exist yet? Just final­ly, again, how easy is it to use this plat­form? These are some of the key things that you’ll be want­i­ng to ask your­self before get­ting start­ed with your online experiments.

Dr. Joshua Bal­sters:
So, I’m going to talk now about some prac­ti­cal con­crete exam­ples of dif­fer­ent par­a­digms that I’ve seen online and ways that they’ve been able to, I think, bridge the gap, espe­cial­ly between lab-based and online research. Now, this is a real­ly clever study. What they were able to do here, they were able to cre­ate an Airbnb type sit­u­a­tion. I believe this was built in Task Builder. There was­n’t any code nec­es­sary. It was quite straight­for­ward. It’s images and dif­fer­ent buttons.

Dr. Joshua Bal­sters:
What hap­pens is when you want to see a piece of infor­ma­tion, you will click on the gold coin. That means you’re going to buy that con­tent essen­tial­ly. It will unveil and reveal that con­tent to you. They’re able to do some real­ly great research to look at, first of all, “What kinds of infor­ma­tion are peo­ple will­ing to pay for?” They’re able to say, as well, “How much do they spend?” They’re able to say as well, “How does that influ­ence your infor­ma­tion? How does that influ­ence your deci­sion mak­ing process­es?” It’s a real­ly great study. It’s worth hav­ing a look at this right now. I think it’s a won­der­ful exper­i­ment, very eas­i­ly for it online.

Dr. Joshua Bal­sters:
So, I’m just going to go through an exam­ple now of how we can think about mov­ing past tra­di­tion­al online stud­ies. So, for exam­ple, this was an exper­i­ment I built for Goril­la Acad­e­my. It’s part of the learn­ing sec­tion. Put that one on pause. It’s your clas­sic psy­chol­o­gy exper­i­ments. You’ve got two icons, a green and a blue one. You click on the right one, you’ll see win. You click on the wrong one, then it tells you you lose. There’s lit­tle points update. It’s not very catchy, I have to say that. It’s exact­ly the stuff that I built through­out my PhD and through­out my post­docs. It’s the old clas­sic psy­chol­o­gy exper­i­ment. But I think we’re get­ting to a place now where we can do some­thing bet­ter, more sophis­ti­cat­ed. That’s real­ly what we’re going to talk about a lot today with Game Builder.

Dr. Joshua Bal­sters:
If I just take this one back to the begin­ning now, this is the exact same exper­i­ment, but what we have here are we have cards. When you choose the cor­rect card, you see the star, you get the spins and the par­ti­cle effects, these sparkles. So, we have all of these things, which are real­ly great. I can imag­ine doing 100 tri­als of this far more hap­pi­ly than I would 100 tri­als of the pre­vi­ous exper­i­ment. I think that’s very impor­tant. If you want to make sure you get high data qual­i­ty and you want to keep peo­ple engaged in your exper­i­ments, then it’s impor­tant that we start think­ing about these kinds of issues. How can I make my exper­i­ments more engag­ing? How can I make them more inter­est­ing and more fun to do?

Dr. Joshua Bal­sters:
I’m remind­ed recent­ly of look­ing at Sea Hero Quest that you may have heard about, a bril­liant game. It was from Pro­fes­sor Hugo Spears over at UCL. What they did is they took a tra­di­tion­al par­a­digm called the Mor­ris water maze and they turned it into a real­ly fun game that peo­ple explore worlds on jet skis. What they’ve been able to do with that game is incred­i­ble. They’ve had some­thing like four mil­lion unique down­loads. I think they’ve col­lect­ed data that adds up to, I think, 117 years of data.

Dr. Joshua Bal­sters:
So, I think that is incred­i­bly impres­sive and that’s what you get with this gam­i­fied research. You don’t need to wor­ry about pay­ing as much for these par­tic­i­pants. You cre­ate a game that is so nat­u­ral­ly engag­ing, peo­ple want to do your exper­i­ment, rather than you hav­ing to force peo­ple to do the exper­i­ment. It’s some­thing worth think­ing about and we’ll talk a lit­tle bit about that today.

Dr. Joshua Bal­sters:
Now, this is from a sep­a­rate exper­i­ment. It’s not the one that I’ve just spo­ken about, but it was a dif­fer­ent val­ue-based deci­sion mak­ing study with sim­i­lar para­me­ters. What we can see here is that there is quite a good degree of over­lap between what they get from the lab and what they get with online research. I think that, again, it’s nice to have these san­i­ty checks. I stand by what I said before­hand. I don’t think we should nec­es­sar­i­ly con­sid­er the lab to be the ground truth, but I think these kinds of san­i­ty checks are very help­ful and I think can real­ly add con­fi­dence in what you’re doing.

Dr. Joshua Bal­sters:
I’m going to talk a lit­tle bit now about eye track­ing. A lot of peo­ple when they’ve gone online wants to do eye track­ing research. It’s dif­fi­cult. I’m not going to lie, I think it’s an incred­i­bly dif­fi­cult thing to achieve. There are a lot of rea­sons for that. We ran some­thing called Goril­la Presents where we talk to a pan­el of experts about these prob­lems. If you want to do eye track­ing research or mouse track­ing research, I high­ly rec­om­mend going on to our web page and look­ing at this Goril­la Presents webi­nar. There’s so many tips. It’s a gold­mine of infor­ma­tion, I can’t stress it enough.

Dr. Joshua Bal­sters:
Actu­al­ly, the data that we can see up in here, this is data from Jen­s’s paper. He did­n’t use Goril­la. He used the WebGaz­er tool that Goril­la also uses. What he did is he did an eye track­ing study in the lab, which you can see here in blue. This is the hor­i­zon­tal gaze. So, when some­one looks left, there’s a move­ment to one direc­tion. When some­one looks right, there’s a move­ment in anoth­er direc­tion. What we can see is that actu­al­ly, there’s a very high degree of sim­i­lar­i­ty between lab-based test­ing and at-home test­ing for eye track­ing. There’s obvi­ous­ly far more noise in the at-home, but the actu­al gen­er­al mean tra­jec­to­ries are incred­i­bly similar.

Dr. Joshua Bal­sters:
In fact, there were cor­re­la­tions of 0.91 and 0.83, so very high cor­re­la­tions between lab-based test­ing and home test­ing. So, it is pos­si­ble to do eye track­ing at home using web cam­eras. I should you say though, it is incred­i­bly dif­fi­cult. Watch the Goril­la Presents webi­nar, because it’s going to have so many tips there. It real­ly was a gold­mine. I think it had every­thing from shav­ing was a big thing. So, try to get beard­less peo­ple. I remem­ber that was one of the big take home mes­sages from the talk.

Dr. Joshua Bal­sters:
If you can, it’s not always pos­si­ble, but if you can, I would prob­a­bly rec­om­mend mouse track­ing over eye track­ing. I think there are a lot of ben­e­fits to doing mouse track­ing stud­ies. Obvi­ous­ly, it’s far more accu­rate than eye track­ing on online com­put­er-based test­ing. These are two exam­ples that have used mouse track­ing in dif­fer­ent but very inter­est­ing ways. We can see here that for exam­ple, in this top study, what they’ve done is they’ve used mouse clicks as a way of gaug­ing atten­tion. The yel­low, red heat maps are from where the peo­ple clicked. The white heat maps are com­pu­ta­tion­al mod­els of salience. You can see that essen­tial­ly, these two mod­els are over­lap­ping fan­tas­ti­cal­ly well.

Dr. Joshua Bal­sters:
So, what we can do is we can replace. Rather than wor­ry­ing about eye track­ing and you do lose a lot of data with eye track­ing data, with mouse track­ing, basi­cal­ly, your attri­tion is far, far less. You’re get­ting what looks like very sim­i­lar results, which is incred­i­bly infor­ma­tive. I think incred­i­bly, it’s help­ful to know.

Dr. Joshua Bal­sters:
Now, this is from Jonathan Tsay’s work. What he did is he took the clas­sic motor adap­ta­tion par­a­digm, one of these ones where you have a robot­ic arm. That robot­ic arm, you manip­u­late it so that even though I’m mov­ing my arm direct­ly for­ward in front of me, the image looks like it’s mov­ing out to the side. This is the kind of work that’s been done in most con­trol labs across the world for decades, real­ly great exper­i­ments, but what they can now do is they can take it online.

Dr. Joshua Bal­sters:
You can see, when we look at the tra­jec­to­ries for in-per­son test­ing com­pared to online test­ing, we’re get­ting the same shape or func­tion, both in per­son and online, which is again, very reas­sur­ing. As I men­tioned ear­li­er, this means that we can now do motor adap­ta­tion work with a whole new group of par­tic­i­pants, peo­ple who, because of per­haps of their motor con­di­tions, weren’t able to get into the lab before­hand. We can now work with those indi­vid­u­als as well, which is incred­i­bly rewarding.

Dr. Joshua Bal­sters:
Okay. So, I’m just going to wrap up now and hope­ful­ly, have a lit­tle bit of time if we have to some ques­tions. We’ve talked about this idea of, “Why should we go online? What are the ben­e­fits of online research?” I’d just remind you, speed, scale, rich. If any­one want to ask, those are the three things, three pil­lars why you should go online. We talked about the chal­lenges. I think a lot of these aren’t nec­es­sar­i­ly chal­lenges or they’re cer­tain­ly not chal­lenges that are exclu­sive to online research. These are the same chal­lenges every­one has in lab-based research. There are prac­ti­cal con­sid­er­a­tions before you get start­ed. You’ll need to pilot, pilot immense­ly. I can’t stress that enough.

Dr. Joshua Bal­sters:
You need to think about your exper­i­men­tal plat­form. Can it do what I need it to do? Does it have all the ethics require­ments that I need to do what I want to do? There are a lot of prac­ti­cal con­sid­er­a­tions to think about before you take your exper­i­ments online. And then final­ly, I’ve just giv­en you a hand­ful of exam­ples there about what you can do with online research and the way that online research in some cas­es, replic­a­ble of what we’re doing in the lab. In oth­er cas­es, it’s real­ly just expand­ing and chang­ing what we can do with research. So, thank you ever so much for join­ing me today. I’m hap­py to take any ques­tions you might have or if there’s any­thing you’re inter­est­ed in ask­ing right now.

Sophie Scott:
Josh, there are a cou­ple of ques­tions com­ing through on the Q&A. Can I ask you those?

Dr. Joshua Bal­sters:
Yeah, absolute­ly. Please, go ahead.

Sophie Scott:
So, some­body asked a very spe­cif­ic ques­tion. I think it’s an inter­est­ing one about the use of progress bars. Do you think there’s any pos­si­bil­i­ty that that might actu­al­ly have an influ­ence on they start get­ting bored or they go, “No, [inaudi­ble 00:36:17] ages to go. I’m near­ly at the end. I’m going to rush through it.” Do you know what I mean? What’s your feel­ing about the util­i­ty of those?

Dr. Joshua Bal­sters:
Yeah, that’s a very good point. I think it could get peo­ple drop­ping out soon­er, but I think in some ways that might be ben­e­fi­cial. I think it cer­tain­ly is nicer from the par­tic­i­pan­t’s point of view. The par­tic­i­pant says, “Oh, my God, I’ve been at this for ages and I’ve hard­ly scratched sur­face. Nev­er mind.” Well, if they have oth­er com­mit­ments in their life, they know the post­man’s due to come around that time, they think, “Oh, I can’t com­mit to this right now. I know, I’m going to have to engage in some­thing. I should drop out now. If it’s still avail­able lat­er on, I can come back to it.” In my opin­ion, it’s ben­e­fi­cial. Yeah, the pos­i­tives out­weigh the neg­a­tives in my opinion.

Sophie Scott:
There’s also just a ques­tion that’s appeared in the chat. I would ask if you can put these in the Q&A, please do so. Are there any eth­i­cal issues around using eye track­ing and mouse track­ing online? Is there any­thing par­tic­u­lar that we need to think about?

Dr. Joshua Bal­sters:
I don’t believe so, noth­ing that you would­n’t have with tra­di­tion­al exper­i­ments. I think with a lot of the eye track­ing soft­ware, you don’t have to wor­ry, because it does­n’t cap­ture any facial infor­ma­tion. With eye track­ing in par­tic­u­lar, what it tends to do is it cre­ates a face mod­el, which is a grid of where your face is, but it does­n’t cap­ture any­thing more than that. So, there’s no iden­ti­fi­able infor­ma­tion that you need to wor­ry about being sent to dif­fer­ent peo­ple. In the tools them­selves, I can’t think of any eth­i­cal issues. How you use them, obvi­ous­ly, that changes the situation.

Dr. Joshua Bal­sters:
Lat­er on, today, we’re doing some ses­sions on Mou­se­View, actu­al­ly, which is a won­der­ful alter­na­tive to eye track­ing. Well, I think one of the talks is about using this in sui­ci­dal research. So, obvi­ous­ly, when you use it in such a con­text, then there are eth­i­cal issues. But the tool in itself, I think, is fine. I don’t think there are any eth­i­cal issues to wor­ry about.

Sophie Scott:
There’s a very gen­er­al ques­tion here from Nina, which is, “When we decide to do an online research study, how can we be sure that the study will be reli­able and valid?”

Dr. Joshua Bal­sters:
Yeah, I think that’s a very big ques­tion. To be sure, I think, cer­tain­ty is quite dif­fi­cult to come by. I think you obvi­ous­ly do the best that you can. Like I said, I think the best tips are things we’ve men­tioned as well. Pilot­ing is incred­i­bly impor­tant and that’s going to tell you a lot of what you need to know. What I would also say is build­ing on the back of oth­er stud­ies, I stand by it. I hope I don’t sound hyp­o­crit­i­cal, because I know I’ve said and I do stand by what I said as well, that lab-based should­n’t be treat­ed as nec­es­sar­i­ly being the ground truth, but obvi­ous­ly, repli­ca­tion is fantastic.

Dr. Joshua Bal­sters:
So, tak­ing tried and test­ed exper­i­ments that have been done time after time, work­ing with those kinds of par­a­digms and build­ing up on those par­a­digms is a real­ly great thing. So, I think that’s real­ly one of the best things you can do. Look for those reli­able par­a­digms that proved them­selves, par­a­digms that have what I would call san­i­ty checks in them, things that you know should happen.

Sophie Scott:
I would com­plete­ly agree with you on that and this would apply to lab-based sci­ence as well. This is just doing sci­ence. It is a good idea to have things built in, that you know things are work­ing. I should expect to see this result based on what we already know. I think just very briefly, going to your point about the ground truth, I think you’re absolute­ly right. There’s no ground base­line, ground truth for lab-based stuff. Actu­al­ly, that’s true across psy­chol­o­gy. I think one of the things that’s quite use­ful to think about is you’re nev­er going to get to an answer if you just try one method.

Sophie Scott:
So, keep your focus broad­ened. Think about your research ques­tion. “What are dif­fer­ent ways you could address that ques­tion?” is also anoth­er impor­tant thing, because we don’t have these ground truths. So, hav­ing a broad­er per­spec­tive on it, I think, can real­ly help. Enough for me, though. There’s a qual­i­ta­tive ques­tion from Wal­ter. Do you have any con­sid­er­a­tions regard­ing selec­tion bias in online studies?

Dr. Joshua Bal­sters:
I think there’s been some inter­est­ing stuff out there about dif­fer­ent plat­forms. So, I’m think­ing there has been some research out there about Pro­lif­ic ver­sus MTurk and the indi­vid­u­als inside those plat­forms and how rep­re­sen­ta­tive they are. I know Pro­lif­ic has this real­ly great fea­ture now where I think you can actu­al­ly have what they refer to as a rep­re­sen­ta­tive pop­u­la­tion, which is quite a nice fea­ture to try and make sure that you com­bat selec­tion bias and you get the whole spec­trum of indi­vid­u­als with­in your exper­i­ment, which I think is a won­der­ful idea. I don’t think there are as many con­sid­er­a­tions to be hon­est. I would hope that with these tools, at least we’re mov­ing for­ward com­pared to lab-based research.

Dr. Joshua Bal­sters:
I think I can speak now to the lab-based work I’ve done in the past and I know that they’re quite homo­ge­neous pop­u­la­tion some­times. So, you can see from ques­tion­naire data look­ing at indi­vid­ual dif­fer­ences that there aren’t that many indi­vid­ual dif­fer­ences. You can see there’s a very strong bell curve with quite a nar­row pre­ci­sion to it. So, I don’t think there are as many con­sid­er­a­tions. My feel­ing that is a lit­tle bit of intu­ition and gut feel­ing is that we’re just get­ting away from selec­tion bias with online stud­ies, not com­plete­ly, but it’s a step in the right direction.

Sophie Scott:
So, some­thing that my PhD stu­dent, Alex­is McIn­tyre, has been look­ing at is vari­a­tion in rhythm pro­cess­ing. She has been using a rhythm pro­cess­ing task that she devel­oped for sev­er­al years. You expect to get a wide range of per­for­mance on it. When she moved her stud­ies online last year, she found that scores on that went real­ly high and much more nar­row. This is through Pro­lif­ic. When she fil­tered out for peo­ple who did­n’t play com­put­er games, she found actu­al­ly, you saw some­thing a lot more nor­mal, but actu­al­ly, they also dropped down by an order of 100 fold with the peo­ple who end­ed up in this study.

Sophie Scott:
So, I think some­times actu­al­ly, if you have peo­ple who play a lot of com­put­er games and cer­tain­ly, on her Pro­lif­ic study, there were a lot of these peo­ple. They were doing real­ly well. I think that it does­n’t take away from the results of the study, but it was inter­est­ing in terms of look­ing at indi­vid­ual vari­a­tion, where you would expect to see some­thing broad­er. There were some inter­est­ing rea­sons as to why that was and it’s just worth bear­ing that in mind. There isn’t going to be one truth for every­thing out there and it is def­i­nite­ly worth bear­ing in mind.

Sophie Scott:
How­ev­er, again, enough for me, maybe we’ve got time for one more ques­tion. So, sev­er­al points of this com­ing through. What hap­pens if it’s very dif­fi­cult to make your study more inter­est­ing? Would that affect repli­ca­tion? But also, if you can’t make it more inter­est­ing, so with a more engag­ing study, with a more eas­i­ly repli­cat­ed, but also, what should you do if you just can’t make it more fun?

Dr. Joshua Bal­sters:
I think unfor­tu­nate­ly, I think that’s very dif­fi­cult. I think you can make things more fun. I know, we’ve got one of our speak­ers lat­er on in the game ses­sion. He took the go/no-go task, which is a bor­ing study. It’s not an inter­est­ing task, espe­cial­ly after a few rep­e­ti­tions. But he took that and he put a whole skin over it and turned it into this amaz­ing game, where you had avatars. Instead of just being press red cir­cle, press green cir­cle, release, it became a drag­on and you had to har­vest gold to avoid the dragon.

Dr. Joshua Bal­sters:
So, I think there are a lot of real­ly great cre­ative solu­tions out there. I think you can do a lot just by chang­ing essen­tial­ly the skin of it. So, even if it’s not cre­at­ing a whole nar­ra­tive, just mak­ing some­thing that’s visu­al­ly more appeal­ing can actu­al­ly be real­ly use­ful, I think. So, I think there are a lot of oppor­tu­ni­ties to do that. I think if you real­ly, real­ly can’t do that, it’s very dif­fi­cult, I think you might end up just hav­ing to pay peo­ple. I think it’s the only oth­er option, exter­nal motivators.

Sophie Scott:
Thank you. Thank you.

 

Get on the Waitlist

BeOnline is the conference to learn all about online behavioral research. It's the ideal place to discover the challenges and benefits of online research and to learn from pioneers. If that sounds interesting to you, then click the button below and sign up for our newsletter. You will be the first to know when we release new content and open applications for BeOnline 2022.

With thanks to our sponsors!

Get­ting Research Online