Get­ting Research Online

Dr. Joshua Bal­sters:
Okay. Thank you very much, Joe. Thanks for get­ting me up here. So, can you write in the chat, just tell me you can see the screen? Type, “See screen” into the chat for me, just so I know that you can see my slides. I’m not going to be just talk­ing to air. All right, bril­liant. Thank you, every­one. That’s great. So, thank you for com­ing to join us today. My name is Joshua Bal­sters. It feels like a long time ago now. I used to be a lec­tur­er in the Psy­chol­o­gy Depart­ment at Roy­al Holloway.

Dr. Joshua Bal­sters:
Well, I guess before that, I was just going from post­doc to post­doc. I spent quite a few years doing almost every kind of research real­ly. I’ve done atten­tion stud­ies, mem­o­ry, role-based learn­ing, social deci­sion mak­ing, val­ue-based deci­sion mak­ing, some brain imag­ing stuff. I think that’s why Jo want­ed me to come online to talk a bit about this, try­ing to con­dense all of that research expe­ri­ence to give you some hints and tips about tak­ing work online. Some of the things that we’ve seen, espe­cial­ly over the last year with Goril­la users, what are some of the best ways to take your research online?

Dr. Joshua Bal­sters:
So essen­tial­ly, we’re going to start off just by intro­duc­ing this ques­tion, why should you go online? What are the ben­e­fits to you? We’re going to talk a lit­tle bit about the chal­lenges and some of the rebut­tals, the things that pesky review or two is going to prob­a­bly say to you. We’re going to talk about some of the prac­ti­cal con­sid­er­a­tions, some of the real­ly use­ful tips and tricks that we’ve been pick­ing up at Goril­la and things we’ve picked up in oth­er places to try and help you out and help you cre­ate the best online exper­i­ment pos­si­ble. I’m going to go through some inspi­ra­tional sto­ries as well, some real­ly bril­liant exam­ples of some of the kinds of just astound­ing research that you can take and you can take online.

Dr. Joshua Bal­sters:
So, I think in three sim­ple words, why should you take your research online? Speed, scale, and reach. I think these are real­ly the three pil­lars of online research. The speed with which you can col­lect data when you’re work­ing online, I still find this absolute­ly fas­ci­nat­ing. It’s just incred­i­ble think­ing about the months I used to spend doing non-online research, doing lab-based research. In fact, now with online research, I grab a sand­wich. By the time I get back, my exper­i­men­t’s done. I think it’s absolute­ly incred­i­ble the speed that you can col­lect data with online. The impor­tant thing about that speed and it’s something

Dr. Joshua Bal­sters:
I think for every­one, no mat­ter what stage in your career you are, whether you decide to stay in acad­e­mia or leave, that speed allows you to buy back your time. That’s prob­a­bly one of the most impor­tant assets we have, our time. So, in my per­son­al opin­ion, you don’t real­ly learn that much after hav­ing 10 par­tic­i­pants through the lab and you’ve prob­a­bly refined your spiel, you’re bet­ter at explain­ing your exper­i­ments to people.

Dr. Joshua Bal­sters:
After that, you’re not real­ly gain­ing any­thing in terms of skills. It’s just time that’s being con­sumed. Where­as with online research, you’re buy­ing back that time, because it is just done for you in a mat­ter of hours rather than months. That means you’ve got free time on your hands and you can decide what to do. You can decide whether it’s bet­ter to learn a new analy­sis type skill, read around your sub­ject more, get around to writ­ing your paper, or just have a glass of wine and watch some­thing nice on Net­flix and take a bit of time for you. Buy­ing back your time is prob­a­bly the most impor­tant thing, I think, about mov­ing your research online.

Dr. Joshua Bal­sters:
But on top of that, we also have the scale of online research. With lab-based research, there’s this lin­ear rela­tion­ship. If I want twice as many sub­jects, it takes twice as long to col­lect. You don’t have that with online research. If I want to go 2 times, 3 times, 10 times as many par­tic­i­pants, it’s usu­al­ly a small frac­tion of time that you are adding on to your data col­lec­tion, rather than dou­bling, tripling, expo­nen­tial­ly increas­ing your time to col­lect data. With that, you also get rich. Rich works in two ways, which is quite inter­est­ing. The first of all, it allows you to find par­tic­i­pants who you would­n’t nor­mal­ly get to work with. I found this when I was in Ire­land, I did a lot of aging research.

Dr. Joshua Bal­sters:
The elder­ly indi­vid­u­als were absolute­ly won­der­ful, but I think after a while you real­ize the elder­ly indi­vid­u­als who are able to make the efforts to come into a lab­o­ra­to­ry and do research with you, these aren’t per­haps the typ­i­cal elder­ly indi­vid­u­als. You’re miss­ing out, for exam­ple, on lots of peo­ple who might have dif­fer­ent kinds of phys­i­cal pain, hip prob­lems, knee prob­lems. These peo­ple can’t come into the lab and see you. You’re miss­ing out on research­ing with them. So, online research means that you can send research into peo­ple’s homes. So, it means you get a more rep­re­sen­ta­tive sam­ple and it means in line with the idea of scale. You’re increas­ing the num­ber of par­tic­i­pants in what is some­times quite dif­fi­cult to acquire populations.

Dr. Joshua Bal­sters:
This is great, obvi­ous­ly, in terms of repro­ducibil­i­ty. The BeOnline Con­fer­ence, well, this con­fer­ence obvi­ous­ly, we’ve got a great ses­sion tomor­row after­noon, wrap­ping every­thing up with Jo and a num­ber of oth­er won­der­ful speak­ers to talk about Repro­ducibil­i­ty 2.0. The oth­er thing as well is we have, just before that, Jonathan Tsay. He’s going to be doing a sym­po­sium about motor con­trol online. In pre­vi­ous chats, he’s told us that the abil­i­ty to take his work online has opened up a whole new par­tic­i­pant pool and we’re increas­ing the sam­ple sizes in that par­tic­i­pant pool. So, I think these are some of the key ben­e­fits for online research. Online research was on the rise any­way pre-pandemic.

Dr. Joshua Bal­sters:
From what we hear from our clients, it’s been the push they need­ed to get online and start doing online research. Most like­ly, what’s going to hap­pen is that online research is here to stay. What we’re hear­ing is that most peo­ple are going to do a com­bi­na­tion of lab-based and online stud­ies now on. So, it’s impor­tant that you get famil­iar with online research as it becomes increas­ing­ly com­mon­place for all the rea­sons that I’ve just mentioned.

Dr. Joshua Bal­sters:
For all the super­vi­sors out there, this quote means a lot to me. Being able to test online means my research does­n’t grind to a halt. This was espe­cial­ly impor­tant for my PhD stu­dents, I did not want their research and men­tal health suf­fer, because they could­n’t car­ry out their stud­ies. Now, I was in the sit­u­a­tion with my own PhD stu­dent. She had a mag­nif­i­cent MRI study. The day she was about to start test­ing, we had to shut the whole MRI scan­ner down.

Dr. Joshua Bal­sters:
I think this hap­pened to a lot of peo­ple. I’m sure there’s a lot of sym­pa­thy out there for peo­ple who’ve been in this sit­u­a­tion. Online research, I think, it’s going to help peo­ple, like I said, buy back their time and it’s going to increase the well-being of a lot of peo­ple doing research. For me, that’s real­ly the most impor­tant thing, buy­ing back your time with online research.

Dr. Joshua Bal­sters:
I’m going to jump now and talk a lit­tle bit about the chal­lenges of online research, the cri­tiques that you often hear about online research. This is David Ogilvy who you might not have heard of. Appar­ent­ly, he was the inspi­ra­tion for Don Drap­er in Mad Men. He was quite a famous mar­ket researcher. He has this very famous quote that “The trou­ble with mar­ket research is that peo­ple don’t think how they feel, they don’t say what they think, and they don’t do what they say.” I think this is a great quote for all behav­ioral sci­ence, to be hon­est, but I think a lot of peo­ple have always con­sid­ered online research to just be sur­vey based.

Dr. Joshua Bal­sters:
This is their cri­tique. Well, I need to do some­thing that gets at some of the implic­it behav­ioral para­me­ters, things that go beyond what peo­ple say, go beyond your tra­di­tion­al sur­vey-based research. How can I do that online? Well, now we know there are increas­ing­ly a num­ber of plat­forms that are mak­ing that avail­able and allow­ing you to cre­ate absolute­ly won­der­ful exper­i­ments online, which are incred­i­bly com­plex and sophis­ti­cat­ed. I’m going to talk about just a cou­ple of these at the end of this talk.

Dr. Joshua Bal­sters:
Now, yeah, once we move online, you might have to think, “Oh, God, do I need to learn anoth­er pro­gram­ming lan­guage?” If you were like me, I trained in MATLAB, which is becom­ing pret­ty much a defunct lan­guage. I start­ed learn­ing Python and R. And then online research came and I just thought, “Oh, my God. Now I need to learn HTML and JavaScript. Real­ly?” Basi­cal­ly, you don’t have to any­more. There are a lot of amaz­ing tools out there, which allow you to do a lot of things just using a stan­dard graph­i­cal user inter­face, nice drag and drop interface.

Dr. Joshua Bal­sters:
In fact, if you want to find out more about that, I wrote a blog piece for Goril­la a cou­ple weeks ago. It was a very cathar­tic blog piece, I have to say. I got a lot of demons out talk­ing about those evil red lines of code. You can have a read of that. Essen­tial­ly, the point is you don’t need to wor­ry about learn­ing a new pro­gram­ming lan­guage. You can go out there and you can cre­ate won­der­ful exper­i­ments with­out need­ing a line of code.

Dr. Joshua Bal­sters:
Now, I think the biggest chal­lenge for online research is this idea that online research is inher­ent­ly flawed and that par­tic­i­pants are going to behave bad­ly. They’re not going to give you the type of data qual­i­ty that you get in the lab. I would chal­lenge this asser­tion. I don’t think it’s nec­es­sar­i­ly true. In fact, Jo Ever­shed wrote a real­ly nice blog piece pre­vi­ous­ly about the illu­sion of con­trol. We talk about these cri­tiques of online research, this idea that, “Oh, well, obvi­ous­ly, they are going to be just press­ing the space­bar with one hand while play­ing Fort­nite with the oth­er hand or watch­ing the Eng­land game lat­er on,” for example.

Dr. Joshua Bal­sters:
I don’t think that’s a fair crit­i­cism. I think when we think about lab-based test­ing, we paint it as this pic­ture of puri­ty, when it real­ly is not. For exam­ple, I’m sure every now and again, we’ve drift­ed off in the mid­dle of an exper­i­ment in a lab and our mind has gone off to wan­der­ing, espe­cial­ly if it’s a bit of a bor­ing exper­i­ment. Now, as well, you can see in terms of qual­i­ty, well, there are obvi­ous­ly dis­trac­tions in the lab. Peo­ple go in with their phones. They go in with their smart­watch­es on. They’ll get buzzers. There’ll be oth­er dis­trac­tions. So, the point that I would like to raise, I would, first of all, like to chal­lenge this idea that the lab is some­how the ground truth of behavior.

Dr. Joshua Bal­sters:
I think in a lot of ways you can argue it is an unre­al­is­tic sit­u­a­tion. Again, think­ing lat­er on, I think there’s going to be some chats about this in the Repro­ducibil­i­ty 2.0 ses­sion tomor­row. So, idea that lab-based test­ing may not be the ground truth we think it is. That said, I would also say that if you design good exper­i­ments, then you can get just as good data qual­i­ty out of an online study that you would get in the lab. I’m going to show you just a hand­ful of exam­ples that prove that is the case.

Dr. Joshua Bal­sters:
These are what we think about, our sev­en-step guide for suc­cess­ful­ly doing an online study. These are just some sug­ges­tions and a way of think­ing about build­ing your next online exper­i­ments and the things that you should con­sid­er. First of all, con­sid­er your study, what is the best study to get start­ed with, espe­cial­ly if you are new to online research? What’s the best study to take? I think, per­son­al­ly, I would say start with some­thing that you’re famil­iar with. It’s the best way to get start­ed on this.

Dr. Joshua Bal­sters:
Rather than tak­ing a whole brand new top­ic, as well as tak­ing on a whole brand new domain with online research. Why don’t you start say­ing, “Well, this is what I did in the lab. So, let’s try it online”? It’s always great to have these kinds of san­i­ty checks, lit­tle things that can be built into your study that will help you feel con­fi­dent about what you are doing. So, when you’re select­ing a study, I’ve rec­om­mend try­ing to do that.

Dr. Joshua Bal­sters:
The next thing we’re going to talk about is build­ing the study. What do you need to build your exper­i­ment? These are a cou­ple things to bear in mind, I think, some key things relat­ed to online research. First thing, I would say, is you need clear and con­cise instruc­tions. I can’t stress this enough. What I would say is that you’re not con­fined any more to the lab-based design, where you typ­i­cal­ly would have an A4 sheet of paper with your instruc­tions writ­ten all over it and you would say, “Oh, read this. If you have any ques­tions, let me know.” First of all, you’re not there to answer any ques­tions, but also, it’s a very bor­ing and for­mal way of doing things. As you’ve moved online, why not change essen­tial­ly? We can do things in any for­mat now.

Dr. Joshua Bal­sters:
One of the things I real­ly enjoy as part of my time at Goril­la is Goril­la Acad­e­my. One of the things I have in there is a social influ­ence task, a real fun thing that I made, where we see the influ­ence Rot­ten Toma­toes has on your behav­ior. With­in that, what I decid­ed to do, rather than hav­ing a lengthy block of text to explain par­a­digm, I just made a video of me doing the task and talk­ing through the tri­al. I think some­times that’s just the eas­i­est way of doing things. So, don’t feel con­strained by the way you used to do things in the lab. You can adapt and do lots of crazy new things with online research.

Dr. Joshua Bal­sters:
In fact, this YouTube link, which we’re going to make acces­si­ble lat­er on, this links to Simone’s research. Simone is actu­al­ly going to be speak­ing next after me, the first speak­er in the Buf­fet of Research sec­tion. So, def­i­nite­ly, you want to lis­ten to her. She’s got so many top tips, it’s not even real. Yeah, her videos are amaz­ing. She talks to how to take an incred­i­bly com­pli­cat­ed eye track­ing study. She has the most beau­ti­ful, detailed instruc­tions. That’s the kind of thing that you want to be going for.

Dr. Joshua Bal­sters:
So, next thing I want to talk about is keep it short. This is some­thing that we rec­om­mend with our online stud­ies. If you can, it’s not always pos­si­ble, but if you can, try to keep your stud­ies to 5 to 10 min­utes. That’s going to real­ly help. The oth­er thing as well is to use progress bars and this real­ly helps a lot with attri­tion. If a par­tic­i­pant can see how many tri­als along they are, then they’re going to be able to say, “Oh, I’ve only got a small sec­tion left. I should keep going.” Rather than some­body who’s done 95% of the study, drop­ping out are the last 5%. So, I think that’s going to be incred­i­bly help­ful. If you can, build in breaks.

Dr. Joshua Bal­sters:
For a lot of rea­sons with online stud­ies, if peo­ple know that they’re going to get a break every five min­utes, that’s incred­i­bly help­ful for them. So, that’s some­thing to bear in mind as well. If you can, include vig­i­lance checks. So, with Goril­la Acad­e­my, again, this was a dif­fer­ent one. This was the atten­tion study. It was a beau­ti­ful exper­i­ment by Pol­ly Dal­ton look­ing at dichot­ic lis­ten­ing and you hear the voic­es all around your head. It’s like you’re in the room, but it only works if you have head­phones in. So, if some­body just has their speak­ers on, it won’t work.

Dr. Joshua Bal­sters:
So, in order to get around that and make sure peo­ple were lis­ten­ing with their head­phones, I had a lit­tle lit­mus check that just said, “Can you hear this noise on the left or the right?” You would­n’t be able to tell from your speak­ers, but in the head­phones, it’s obvi­ous. So, have these lit­tle vig­i­lance checks built into your study. What you’ll see when I went to ana­lyze the data is that by exclud­ing indi­vid­u­als that failed the vig­i­lance check, I increased my effect size. I was able to actu­al­ly pre-reg­is­ter those exclu­sion cri­te­ria because of pilot­ing, which is what I’m actu­al­ly going to come to in a second.

Dr. Joshua Bal­sters:
Now, the next thing as well is this debrief ques­tion­naire. So, just ask peo­ple. I can’t rec­om­mend that high­ly enough. If you’ve got par­tic­i­pants doing your exper­i­ments, you can just have a form at the end that says, “Was there any­thing unusu­al? Was there any­thing you think could be explained bet­ter? Did this make sense? Did you enjoy it?” Ask peo­ple for their feed­back. It’s just so sim­ple and it will give you so much infor­ma­tion before you put your exper­i­ments online. Hav­ing a debrief ques­tion­naire is incred­i­bly help­ful, espe­cial­ly when you’re piloting.

Dr. Joshua Bal­sters:
Before I get into pilot­ing, I just want to briefly men­tion ethics. Prob­a­bly before, in fact, you’d get into all the effort of build­ing your exper­i­ments, when you’re think­ing about your online study, you need to think about the ethics and think­ing specif­i­cal­ly about the plat­form. Does your plat­form con­form to infor­ma­tion secu­ri­ty rules in your depart­ment? Does it con­form to things like GDPR? Now, these are hor­ri­ble, bor­ing things that nobody real­ly wants to have to deal with. So, if you can, find a plat­form that has that infor­ma­tion and says, “Don’t wor­ry, we’ve got this.” Your ethics com­mit­tee will prob­a­bly ask you about it. So, look into it definitely.

Dr. Joshua Bal­sters:
Now, I want to talk about pilot­ing. Emi­ly Breese gave the most won­der­ful talk at last year’s BeOnline all about pilot­ing and how cru­cial it is to do pilot­ing in your exper­i­ments. Now, there are a cou­ple rea­sons for this. I think one of those rea­sons is that typ­i­cal­ly, the par­tic­i­pants we get in psy­chol­o­gy depart­ments at uni­ver­si­ties, they’re psy­chol­o­gy stu­dents at the uni­ver­si­ty or they’re peo­ple who are expe­ri­enced par­tic­i­pants. So, these expe­ri­enced par­tic­i­pants, essen­tial­ly, they’re going to be able to fill in the gaps if you don’t have good enough instruc­tions. But when you move online, this prob­a­bly isn’t the case. So, you’re not around to answer their ques­tions. So, please make sure to pilot with peo­ple and fig­ure these things out.

Dr. Joshua Bal­sters:
Yeah, Emi­ly gives these won­der­ful exam­ples from the trail mak­ing test, fun­da­men­tal of neu­ropsy­chol­o­gy, where you have to either draw the line between one, two, three, four, five or the alter­nat­ing one, A, two, B, three, C. She talks about all the kinds of issues she had and some of the crazy answers peo­ple came up with that helped her refine her instruc­tions. Emi­ly goes through this much bet­ter than I do and she actu­al­ly has a won­der­ful frame­work in her video. So, I strong­ly rec­om­mend hav­ing a look at that, but don’t think about pilot­ing just for your instructions.

Dr. Joshua Bal­sters:
Pilot­ing is also good to look at your data out­put. Is your data out­put cor­rect? Do you have all the infor­ma­tion you need to ana­lyze your data? Because the speed of online test­ing is absolute­ly won­der­ful, but the flip side of that is that if you col­lect all your data so quick­ly, then unfor­tu­nate­ly, if some­thing’s wrong, you’ve just had to pay 100 or 500 peo­ple for their time when you’ve got an error. That’s not your par­tic­i­pan­t’s fault. That’s on you. So, make sure before you do your big reveal, your big release, that you check your data out­put is cor­rect and has every­thing you need.

Dr. Joshua Bal­sters:
Like I’ve men­tioned just before that, you can use this pilot­ing for exclu­sion cri­te­ria. So, you can fig­ure out, “Oh, any­one who seems to think this, they weren’t doing it right.” So, I can say that is an exclu­sion cri­te­ria. I can’t stress it enough. Pilot, pilot, pilot is prob­a­bly the most impor­tant thing you can do. You do need to pilot more with online test­ing than you would need to with lab-based testing.

Dr. Joshua Bal­sters:
I’ve already talked a lit­tle bit there about pre-reg­is­tra­tion. Obvi­ous­ly, pilot­ing is a great way to pre-reg­is­ter any exclu­sion cri­te­ria that you might have in your exper­i­ments. I’ve seen some­thing float­ing around Twit­ter recent­ly. I’m not sure if it’s a preprint or a pub­lished paper, but essen­tial­ly, they com­pared pre-reg­is­tered reports with non-reg­is­tered equiv­a­lents. They tend to find that in terms of mark­ers of cre­ativ­i­ty, every­thing is stable.

Dr. Joshua Bal­sters:
So, all those con­cerns before about pre-reg­is­tra­tion is going to ruin dis­cov­ery sci­ence, they don’t seem to be valid, but they did find that the sta­tis­ti­cal qual­i­ty and method­olog­i­cal rig­or of pre-reg­is­tered work was greater than the non-reg­is­tered con­trols. So, I think pre-reg­is­tra­tion is going to be real­ly impor­tant. Again, just to give anoth­er plug for that Repro­ducibil­i­ty 2.0 ses­sion, we’re going to get into all of these kinds of ques­tions and details by the end of BeOnline.

Dr. Joshua Bal­sters:
And then that’s it. It’s time to col­lect your data, time to ana­lyze your data. There are some won­der­ful advances com­ing up as well, things like our mark­down, which means you’re going to be able to cre­ate these beau­ti­ful, curat­ed scripts that are able to keep track of all your analy­sis and do all of your basi­cal­ly work with the raw data to pro­duce a beau­ti­ful, clear tran­script to the end of your results. There’s a lot of stuff com­ing through. A lit­tle sneak hint, there might be some­thing com­ing up in the future, long in the future with Goril­la and data, but that’s some­thing prob­a­bly for our next BeOnline Conference.

Dr. Joshua Bal­sters:
What are the things you want to con­sid­er in your exper­i­men­tal plat­form? Well, does my exper­i­men­tal plat­form do what I need it to do? That’s prob­a­bly the most cru­cial ques­tion. So, it’s worth lay­ing out. What do you need it to do? What do you need your exper­i­men­tal plat­form to do? Most peo­ple are think­ing prob­a­bly Task Builder. I need some way of pre­sent­ing images, respons­es, et cetera, but you prob­a­bly want to sur­vey as well. If you’re not doing sur­vey-based research, the abil­i­ty to cre­ate a demo­graph­ics form is incred­i­bly useful.

Dr. Joshua Bal­sters:
I found in the past that there’s some­times a mis­match between recruit­ment pan­els and cri­te­ria I set out and peo­ple I actu­al­ly end up with, because I think it’s not real­ly any­one’s pri­or­i­ty at the moment to update their recruit­ment pan­el set­tings and update their cur­rent life sit­u­a­tion. So, it’s worth hav­ing your own lit­tle sur­vey to work with that.

Dr. Joshua Bal­sters:
Now, then you have the Task Builder. Can the Task Builder do what I want? Does it have the fea­tures that I need? Are there spe­cial­ist tools that I need, eye track­ing or mouse track­ing? Is there a spe­cial kind of par­a­digm that I’m look­ing at? What did oth­er peo­ple use? I think these are all the impor­tant ques­tions you need to ask your­self. I think as well, do I need to do this in full code edi­tor or maybe you want to say, “Can I do this in the code edi­tor?” Do I have the free­dom to write some­thing from scratch or is this some­thing that I can get nice and eas­i­ly built using a sim­ple graph­i­cal user inter­face? These are the kinds of ques­tions you want to be ask­ing about your exper­i­ment before you get start­ed, before you choose an exper­i­men­tal platform.

Dr. Joshua Bal­sters:
The things we’re going to talk about today are a cou­ple of new tools. There’s the Game Builder. We’re going to talk about that today, which is absolute­ly incred­i­ble. It’s real­ly, I think, going to change a lot of the way we do research. We have a won­der­ful Game Builder pan­el, who are going to talk about the research they’ve been doing with games. And then we’re going to get a won­der­ful demon­stra­tion from Nick, our CTO. Tomor­row, we’ve got again, a great con­sumer deci­sion mak­ing pan­el talk­ing about Shop Builder and we’re going to get a Shop Builder demo. This is such a cool tech­nique. It’s such a cool tool for being able to do shop­ping-based research.

Dr. Joshua Bal­sters:
Anoth­er ques­tion, I think, you want to ask your­self is, “How well will I be sup­port­ed in what I’m doing? Are there good forums out there? Are there YouTube videos? What kinds of sup­port doc­u­men­ta­tion is there out there? What’s the com­mu­ni­ty like out there?” These are things you’ll want to be ask­ing your­self. One of the things I love about Goril­la and I work for Goril­la, this is what I’m famil­iar with. There are oth­er tools out there that have equal­ly great forms. I’d just say that out. But what I love about Goril­la from when I was a lec­tur­er, before I even joined them was the col­lab­o­rate tool, because it was anoth­er way of buy­ing back my time.

Dr. Joshua Bal­sters:
I was able to basi­cal­ly mon­i­tor my third year project stu­dents and sup­port them remote­ly, which meant at that time, I was able to go to a con­fer­ence in Paris, rather than spend­ing a cou­ple weeks in Egham. Egham’s love­ly, don’t get me wrong, but it’s no Paris. And then I could go to this amaz­ing con­fer­ence in Paris and I could look at my lap­top. I could check my emails and I could see if the stu­dent had an issue or prob­lem. I was able to sup­port them remote­ly. I found that col­lab­o­rate func­tion incred­i­bly use­ful. It bought me back time. Same with the send func­tion. The send func­tion is so use­ful, because in a lot of these dif­fer­ent exper­i­ment plat­forms, you don’t have to wor­ry any­more about mak­ing all the sys­tem diag­nos­tics match up. So, it’s like, “Here’s my task.”

Dr. Joshua Bal­sters:
Now, I am assum­ing that you’re run­ning a com­put­er that is Win­dows 7, ver­sion 2.93 bit. It’s 64 bits. It needs to be run­ning MATLAB 2014b. I think like a lot of peo­ple out there, I’ve got exper­i­ments that will only run in a very spe­cif­ic sit­u­a­tion that it’s a Tues­day and the sun’s just right. You don’t have to wor­ry about that stuff with lots of online plat­forms, because all the com­put­ing infra­struc­ture is behind the scenes. You don’t have to wor­ry. So, if you click “Send”, they’re going to get a work­ing ver­sion of your exper­i­ment. You don’t have to spend time help­ing them to debug your exper­i­ments and get it up and run­ning on their system.

Dr. Joshua Bal­sters:
Final­ly, I love our Open Mate­ri­als. It’s a great plat­form. It’s a great place for researchers to dis­sem­i­nate their projects and a great place to share stim­uli, which is so help­ful rather than hav­ing spent hours cre­at­ing stim­u­lus sets. This one is from Sarah Blake­more’s group. This was a vari­a­tion of the Raven’s matri­ces task. Now, that stim­u­lus set is online and any­one can take it. You don’t even need to ask per­mis­sion. They’re held in a repos­i­to­ry. You can just search for it and use it as you see fit. There are a num­ber of par­a­digms on Goril­la Open Mate­ri­als. Also, oth­er tools as well. There are lots of things out there, in JSI, in Python, for exam­ple. There are loads of tools out there, loads of exper­i­ments already out there ready for you to down­load and use, which is just so helpful.

Dr. Joshua Bal­sters:
Has your test­ing been val­i­dat­ed by oth­ers? What did they use to val­i­date it? Which plat­form did they use basi­cal­ly? You can also ask just about the plat­form itself. Is there a paper that val­i­dates the plat­form or is that some­thing that does­n’t exist yet? Just final­ly, again, how easy is it to use this plat­form? These are some of the key things that you’ll be want­i­ng to ask your­self before get­ting start­ed with your online experiments.

Dr. Joshua Bal­sters:
So, I’m going to talk now about some prac­ti­cal con­crete exam­ples of dif­fer­ent par­a­digms that I’ve seen online and ways that they’ve been able to, I think, bridge the gap, espe­cial­ly between lab-based and online research. Now, this is a real­ly clever study. What they were able to do here, they were able to cre­ate an Airbnb type sit­u­a­tion. I believe this was built in Task Builder. There was­n’t any code nec­es­sary. It was quite straight­for­ward. It’s images and dif­fer­ent buttons.

Dr. Joshua Bal­sters:
What hap­pens is when you want to see a piece of infor­ma­tion, you will click on the gold coin. That means you’re going to buy that con­tent essen­tial­ly. It will unveil and reveal that con­tent to you. They’re able to do some real­ly great research to look at, first of all, “What kinds of infor­ma­tion are peo­ple will­ing to pay for?” They’re able to say, as well, “How much do they spend?” They’re able to say as well, “How does that influ­ence your infor­ma­tion? How does that influ­ence your deci­sion mak­ing process­es?” It’s a real­ly great study. It’s worth hav­ing a look at this right now. I think it’s a won­der­ful exper­i­ment, very eas­i­ly for it online.

Dr. Joshua Bal­sters:
So, I’m just going to go through an exam­ple now of how we can think about mov­ing past tra­di­tion­al online stud­ies. So, for exam­ple, this was an exper­i­ment I built for Goril­la Acad­e­my. It’s part of the learn­ing sec­tion. Put that one on pause. It’s your clas­sic psy­chol­o­gy exper­i­ments. You’ve got two icons, a green and a blue one. You click on the right one, you’ll see win. You click on the wrong one, then it tells you you lose. There’s lit­tle points update. It’s not very catchy, I have to say that. It’s exact­ly the stuff that I built through­out my PhD and through­out my post­docs. It’s the old clas­sic psy­chol­o­gy exper­i­ment. But I think we’re get­ting to a place now where we can do some­thing bet­ter, more sophis­ti­cat­ed. That’s real­ly what we’re going to talk about a lot today with Game Builder.

Dr. Joshua Bal­sters:
If I just take this one back to the begin­ning now, this is the exact same exper­i­ment, but what we have here are we have cards. When you choose the cor­rect card, you see the star, you get the spins and the par­ti­cle effects, these sparkles. So, we have all of these things, which are real­ly great. I can imag­ine doing 100 tri­als of this far more hap­pi­ly than I would 100 tri­als of the pre­vi­ous exper­i­ment. I think that’s very impor­tant. If you want to make sure you get high data qual­i­ty and you want to keep peo­ple engaged in your exper­i­ments, then it’s impor­tant that we start think­ing about these kinds of issues. How can I make my exper­i­ments more engag­ing? How can I make them more inter­est­ing and more fun to do?

Dr. Joshua Bal­sters:
I’m remind­ed recent­ly of look­ing at Sea Hero Quest that you may have heard about, a bril­liant game. It was from Pro­fes­sor Hugo Spears over at UCL. What they did is they took a tra­di­tion­al par­a­digm called the Mor­ris water maze and they turned it into a real­ly fun game that peo­ple explore worlds on jet skis. What they’ve been able to do with that game is incred­i­ble. They’ve had some­thing like four mil­lion unique down­loads. I think they’ve col­lect­ed data that adds up to, I think, 117 years of data.

Dr. Joshua Bal­sters:
So, I think that is incred­i­bly impres­sive and that’s what you get with this gam­i­fied research. You don’t need to wor­ry about pay­ing as much for these par­tic­i­pants. You cre­ate a game that is so nat­u­ral­ly engag­ing, peo­ple want to do your exper­i­ment, rather than you hav­ing to force peo­ple to do the exper­i­ment. It’s some­thing worth think­ing about and we’ll talk a lit­tle bit about that today.

Dr. Joshua Bal­sters:
Now, this is from a sep­a­rate exper­i­ment. It’s not the one that I’ve just spo­ken about, but it was a dif­fer­ent val­ue-based deci­sion mak­ing study with sim­i­lar para­me­ters. What we can see here is that there is quite a good degree of over­lap between what they get from the lab and what they get with online research. I think that, again, it’s nice to have these san­i­ty checks. I stand by what I said before­hand. I don’t think we should nec­es­sar­i­ly con­sid­er the lab to be the ground truth, but I think these kinds of san­i­ty checks are very help­ful and I think can real­ly add con­fi­dence in what you’re doing.

Dr. Joshua Bal­sters:
I’m going to talk a lit­tle bit now about eye track­ing. A lot of peo­ple when they’ve gone online wants to do eye track­ing research. It’s dif­fi­cult. I’m not going to lie, I think it’s an incred­i­bly dif­fi­cult thing to achieve. There are a lot of rea­sons for that. We ran some­thing called Goril­la Presents where we talk to a pan­el of experts about these prob­lems. If you want to do eye track­ing research or mouse track­ing research, I high­ly rec­om­mend going on to our web page and look­ing at this Goril­la Presents webi­nar. There’s so many tips. It’s a gold­mine of infor­ma­tion, I can’t stress it enough.

Dr. Joshua Bal­sters:
Actu­al­ly, the data that we can see up in here, this is data from Jen­s’s paper. He did­n’t use Goril­la. He used the WebGaz­er tool that Goril­la also uses. What he did is he did an eye track­ing study in the lab, which you can see here in blue. This is the hor­i­zon­tal gaze. So, when some­one looks left, there’s a move­ment to one direc­tion. When some­one looks right, there’s a move­ment in anoth­er direc­tion. What we can see is that actu­al­ly, there’s a very high degree of sim­i­lar­i­ty between lab-based test­ing and at-home test­ing for eye track­ing. There’s obvi­ous­ly far more noise in the at-home, but the actu­al gen­er­al mean tra­jec­to­ries are incred­i­bly similar.

Dr. Joshua Bal­sters:
In fact, there were cor­re­la­tions of 0.91 and 0.83, so very high cor­re­la­tions between lab-based test­ing and home test­ing. So, it is pos­si­ble to do eye track­ing at home using web cam­eras. I should you say though, it is incred­i­bly dif­fi­cult. Watch the Goril­la Presents webi­nar, because it’s going to have so many tips there. It real­ly was a gold­mine. I think it had every­thing from shav­ing was a big thing. So, try to get beard­less peo­ple. I remem­ber that was one of the big take home mes­sages from the talk.

Dr. Joshua Bal­sters:
If you can, it’s not always pos­si­ble, but if you can, I would prob­a­bly rec­om­mend mouse track­ing over eye track­ing. I think there are a lot of ben­e­fits to doing mouse track­ing stud­ies. Obvi­ous­ly, it’s far more accu­rate than eye track­ing on online com­put­er-based test­ing. These are two exam­ples that have used mouse track­ing in dif­fer­ent but very inter­est­ing ways. We can see here that for exam­ple, in this top study, what they’ve done is they’ve used mouse clicks as a way of gaug­ing atten­tion. The yel­low, red heat maps are from where the peo­ple clicked. The white heat maps are com­pu­ta­tion­al mod­els of salience. You can see that essen­tial­ly, these two mod­els are over­lap­ping fan­tas­ti­cal­ly well.

Dr. Joshua Bal­sters:
So, what we can do is we can replace. Rather than wor­ry­ing about eye track­ing and you do lose a lot of data with eye track­ing data, with mouse track­ing, basi­cal­ly, your attri­tion is far, far less. You’re get­ting what looks like very sim­i­lar results, which is incred­i­bly infor­ma­tive. I think incred­i­bly, it’s help­ful to know.

Dr. Joshua Bal­sters:
Now, this is from Jonathan Tsay’s work. What he did is he took the clas­sic motor adap­ta­tion par­a­digm, one of these ones where you have a robot­ic arm. That robot­ic arm, you manip­u­late it so that even though I’m mov­ing my arm direct­ly for­ward in front of me, the image looks like it’s mov­ing out to the side. This is the kind of work that’s been done in most con­trol labs across the world for decades, real­ly great exper­i­ments, but what they can now do is they can take it online.

Dr. Joshua Bal­sters:
You can see, when we look at the tra­jec­to­ries for in-per­son test­ing com­pared to online test­ing, we’re get­ting the same shape or func­tion, both in per­son and online, which is again, very reas­sur­ing. As I men­tioned ear­li­er, this means that we can now do motor adap­ta­tion work with a whole new group of par­tic­i­pants, peo­ple who, because of per­haps of their motor con­di­tions, weren’t able to get into the lab before­hand. We can now work with those indi­vid­u­als as well, which is incred­i­bly rewarding.

Dr. Joshua Bal­sters:
Okay. So, I’m just going to wrap up now and hope­ful­ly, have a lit­tle bit of time if we have to some ques­tions. We’ve talked about this idea of, “Why should we go online? What are the ben­e­fits of online research?” I’d just remind you, speed, scale, rich. If any­one want to ask, those are the three things, three pil­lars why you should go online. We talked about the chal­lenges. I think a lot of these aren’t nec­es­sar­i­ly chal­lenges or they’re cer­tain­ly not chal­lenges that are exclu­sive to online research. These are the same chal­lenges every­one has in lab-based research. There are prac­ti­cal con­sid­er­a­tions before you get start­ed. You’ll need to pilot, pilot immense­ly. I can’t stress that enough.

Dr. Joshua Bal­sters:
You need to think about your exper­i­men­tal plat­form. Can it do what I need it to do? Does it have all the ethics require­ments that I need to do what I want to do? There are a lot of prac­ti­cal con­sid­er­a­tions to think about before you take your exper­i­ments online. And then final­ly, I’ve just giv­en you a hand­ful of exam­ples there about what you can do with online research and the way that online research in some cas­es, replic­a­ble of what we’re doing in the lab. In oth­er cas­es, it’s real­ly just expand­ing and chang­ing what we can do with research. So, thank you ever so much for join­ing me today. I’m hap­py to take any ques­tions you might have or if there’s any­thing you’re inter­est­ed in ask­ing right now.

Sophie Scott:
Josh, there are a cou­ple of ques­tions com­ing through on the Q&A. Can I ask you those?

Dr. Joshua Bal­sters:
Yeah, absolute­ly. Please, go ahead.

Sophie Scott:
So, some­body asked a very spe­cif­ic ques­tion. I think it’s an inter­est­ing one about the use of progress bars. Do you think there’s any pos­si­bil­i­ty that that might actu­al­ly have an influ­ence on they start get­ting bored or they go, “No, [inaudi­ble 00:36:17] ages to go. I’m near­ly at the end. I’m going to rush through it.” Do you know what I mean? What’s your feel­ing about the util­i­ty of those?

Dr. Joshua Bal­sters:
Yeah, that’s a very good point. I think it could get peo­ple drop­ping out soon­er, but I think in some ways that might be ben­e­fi­cial. I think it cer­tain­ly is nicer from the par­tic­i­pan­t’s point of view. The par­tic­i­pant says, “Oh, my God, I’ve been at this for ages and I’ve hard­ly scratched sur­face. Nev­er mind.” Well, if they have oth­er com­mit­ments in their life, they know the post­man’s due to come around that time, they think, “Oh, I can’t com­mit to this right now. I know, I’m going to have to engage in some­thing. I should drop out now. If it’s still avail­able lat­er on, I can come back to it.” In my opin­ion, it’s ben­e­fi­cial. Yeah, the pos­i­tives out­weigh the neg­a­tives in my opinion.

Sophie Scott:
There’s also just a ques­tion that’s appeared in the chat. I would ask if you can put these in the Q&A, please do so. Are there any eth­i­cal issues around using eye track­ing and mouse track­ing online? Is there any­thing par­tic­u­lar that we need to think about?

Dr. Joshua Bal­sters:
I don’t believe so, noth­ing that you would­n’t have with tra­di­tion­al exper­i­ments. I think with a lot of the eye track­ing soft­ware, you don’t have to wor­ry, because it does­n’t cap­ture any facial infor­ma­tion. With eye track­ing in par­tic­u­lar, what it tends to do is it cre­ates a face mod­el, which is a grid of where your face is, but it does­n’t cap­ture any­thing more than that. So, there’s no iden­ti­fi­able infor­ma­tion that you need to wor­ry about being sent to dif­fer­ent peo­ple. In the tools them­selves, I can’t think of any eth­i­cal issues. How you use them, obvi­ous­ly, that changes the situation.

Dr. Joshua Bal­sters:
Lat­er on, today, we’re doing some ses­sions on Mou­se­View, actu­al­ly, which is a won­der­ful alter­na­tive to eye track­ing. Well, I think one of the talks is about using this in sui­ci­dal research. So, obvi­ous­ly, when you use it in such a con­text, then there are eth­i­cal issues. But the tool in itself, I think, is fine. I don’t think there are any eth­i­cal issues to wor­ry about.

Sophie Scott:
There’s a very gen­er­al ques­tion here from Nina, which is, “When we decide to do an online research study, how can we be sure that the study will be reli­able and valid?”

Dr. Joshua Bal­sters:
Yeah, I think that’s a very big ques­tion. To be sure, I think, cer­tain­ty is quite dif­fi­cult to come by. I think you obvi­ous­ly do the best that you can. Like I said, I think the best tips are things we’ve men­tioned as well. Pilot­ing is incred­i­bly impor­tant and that’s going to tell you a lot of what you need to know. What I would also say is build­ing on the back of oth­er stud­ies, I stand by it. I hope I don’t sound hyp­o­crit­i­cal, because I know I’ve said and I do stand by what I said as well, that lab-based should­n’t be treat­ed as nec­es­sar­i­ly being the ground truth, but obvi­ous­ly, repli­ca­tion is fantastic.

Dr. Joshua Bal­sters:
So, tak­ing tried and test­ed exper­i­ments that have been done time after time, work­ing with those kinds of par­a­digms and build­ing up on those par­a­digms is a real­ly great thing. So, I think that’s real­ly one of the best things you can do. Look for those reli­able par­a­digms that proved them­selves, par­a­digms that have what I would call san­i­ty checks in them, things that you know should happen.

Sophie Scott:
I would com­plete­ly agree with you on that and this would apply to lab-based sci­ence as well. This is just doing sci­ence. It is a good idea to have things built in, that you know things are work­ing. I should expect to see this result based on what we already know. I think just very briefly, going to your point about the ground truth, I think you’re absolute­ly right. There’s no ground base­line, ground truth for lab-based stuff. Actu­al­ly, that’s true across psy­chol­o­gy. I think one of the things that’s quite use­ful to think about is you’re nev­er going to get to an answer if you just try one method.

Sophie Scott:
So, keep your focus broad­ened. Think about your research ques­tion. “What are dif­fer­ent ways you could address that ques­tion?” is also anoth­er impor­tant thing, because we don’t have these ground truths. So, hav­ing a broad­er per­spec­tive on it, I think, can real­ly help. Enough for me, though. There’s a qual­i­ta­tive ques­tion from Wal­ter. Do you have any con­sid­er­a­tions regard­ing selec­tion bias in online studies?

Dr. Joshua Bal­sters:
I think there’s been some inter­est­ing stuff out there about dif­fer­ent plat­forms. So, I’m think­ing there has been some research out there about Pro­lif­ic ver­sus MTurk and the indi­vid­u­als inside those plat­forms and how rep­re­sen­ta­tive they are. I know Pro­lif­ic has this real­ly great fea­ture now where I think you can actu­al­ly have what they refer to as a rep­re­sen­ta­tive pop­u­la­tion, which is quite a nice fea­ture to try and make sure that you com­bat selec­tion bias and you get the whole spec­trum of indi­vid­u­als with­in your exper­i­ment, which I think is a won­der­ful idea. I don’t think there are as many con­sid­er­a­tions to be hon­est. I would hope that with these tools, at least we’re mov­ing for­ward com­pared to lab-based research.

Dr. Joshua Bal­sters:
I think I can speak now to the lab-based work I’ve done in the past and I know that they’re quite homo­ge­neous pop­u­la­tion some­times. So, you can see from ques­tion­naire data look­ing at indi­vid­ual dif­fer­ences that there aren’t that many indi­vid­ual dif­fer­ences. You can see there’s a very strong bell curve with quite a nar­row pre­ci­sion to it. So, I don’t think there are as many con­sid­er­a­tions. My feel­ing that is a lit­tle bit of intu­ition and gut feel­ing is that we’re just get­ting away from selec­tion bias with online stud­ies, not com­plete­ly, but it’s a step in the right direction.

Sophie Scott:
So, some­thing that my PhD stu­dent, Alex­is McIn­tyre, has been look­ing at is vari­a­tion in rhythm pro­cess­ing. She has been using a rhythm pro­cess­ing task that she devel­oped for sev­er­al years. You expect to get a wide range of per­for­mance on it. When she moved her stud­ies online last year, she found that scores on that went real­ly high and much more nar­row. This is through Pro­lif­ic. When she fil­tered out for peo­ple who did­n’t play com­put­er games, she found actu­al­ly, you saw some­thing a lot more nor­mal, but actu­al­ly, they also dropped down by an order of 100 fold with the peo­ple who end­ed up in this study.

Sophie Scott:
So, I think some­times actu­al­ly, if you have peo­ple who play a lot of com­put­er games and cer­tain­ly, on her Pro­lif­ic study, there were a lot of these peo­ple. They were doing real­ly well. I think that it does­n’t take away from the results of the study, but it was inter­est­ing in terms of look­ing at indi­vid­ual vari­a­tion, where you would expect to see some­thing broad­er. There were some inter­est­ing rea­sons as to why that was and it’s just worth bear­ing that in mind. There isn’t going to be one truth for every­thing out there and it is def­i­nite­ly worth bear­ing in mind.

Sophie Scott:
How­ev­er, again, enough for me, maybe we’ve got time for one more ques­tion. So, sev­er­al points of this com­ing through. What hap­pens if it’s very dif­fi­cult to make your study more inter­est­ing? Would that affect repli­ca­tion? But also, if you can’t make it more inter­est­ing, so with a more engag­ing study, with a more eas­i­ly repli­cat­ed, but also, what should you do if you just can’t make it more fun?

Dr. Joshua Bal­sters:
I think unfor­tu­nate­ly, I think that’s very dif­fi­cult. I think you can make things more fun. I know, we’ve got one of our speak­ers lat­er on in the game ses­sion. He took the go/no-go task, which is a bor­ing study. It’s not an inter­est­ing task, espe­cial­ly after a few rep­e­ti­tions. But he took that and he put a whole skin over it and turned it into this amaz­ing game, where you had avatars. Instead of just being press red cir­cle, press green cir­cle, release, it became a drag­on and you had to har­vest gold to avoid the dragon.

Dr. Joshua Bal­sters:
So, I think there are a lot of real­ly great cre­ative solu­tions out there. I think you can do a lot just by chang­ing essen­tial­ly the skin of it. So, even if it’s not cre­at­ing a whole nar­ra­tive, just mak­ing some­thing that’s visu­al­ly more appeal­ing can actu­al­ly be real­ly use­ful, I think. So, I think there are a lot of oppor­tu­ni­ties to do that. I think if you real­ly, real­ly can’t do that, it’s very dif­fi­cult, I think you might end up just hav­ing to pay peo­ple. I think it’s the only oth­er option, exter­nal motivators.

Sophie Scott:
Thank you. Thank you.