The role of intrin­sic reward in ado­les­cent word learning


By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

Dr Saloni Krish­nan — Roy­al Hol­loway, Uni­ver­si­ty of London


We con­stant­ly learn words from con­text, even in the absence of explic­it rewards or feed­back. In adults, intrin­sic reward expe­ri­enced dur­ing word learn­ing is linked to a dopamin­er­gic cir­cuit in the brain. The expe­ri­ence of reward is also linked to enhance­ments in mem­o­ry for those words. In this reg­is­tered report, we exam­ine if ado­les­cents aged 10–18 report enhanced reward or enjoy­ment, and ensu­ing mem­o­ry ben­e­fits, when they suc­cess­ful­ly learn words from sen­tence con­text. We have test­ed 300+ chil­dren so far. In addi­tion to talk­ing about devel­op­men­tal change in reward expe­ri­enced dur­ing this peri­od, I will focus on some of the advan­tages and chal­lenges of con­duct­ing this research online.


Full Tran­script:
Over to you Saloni

Hel­lo, sor­ry, I just switched off my phone, which is why you prob­a­bly saw me dis­ap­pear­ing. So hope­ful­ly it’s not too noisy. And you can hear me nice­ly and clear­ly. So I have a ter­ri­ble honk­ing cold. It is, luck­i­ly not COVID. But I don’t rec­om­mend any sum­mer cold. So if I kind of doze off in the mid­dle, you’ll know why.

But I want­ed to tell you about some work that I’ve been car­ry­ing out in my lab over the last two or three years. What are the I guess I would define myself as a devel­op­men­tal cog­ni­tive neu­ro­sci­en­tist. And so in gen­er­al, I tell peo­ple that I like to put kids in scan­ners and find out what’s dif­fer­ent about their brains. Of course, over the last two years, because of a cer­tain virus, it’s been incred­i­bly dif­fi­cult to do any­thing like that. So a lot of my research has kind of piv­ot­ed to be online. And I’ve been try­ing to use, par­tic­u­lar­ly goril­la to kind of deliv­er and ask inter­est­ing new ques­tions that did­n’t nec­es­sar­i­ly involve putting chil­dren in scans.

So today, we’re talk­ing about the role of reward in ado­les­cent word learn­ing. So I guess I want to start off by mak­ing a pitch giv­en that this is a fair­ly diverse audi­ence in terms of why vocab­u­lary is impor­tant. What one rea­son is that we con­stant­ly encounter and learn new words through our lives. Brex­it, you prob­a­bly did­n’t real­ly know about Brex­it, pre 2015, COVID def­i­nite­ly did­n’t know about that pre 2020 kro­ner, we kind of encoun­tered these new words all the time. And we also know that chil­dren with read­ing and lan­guage dis­or­ders can strug­gle with word learn­ing, par­tic­u­lar­ly acquir­ing youth from log­i­cal forms, that we don’t think like vocab­u­lary knowl­edge are a pre­dic­tor of school suc­cess, and even­tu­al­ly also life achievement.

So it seems that fol­low this process around kind of acquir­ing new words is real­ly, real­ly impor­tant. And lots and lots of researchers focus on kind of dif­fer­ent aspects of this prob­lem. So most the­o­ries focus on how words on them. So like, how do you assign a spe­cif­ic word to a spe­cif­ic object? But one of the things that I real­ly care about and start­ed to care about a lot in the last few years is why do we learn new words? Why do we inher­ent­ly find this inter­est­ing, excit­ing? Why do we seek this out?

So I guess the ques­tion I’m going to be pos­ing today is word learn­ing intrin­si­cal­ly reward­ing, because it’s not that some­one comes and gives you five pounds, every time you encounter a new word, you learn a new word. So is there some­thing that is an intrin­sic dri­ver from us, and if so, is word learn­ing intrin­si­cal­ly reward­ing for all groups. So I’m going to be talk­ing about one par­tic­u­lar par­a­digm today, which is learn­ing words in con­text. This is not the only par­a­digm that I’ve used. But in the inter­est of time, I thought it might be bet­ter to kind of keep things kind of lim­it­ed to one par­a­digm that I could explain well, so this was a real­ly, real­ly clas­sic psy­cholin­guis­tic par­a­digm. It’s been around since the 70s. And it’s been used loads and loads of time. And the basic premise would be that we can learn the mean­ing of new words by the con­text that we occur in.

So if I give you the sen­tence view, coun­tries are now ruled by a side note site is not a real line of made up this mode. But the ques­tion is like, you might be able to kind of guess maybe what the mean­ing of this word is. And if you see the next sen­tence in the palace, the lips, the king and the stage, then you might have a read like a much bet­ter guess. And if you’re think­ing cyto­sine some­thing like a queen, then you will be right now lots of exam­ples of how we do this sort of in the wild.

And again, lots of peo­ple have been real­ly inter­est­ed in sort of the diver­si­ty of the sen­tence con­texts that use a num­ber of encoun­ters that you need, and so on. Well, one of the real­ly nice twists in this kind of par­a­digm, I think, came from one of my col­lab­o­ra­tors at NYU, Pablo Ripolles, who’s now an assis­tant pro­fes­sor at NYU. What he showed is in those kinds of par­a­digms, when you learn what mean­ing from con­text, this is asso­ci­at­ed with intrin­sic reward, and he demon­strat­ed this in mul­ti­ple ways.

So he showed this that in terms of peo­ple’s behav­iour­al train­ing, they tend­ed to share say that learn­ing these kinds of words was more plea­sur­able rel­a­tive to when they could­n’t extract mean­ing of words. Brain activ­i­ty and reward pro­cess­ing regions like the stria­tum was high­er when when you tend­ed to suc­cess­ful­ly learn the mean­ings of these words, unlike me, if you did­n’t. Impor­tant­ly, also, like you can also think about the effects of med­ica­tion. So a cou­ple of its real­ly nice work look­ing at lev­odopa, which had kind of increased reward respons­es and Risperi­done which is going to inhib­it risk reward respons­es. And he was able to show that when you give peo­ple love dopa can boost­ed their learn­ing when you give peo­ple Risperi­done it,

kind of decrease or foot­stone Vito­ria can decrease that learn­ing Um, but the oth­er thing that’s real­ly, real­ly impor­tant about this kind of expe­ri­ence of reward is that actu­al­ly expe­ri­enc­ing that reward seems to fuel word learn­ing. So you get mem­o­ry ben­e­fits from sort of encoun­ter­ing, and enjoy­ing learn­ing a new word. So you’re much more like­ly to remem­ber that Oba­ma had done a lot of this work in Ger­man. So all of his stud­ies have been in Ger­man, and all of them had been with kind of neu­rotyp­i­cal adults.

But in my work, I was real­ly inter­est­ed in try­ing to kind of open this up and poten­tial­ly look at peo­ple, chil­dren, chil­dren with dyslex­ia, etc. And obvi­ous­ly, being in the UK, my stud­ies are in Eng­lish. So we did a sort of a cou­ple of dif­fer­ent stud­ies to try and get a han­dle on cod­ing and build­ing on some of But­ler’s work. So the first study I want to kind of give you a pre­am­ble onto is just to sort of say, what is the influ­ence of modal­i­ty on the sort of effects. And so he’ll prob­a­bly done all of his work look­ing at peo­ple read­ing these kinds of sentences.

We also know from the lit­er­a­ture that actu­al­ly when peo­ple lis­ten to the same kind of sen­tences, they also kind of extract words in the same way extract mean­ing from the same, right? So we hypoth­e­sise that learn­ing new words would be intrin­si­cal­ly reward­ing in both cre­at­ing con­di­tions as well as lis­ten­ing con­di­tions. And we sort of said that, yes, intrin­sic reward will also be asso­ci­at­ed with mem­o­ry benefits.

Now, this was actu­al­ly a project that under­grad stu­dents are going Hol­loway car­ry out in 2020 2021. So this is obvi­ous­ly some­thing they have to do online. And it was pret­ty amaz­ing for us that goril­la exist­ed, that we could just go off and get this data because not a prob­lem at all.

So I want to tell you a lit­tle bit about the par­a­digm. So as it as I’d hoped, demon­strate to you that you have these kinds of tem­plates, sen­tences, so you sort of have sen­tences where you encounter anoth­er word at the end, and they’re kind of paired up so you can extract the mean­ing­ful word. Gen­er­al­ly, after peo­ple encoun­tered these words, they would have to type in a guess about what they thought the word was. And they would also have to give us a num­ber of read­ings. So like, How con­fi­dent were they? How tired were they can how much they enjoyed that par­tic­u­lar sentence.

Um, inter­est­ing­ly, we also had a real­ly good con­trol, which is m minus sen­tences. So in these sen­tences, these are basi­cal­ly jum­bled up sen­tences from anoth­er set of n plus sen­tences. But here, you can­not extract the mean­ing of a new word in this kind of real­ly good exper­i­men­tal con­trol device, you’d have things like John need­ed a bat­tery for his bam­boo, the teacher wrote the date on the bam­boo. So it’s real­ly kind of hard to extract the mean­ing for the word sample.

We pre­sent­ed these par­a­digms in in a Kore­an con­di­tion or lis­ten­ing con­di­tion, we also have a read­ing and lis­ten­ing con­di­tion, which is sort of akin to kind of think­ing about sub­ti­tles. And this was a lon­gi­tu­di­nal study in a real­ly, real­ly sim­ple way we’re learn­ing study, because 24 hours lat­er, peo­ple have to com­plete a recog­ni­tion task on these words. It’s also not as easy as it looks. So with that, because what we actu­al­ly did was we did­n’t present these words sequen­tial­ly. So you would tend to get a block of sen­tences. So you would encounter four dif­fer­ent words. And then you would kind of encounter the pair for those words a lit­tle bit lat­er. So it was­n’t as triv­ial as it looks like in my kind of schematic.

But yeah, the main thing is we test­ed native Eng­lish speak­ing adults. And I can say more about this, but just to keep it brief. For the two of them this read­ing par­a­digm 36 Did the lis­ten­ing par­a­digm. And 34 did the read­ing and lis­ten­ing par­a­digm. And I think our goal was just to get to above 30 In all of these con­di­tions. So we were fair­ly hap­py with these num­bers. So what would it be fine?

Well, so I want you to look at the kind of M P sen­tences which are shown in blue over here, enjoy­ment is on the x axis and learn­ing is on the y axis. And what you can very clear­ly see is that when peo­ple report greater enjoy­ment, they’re also show­ing Kore­an learn­ing in the NP case. And this isn’t as much the case in the MM case, which is those kinds of con­trol sen­tences that I talked about.

So in gen­er­al, when you can suc­cess­ful­ly learn a new word, but not in kind of these exper­i­men­tal con­trol con­trols, you do read things as being more plea­sur­able. I just want­ed to over­lay those same kind of curves, one on top of the oth­er. So this is only for the MP con­di­tion where you can extract the mean­ing of the word. And I just want­ed to show you how sim­i­lar this was across the three con­di­tions. So lis­ten­ing, read­ing, read­ing and listening.

One of the oth­er things I want­ed to show you is that on that recog­ni­tion task again, so this is mem­o­ry rather than learn­ing on day one, and again, the more you enjoyed learn­ing that word and deed one, the bet­ter you are at remem­ber­ing it. And this was record­ed to the con­di­tions of lis­ten­ing, read­ing or read­ing or lis­ten­ing. My very sim­ple kind of take­away from that work is just to say that actu­al­ly, intrin­sic reward is asso­ci­at­ed with mem­o­ry ben­e­fits. So we kind of repli­cat­ed what Pablo had already shown but kind of expand­ed it into these new modal­i­ty con­di­tions. So this we felt set us up real­ly well.

And in exper­i­ment two, we decid­ed to test chil­dren. And we decid­ed specif­i­cal­ly that we were going to inves­ti­gate change with devel­op­ment, the age group that we want­ed to focus on with lat­er lessons, and this is because of some work from Lisa Knoll, which had sug­gest­ed that when you train old­er ado­les­cents on kind of rela­tion­al rea­son­ing and numeros­i­ty dis­crim­i­na­tion, old­er ado­les­cents show the kind of great­est jump in per­for­mance, and adults and younger ado­les­cents to chil­dren learn­ing, but they weren’t as pro­nounced as the game.

So Lisa Knoll kind of sug­gest­ed late ado­les­cence might offer a win­dow of oppor­tu­ni­ty for edu­ca­tion­al inter­ven­tions. There’s also some work from the kind of neu­ro­science domain, which is kind of sug­gest­ing that straight line activ­i­ty is pre­dic­tive of lat­er learn­ing per­for­mance. And that’s the gen­er­al idea that increased reward sen­si­tiv­i­ty would lead to increase moti­va­tion and salience. And then this recruits sys­tems for learn­ing. So actu­al­ly, this would be a real­ly impor­tant win­dow ear­li­er, our lessons might offer a real­ly impor­tant win­dow for word­ing as well, and so this is what we decid­ed to test in this par­tic­u­lar reg­is­tered report, where we test on 10 to 18 year olds with the idea of basi­cal­ly sor­ry, and this is sor­ry, this I should say, this is work con­duct­ed with RAs in my lab at the time and with our Amri­ta Bains, Analise Bar­ber and Tam Nell. And the hypoth­e­sis we were look­ing at was that learn­ing new words were read­ing will be intrin­si­cal­ly reward­ing and ear­ly devel­op­men­tal stage.

When we did this work, this hypoth­e­sis had­n’t been test­ed in chil­dren and ado­les­cents. But the kind of premise of the fact that word learn­ing would be record­ing, we would very much expect to see this ear­ly in devel­op­ment if tru­ly reward is hav­ing mean­ing­ful inter­ac­tion to language.

We also hypoth­e­sised that intrin­sic reward would be relat­ed to mem­o­ry ben­e­fits and poten­tial new and appears exper­i­ment. But if only we had a devel­op­men­tal hypoth­e­sis that the kind of reward you expe­ri­ence and the kind of learn­ing from reward would increase in age and all that sort of peak in late ado­les­cence, which we thought would be some­where between 14 to 16, or 16, to 18 years. So this is a reg­is­tered report.

So as you know, we kind of pre reg­is­tered the plans, and we got review­ers, and this has been accept­ed in devel­op­men­tal sci­ence. And for this par­tic­u­lar project and report­ing data from 345, native Eng­lish speak­ing chil­dren, aged 10 to 18. All of this data was col­lect­ed dur­ing the pan­dem­ic. And basi­cal­ly, our goal to get above 84 Chil­dren are based on our pre reg­is­tra­tion and all of these brack­ets that we are ready for 10 to 12 year olds, 14 year olds, 1416 and 16 to 18.

I want to show you a cou­ple of kind of emerg­ing results, I still fin­ished writ­ing this up and sub­mit the stage to sub­mis­sion. So this sort of work in progress, but some very, very clear mes­sages right so far from the data.

The first is like we saw with the behav­iours that I just pre­sent­ed to you learn­ing new words as report­ing. So we here we only did the read­ing con­di­tion. But just like before, you can see that par­tic­u­lar­ly in the kind of m plus sit­u­a­tion when you kind of enjoy when you report greater enjoy­ment, you tend to get greater learn­ing. And you see some indi­ca­tion of this even an M minus con­di­tion in chil­dren aged 10 to 18.

What was the most sur­pris­ing though, and what peo­ple weren’t expect­ing to find is that there were no, there was no effect of age. And so in some ways, hypoth­e­sis one, which is the fact that we would repli­cate this data in chil­dren is kind of sur­pris­ing, because if you had any expe­ri­ence of reg­is­tered reports, what tends to hap­pen is that you try to repli­cate a find­ing, and it doesn’t.

Actu­al­ly, we did repli­cate the first find­ing, but we did­n’t find any evi­dence about kind of sec­ond hypoth­e­sis. So it was­n’t that 14 to 16 year olds or 16 to 18 year olds showed greater reward greater sort of learn­ing due to reward. Or kind of the reg­is­tered hypoth­e­sis, also, we don’t see real­ly good evi­dence for so we don’t real­ly observe mem­o­ry ben­e­fits. And I think this is par­tial­ly for two reasons.

So basi­cal­ly, what we would have expect­ed to find is that enjoy­ment was pre­dic­tive of kind of accu­ra­cy on the sec­ond day of learn­ing. But one of the prob­lems may have been that we made the task real­ly easy. So for chil­dren, what we did is we did­n’t have the com­pli­cat­ed block­ing struc­ture that I showed you in the pre­vi­ous exper­i­ment, we did give them sequen­tial sen­tences, because we thought it would be too much for chil­dren. And in doing so what we may have done is actu­al­ly the learn­ing demands of the task too easy and not being able to observe these kinds of mem­o­ry ben­e­fits. That’s one idea potentially.

So I want to kind of sum­marise what I’ve told you so far, I’m in terms of the sci­ence so chil­dren and adults do find word learn­ing intrin­si­cal­ly. And we want to sug­gest that when tasks are chal­leng­ing enough that we find that reward is asso­ci­at­ed with green and we’re doing some future work on this. So we’re now look­ing at the neur­al basis of reward for word learn­ing in your typ­i­cal chil­dren and chil­dren with dyslex­ia. And we’re also start­ing to under­stand­ed the process­es of moti­va­tion going beyond the expe­ri­ence of reward alone. And this is work being car­ried out on whol­ly by my post­doc Desi.

But as you allud­ed to, I want­ed to talk a lit­tle bit about what is being online done for us. I want­ed to men­tion some pros and some cons and per­haps give you some very quick tips and tricks. So in terms of the pros, the reach of exper­i­ment, like I men­tioned that we test­ed some­thing like 345 data, chil­dren, for I exper­i­ment, there is no way I would have been able to do that, based on kind of strict lab set­tings, it just could not have hap­pened, I believe, to the fact that this was pos­si­ble in a pan­dem­ic mul­ti­ple times, there was again, no way that we could have gone into schools and got data in any oth­er way.

Design­ing these tests was real­ly, real­ly easy. So that ease of design and kind of get­ting under­grad­u­ate project stu­dents in more than doing this and so on was real­ly impor­tant. shar­ing and col­lab­o­ra­tion was eas­i­er. My  kind of col­lab­o­ra­tor was at NYU, but it was real­ly easy to kind of just reach out to him and say, here’s the pow­er, and you can look at it, you can play with it, give me your com­ments. And of course, goril­las can open mate­ri­als makes open sci­ence and shar­ing an opti­cal­ly rec­ol­lec­tion also much easier.

I think there are some cons, we while con­duct­ing this exper­i­ment online, we realised that we were at some point get­ting lots of bots. And that was prob­lem­at­ic, or you know, peo­ple who were just kind of pick­ing up our ads online and doing stuff, we did­n’t have a check for age in any way. So obvi­ous­ly, you’re test­ing a child in real life, you can see that they’re a child, one of the things that wor­ried us when maybe peo­ple are kind of par­tic­i­pat­ing, and there were adults, but they were just pre­tend­ing to be chil­dren to get vouch­ers from US gov­ern­ment. Obvi­ous­ly, one of the things that you get when you are watch­ing some­one do your task is a lot of insight into par­tic­i­pant behav­iour, we did­n’t quite have that.

And that we did have a tech prob­lem and one of the exper­i­ments and one of the par­tic­u­lar tasks that we were using need­ed a key­board response. But appar­ent­ly lots of chil­dren nowa­days use tablets to do every­thing. So they would come to this task and they would fail. And then basi­cal­ly, we need­ed to set up mul­ti­ple ver­sions, includ­ing a whole new day to exper­i­ment just in case they had failed this one par­tic­u­lar child­care cutouts.

And of course, there is a bit more cost to run­ning these things online then. So for exam­ple, if you run things in schools, you can just give peo­ple stick­ers. If it’s online, you need to tempt peo­ple with vouchers.

Tips and tricks, we sug­gest lots and lots of exper­i­men­tal con­trols we had in our exper­i­ment, accu­ra­cy checks, atten­tion checks, we kind of as I allud­ed to run a new ver­sion of just the day 2 exper­i­ment. And I think the main thing that I would say is be respon­sive to par­tic­i­pants and ide­al­ly, run stud­ies in small batch­es so that you can touch up email with your par­tic­i­pants and talk to them. That’s all Sor­ry for going over time, Jo. That’s me done thank you.

That was fab­u­lous. Lonely.

Sor­ry. I was try­ing to unmute myself and fail­ing to click the but­ton. That was absolute­ly bril­liant. I am so impressed with the work that you’ve done. I’m just going to have to change my view here because I can’t bear look­ing at myself when I’m speaking.

Gosh, that was a lot of kids that you’ve recruit­ed. Those are big sam­ples. How did you get them?

Oh, well, that was actu­al­ly recruit­ed even more than I pre­sent­ed here. So I think over­all, we prob­a­bly test­ed some­thing like 540 kids, and then actu­al­ly because of the kind of exper­i­men­tal con­trols that you have you drop peo­ple out. So lots and lots of kids, I guess, schools like we send a lot of emails to schools to be like, Please, can you put this up? And then lots schools are very hap­py to be like, you know, ask­ing us to devote any time. Peo­ple can just go do it online on this par­tic­u­lar link. That was great. What else did we do? Lots and lots of emails, lots and lots of social media kind of pres­sure and peo­ple. I mean, I think part of it is like when you do a reg­is­tered report you’ve com­mit­ted to doing the num­bers just

have to get out and get them. Yes, we’re get­ting the last five hard­er than get­ting the first five.

Actu­al­ly. So we realise at some point that like the, for some rea­son, we got lots of 16 to 18 euros, we got lots of 10 euros, but the kind of in between group that 12 to 14 euros, we had this lit­tle graph where we want­ed to see the num­bers going up and for some rea­son we just nev­er got 12 to 14 year olds. Get­ting those last few 12 to 14 euros was the most chal­leng­ing bit. But yeah, it works out real­ly beau­ti­ful­ly. And

so I think there are some ques­tions com­ing in the q&a. And if you’re in the audi­ence, and you’ve got a ques­tion, please do dump it in the q&a. Now, one more ques­tion from me. What? What types of checks? Did you put into, like san­i­ty checks on your data did you put into to to make sure you’re get­ting sen­si­ble data quality?

Yeah. So with Luck­i­ly, this was a reg­is­tered report. So like, a review is also rec­om­mend­ed, and your san­i­ty checks. There are kind of one san­i­ty check was part of our reg­is­tered report. And we basi­cal­ly said that if we’re real­ly see­ing chil­dren, and this prob­a­bly also answers, some of Matt Davis’s ques­tions in the q&a, is that we would expect the kind of extract­ing mean­ing would be increas­ing with age. And we find a very, very strong rela­tion­ship that so basi­cal­ly, accu­ra­cy in the m plus con­di­tion very strong­ly goes up by age. So for when you’re 10, you’re a lit­tle bit worse at answer­ing those ques­tions. When you’re 18, you’re much bet­ter answer­ing those ques­tions. And that is def­i­nite­ly what we were expect­ed. So that that’s kind of helpful.

And at some point, we hit this vein of peo­ple on Face­book, who were like, email­ing us. And you know, see­ing those emails is real­ly reas­sur­ing, because there’s a cer­tain way a par­ent com­mu­ni­cates. And you can def­i­nite­ly go yes, that def­i­nite­ly feels like an inter­ac­tion with a real client. There are some where we were just not sure. And then we have kind of a lot of them, luck­i­ly, fair kind of reten­tion checks. So we had stuff like we need­ed to have kids com­plete with a cer­tain lev­el of accu­ra­cy. We also have embed­ded atten­tion checks in there. We’ve also got under con­trol. So we’ve asked peo­ple about best sleep. And we’ve asked peo­ple about kind of we did this task Jason Yeat­man group called roar. So we have raw data. And again, on the raw, we can see, which is a lex­i­cal deci­sion task for those of you that means you can again, see the accu­ra­cy is increas­ing with age and reac­tion times kind of decreas­ing, real­ly, so kind of a good set of checks in that.

Very, very cool. Yes, that sounds like a com­pre­hen­sive set of san­i­ty checks. do you what do you want to give a more com­pre­hen­sive esti­mate? Or have you answered that suf­fi­cient­ly? The ques­tion was, do you think it’s pos­si­ble that the null effects of age can be explained by dis­hon­est age reports? I think you’ve just about cov­ered that. So I think, yeah, I

don’t think it would be dis­hon­est. I mean, I’m sure that like every sin­gle per­son in the data we can’t nec­es­sar­i­ly ver­i­fy. And one thing we should have real­ly done is kind of pos­si­bly enforce a mech­a­nism that peo­ple could only do the task ones because I was just like, oh, per­son with the same email, who’s done my task can be anoth­er lit­tle bit of kind of san­i­ty check kind of weed­ing out stuff, but I’m sure if like some­one was smarter and use like a sec­ond email, that’s a bit hard­er to claim

this thing. Did you con­sid­er try­ing to get like, and use the video record­ing zone just to get a screen­shot of the per­son? Or would you not have got that past ethics?

I think ethics would have been chal­leng­ing. I think a few peo­ple have been talk­ing about like get­ting audio record­ings dur­ing con­sent, because obvi­ous­ly, your voice is a real­ly good cue. If your agent if you say some­thing like I can send an obvi­ous­ly, you know that, you know, it’s a native Eng­lish speak­er who gives you hon­est­ly, like, you know, I mean, I know you don’t nec­es­sar­i­ly face some of these issues, but I would be inher­ent­ly extreme­ly sur­prised. And I’m sure we have a cou­ple of these sort of things in the main data set. But I’d be very sur­prised if that was the entire explanation.

Bril­liant. So only could you just go back to your side of top tips. And while cer­tain­ly he’s doing that, Neil, could you get your slides ready for shar­ing in a minute? And what I just want­ed to ask the audi­ence that we’ve got here today, which of the tips and tricks was most use­ful for you today? What What was the les­son you most need­ed to hear? Was it about learn­ing lots of exper­i­men­tal con­trols and sep­a­rat­ing accu­ra­cy checks and atten­tion sets? Or was it about need­ing to be respon­sive when things go wrong? And maybe cre­at­ing a dif­fer­ent ver­sion of exper­i­ment? That was some­thing you might not con­sid­er? Or was it the being respon­sive to par­tic­i­pants, or in these the advice on recruit­ing chil­dren? Bril­liant, thank you so much. It’s just won­der­ful. It’s real­ly great if you are will­ing to be gen­er­ous to Saloni and give the feed­back that’s been most help­ful to you, because it helps Saloni know, she’s done a good job. And it’s dif­fi­cult giv­ing a talk to a room of com­plete­ly blank screens. And that inter­ac­tion from you guys is just real­ly reward­ing to know that we are help­ing you learn, learn mes­sages that are help­ful that you’re going to take into your research and will make your lives bet­ter. Now with that Saloni thank you so much. You were won­der­ful. Thank you for start­ing us and set­ting the note the bar so high.


Get on the Registration List

BeOnline is the conference to learn all about online behavioral research. It's the ideal place to discover the challenges and benefits of online research and to learn from pioneers. If that sounds interesting to you, then click the button below to register for the 2023 conference on Thursday July 6th. You will be the first to know when we release new content and timings for BeOnline 2023.

With thanks to our sponsors!