The role of intrin­sic reward in ado­les­cent word learning

0:00
Over to you Saloni

0:02
Hel­lo, sor­ry, I just switched off my phone, which is why you prob­a­bly saw me dis­ap­pear­ing. So hope­ful­ly it’s not too noisy. And you can hear me nice­ly and clear­ly. So I have a ter­ri­ble honk­ing cold. It is, luck­i­ly not COVID. But I don’t rec­om­mend any sum­mer cold. So if I kind of doze off in the mid­dle, you’ll know why.

But I want­ed to tell you about some work that I’ve been car­ry­ing out in my lab over the last two or three years. What are the I guess I would define myself as a devel­op­men­tal cog­ni­tive neu­ro­sci­en­tist. And so in gen­er­al, I tell peo­ple that I like to put kids in scan­ners and find out what’s dif­fer­ent about their brains. Of course, over the last two years, because of a cer­tain virus, it’s been incred­i­bly dif­fi­cult to do any­thing like that. So a lot of my research has kind of piv­ot­ed to be online. And I’ve been try­ing to use, par­tic­u­lar­ly goril­la to kind of deliv­er and ask inter­est­ing new ques­tions that did­n’t nec­es­sar­i­ly involve putting chil­dren in scans.

So today, we’re talk­ing about the role of reward in ado­les­cent word learn­ing. So I guess I want to start off by mak­ing a pitch giv­en that this is a fair­ly diverse audi­ence in terms of why vocab­u­lary is impor­tant. What one rea­son is that we con­stant­ly encounter and learn new words through our lives. Brex­it, you prob­a­bly did­n’t real­ly know about Brex­it, pre 2015, COVID def­i­nite­ly did­n’t know about that pre 2020 kro­ner, we kind of encoun­tered these new words all the time. And we also know that chil­dren with read­ing and lan­guage dis­or­ders can strug­gle with word learn­ing, par­tic­u­lar­ly acquir­ing youth from log­i­cal forms, that we don’t think like vocab­u­lary knowl­edge are a pre­dic­tor of school suc­cess, and even­tu­al­ly also life achievement.

So it seems that fol­low this process around kind of acquir­ing new words is real­ly, real­ly impor­tant. And lots and lots of researchers focus on kind of dif­fer­ent aspects of this prob­lem. So most the­o­ries focus on how words on them. So like, how do you assign a spe­cif­ic word to a spe­cif­ic object? But one of the things that I real­ly care about and start­ed to care about a lot in the last few years is why do we learn new words? Why do we inher­ent­ly find this inter­est­ing, excit­ing? Why do we seek this out?

So I guess the ques­tion I’m going to be pos­ing today is word learn­ing intrin­si­cal­ly reward­ing, because it’s not that some­one comes and gives you five pounds, every time you encounter a new word, you learn a new word. So is there some­thing that is an intrin­sic dri­ver from us, and if so, is word learn­ing intrin­si­cal­ly reward­ing for all groups. So I’m going to be talk­ing about one par­tic­u­lar par­a­digm today, which is learn­ing words in con­text. This is not the only par­a­digm that I’ve used. But in the inter­est of time, I thought it might be bet­ter to kind of keep things kind of lim­it­ed to one par­a­digm that I could explain well, so this was a real­ly, real­ly clas­sic psy­cholin­guis­tic par­a­digm. It’s been around since the 70s. And it’s been used loads and loads of time. And the basic premise would be that we can learn the mean­ing of new words by the con­text that we occur in.

So if I give you the sen­tence view, coun­tries are now ruled by a side note site is not a real line of made up this mode. But the ques­tion is like, you might be able to kind of guess maybe what the mean­ing of this word is. And if you see the next sen­tence in the palace, the lips, the king and the stage, then you might have a read like a much bet­ter guess. And if you’re think­ing cyto­sine some­thing like a queen, then you will be right now lots of exam­ples of how we do this sort of in the wild.

And again, lots of peo­ple have been real­ly inter­est­ed in sort of the diver­si­ty of the sen­tence con­texts that use a num­ber of encoun­ters that you need, and so on. Well, one of the real­ly nice twists in this kind of par­a­digm, I think, came from one of my col­lab­o­ra­tors at NYU, Pablo Ripolles, who’s now an assis­tant pro­fes­sor at NYU. What he showed is in those kinds of par­a­digms, when you learn what mean­ing from con­text, this is asso­ci­at­ed with intrin­sic reward, and he demon­strat­ed this in mul­ti­ple ways.

So he showed this that in terms of peo­ple’s behav­iour­al train­ing, they tend­ed to share say that learn­ing these kinds of words was more plea­sur­able rel­a­tive to when they could­n’t extract mean­ing of words. Brain activ­i­ty and reward pro­cess­ing regions like the stria­tum was high­er when when you tend­ed to suc­cess­ful­ly learn the mean­ings of these words, unlike me, if you did­n’t. Impor­tant­ly, also, like you can also think about the effects of med­ica­tion. So a cou­ple of its real­ly nice work look­ing at lev­odopa, which had kind of increased reward respons­es and Risperi­done which is going to inhib­it risk reward respons­es. And he was able to show that when you give peo­ple love dopa can boost­ed their learn­ing when you give peo­ple Risperi­done it,

4:55
kind of decrease or foot­stone Vito­ria can decrease that learn­ing Um, but the oth­er thing that’s real­ly, real­ly impor­tant about this kind of expe­ri­ence of reward is that actu­al­ly expe­ri­enc­ing that reward seems to fuel word learn­ing. So you get mem­o­ry ben­e­fits from sort of encoun­ter­ing, and enjoy­ing learn­ing a new word. So you’re much more like­ly to remem­ber that Oba­ma had done a lot of this work in Ger­man. So all of his stud­ies have been in Ger­man, and all of them had been with kind of neu­rotyp­i­cal adults.

But in my work, I was real­ly inter­est­ed in try­ing to kind of open this up and poten­tial­ly look at peo­ple, chil­dren, chil­dren with dyslex­ia, etc. And obvi­ous­ly, being in the UK, my stud­ies are in Eng­lish. So we did a sort of a cou­ple of dif­fer­ent stud­ies to try and get a han­dle on cod­ing and build­ing on some of But­ler’s work. So the first study I want to kind of give you a pre­am­ble onto is just to sort of say, what is the influ­ence of modal­i­ty on the sort of effects. And so he’ll prob­a­bly done all of his work look­ing at peo­ple read­ing these kinds of sentences.

We also know from the lit­er­a­ture that actu­al­ly when peo­ple lis­ten to the same kind of sen­tences, they also kind of extract words in the same way extract mean­ing from the same, right? So we hypoth­e­sise that learn­ing new words would be intrin­si­cal­ly reward­ing in both cre­at­ing con­di­tions as well as lis­ten­ing con­di­tions. And we sort of said that, yes, intrin­sic reward will also be asso­ci­at­ed with mem­o­ry benefits.

Now, this was actu­al­ly a project that under­grad stu­dents are going Hol­loway car­ry out in 2020 2021. So this is obvi­ous­ly some­thing they have to do online. And it was pret­ty amaz­ing for us that goril­la exist­ed, that we could just go off and get this data because not a prob­lem at all.

So I want to tell you a lit­tle bit about the par­a­digm. So as it as I’d hoped, demon­strate to you that you have these kinds of tem­plates, sen­tences, so you sort of have sen­tences where you encounter anoth­er word at the end, and they’re kind of paired up so you can extract the mean­ing­ful word. Gen­er­al­ly, after peo­ple encoun­tered these words, they would have to type in a guess about what they thought the word was. And they would also have to give us a num­ber of read­ings. So like, How con­fi­dent were they? How tired were they can how much they enjoyed that par­tic­u­lar sentence.

Um, inter­est­ing­ly, we also had a real­ly good con­trol, which is m minus sen­tences. So in these sen­tences, these are basi­cal­ly jum­bled up sen­tences from anoth­er set of n plus sen­tences. But here, you can­not extract the mean­ing of a new word in this kind of real­ly good exper­i­men­tal con­trol device, you’d have things like John need­ed a bat­tery for his bam­boo, the teacher wrote the date on the bam­boo. So it’s real­ly kind of hard to extract the mean­ing for the word sample.

7:41
We pre­sent­ed these par­a­digms in in a Kore­an con­di­tion or lis­ten­ing con­di­tion, we also have a read­ing and lis­ten­ing con­di­tion, which is sort of akin to kind of think­ing about sub­ti­tles. And this was a lon­gi­tu­di­nal study in a real­ly, real­ly sim­ple way we’re learn­ing study, because 24 hours lat­er, peo­ple have to com­plete a recog­ni­tion task on these words. It’s also not as easy as it looks. So with that, because what we actu­al­ly did was we did­n’t present these words sequen­tial­ly. So you would tend to get a block of sen­tences. So you would encounter four dif­fer­ent words. And then you would kind of encounter the pair for those words a lit­tle bit lat­er. So it was­n’t as triv­ial as it looks like in my kind of schematic.

But yeah, the main thing is we test­ed native Eng­lish speak­ing adults. And I can say more about this, but just to keep it brief. For the two of them this read­ing par­a­digm 36 Did the lis­ten­ing par­a­digm. And 34 did the read­ing and lis­ten­ing par­a­digm. And I think our goal was just to get to above 30 In all of these con­di­tions. So we were fair­ly hap­py with these num­bers. So what would it be fine?

Well, so I want you to look at the kind of M P sen­tences which are shown in blue over here, enjoy­ment is on the x axis and learn­ing is on the y axis. And what you can very clear­ly see is that when peo­ple report greater enjoy­ment, they’re also show­ing Kore­an learn­ing in the NP case. And this isn’t as much the case in the MM case, which is those kinds of con­trol sen­tences that I talked about.

So in gen­er­al, when you can suc­cess­ful­ly learn a new word, but not in kind of these exper­i­men­tal con­trol con­trols, you do read things as being more plea­sur­able. I just want­ed to over­lay those same kind of curves, one on top of the oth­er. So this is only for the MP con­di­tion where you can extract the mean­ing of the word. And I just want­ed to show you how sim­i­lar this was across the three con­di­tions. So lis­ten­ing, read­ing, read­ing and listening.

One of the oth­er things I want­ed to show you is that on that recog­ni­tion task again, so this is mem­o­ry rather than learn­ing on day one, and again, the more you enjoyed learn­ing that word and deed one, the bet­ter you are at remem­ber­ing it. And this was record­ed to the con­di­tions of lis­ten­ing, read­ing or read­ing or lis­ten­ing. My very sim­ple kind of take­away from that work is just to say that actu­al­ly, intrin­sic reward is asso­ci­at­ed with mem­o­ry ben­e­fits. So we kind of repli­cat­ed what Pablo had already shown but kind of expand­ed it into these new modal­i­ty con­di­tions. So this we felt set us up real­ly well.

And in exper­i­ment two, we decid­ed to test chil­dren. And we decid­ed specif­i­cal­ly that we were going to inves­ti­gate change with devel­op­ment, the age group that we want­ed to focus on with lat­er lessons, and this is because of some work from Lisa Knoll, which had sug­gest­ed that when you train old­er ado­les­cents on kind of rela­tion­al rea­son­ing and numeros­i­ty dis­crim­i­na­tion, old­er ado­les­cents show the kind of great­est jump in per­for­mance, and adults and younger ado­les­cents to chil­dren learn­ing, but they weren’t as pro­nounced as the game.

So Lisa Knoll kind of sug­gest­ed late ado­les­cence might offer a win­dow of oppor­tu­ni­ty for edu­ca­tion­al inter­ven­tions. There’s also some work from the kind of neu­ro­science domain, which is kind of sug­gest­ing that straight line activ­i­ty is pre­dic­tive of lat­er learn­ing per­for­mance. And that’s the gen­er­al idea that increased reward sen­si­tiv­i­ty would lead to increase moti­va­tion and salience. And then this recruits sys­tems for learn­ing. So actu­al­ly, this would be a real­ly impor­tant win­dow ear­li­er, our lessons might offer a real­ly impor­tant win­dow for word­ing as well, and so this is what we decid­ed to test in this par­tic­u­lar reg­is­tered report, where we test on 10 to 18 year olds with the idea of basi­cal­ly sor­ry, and this is sor­ry, this I should say, this is work con­duct­ed with RAs in my lab at the time and with our Amri­ta Bains, Analise Bar­ber and Tam Nell. And the hypoth­e­sis we were look­ing at was that learn­ing new words were read­ing will be intrin­si­cal­ly reward­ing and ear­ly devel­op­men­tal stage.

When we did this work, this hypoth­e­sis had­n’t been test­ed in chil­dren and ado­les­cents. But the kind of premise of the fact that word learn­ing would be record­ing, we would very much expect to see this ear­ly in devel­op­ment if tru­ly reward is hav­ing mean­ing­ful inter­ac­tion to language.

We also hypoth­e­sised that intrin­sic reward would be relat­ed to mem­o­ry ben­e­fits and poten­tial new and appears exper­i­ment. But if only we had a devel­op­men­tal hypoth­e­sis that the kind of reward you expe­ri­ence and the kind of learn­ing from reward would increase in age and all that sort of peak in late ado­les­cence, which we thought would be some­where between 14 to 16, or 16, to 18 years. So this is a reg­is­tered report.

So as you know, we kind of pre reg­is­tered the plans, and we got review­ers, and this has been accept­ed in devel­op­men­tal sci­ence. And for this par­tic­u­lar project and report­ing data from 345, native Eng­lish speak­ing chil­dren, aged 10 to 18. All of this data was col­lect­ed dur­ing the pan­dem­ic. And basi­cal­ly, our goal to get above 84 Chil­dren are based on our pre reg­is­tra­tion and all of these brack­ets that we are ready for 10 to 12 year olds, 14 year olds, 1416 and 16 to 18.

12:38
I want to show you a cou­ple of kind of emerg­ing results, I still fin­ished writ­ing this up and sub­mit the stage to sub­mis­sion. So this sort of work in progress, but some very, very clear mes­sages right so far from the data.

The first is like we saw with the behav­iours that I just pre­sent­ed to you learn­ing new words as report­ing. So we here we only did the read­ing con­di­tion. But just like before, you can see that par­tic­u­lar­ly in the kind of m plus sit­u­a­tion when you kind of enjoy when you report greater enjoy­ment, you tend to get greater learn­ing. And you see some indi­ca­tion of this even an M minus con­di­tion in chil­dren aged 10 to 18.

What was the most sur­pris­ing though, and what peo­ple weren’t expect­ing to find is that there were no, there was no effect of age. And so in some ways, hypoth­e­sis one, which is the fact that we would repli­cate this data in chil­dren is kind of sur­pris­ing, because if you had any expe­ri­ence of reg­is­tered reports, what tends to hap­pen is that you try to repli­cate a find­ing, and it doesn’t.

Actu­al­ly, we did repli­cate the first find­ing, but we did­n’t find any evi­dence about kind of sec­ond hypoth­e­sis. So it was­n’t that 14 to 16 year olds or 16 to 18 year olds showed greater reward greater sort of learn­ing due to reward. Or kind of the reg­is­tered hypoth­e­sis, also, we don’t see real­ly good evi­dence for so we don’t real­ly observe mem­o­ry ben­e­fits. And I think this is par­tial­ly for two reasons.

So basi­cal­ly, what we would have expect­ed to find is that enjoy­ment was pre­dic­tive of kind of accu­ra­cy on the sec­ond day of learn­ing. But one of the prob­lems may have been that we made the task real­ly easy. So for chil­dren, what we did is we did­n’t have the com­pli­cat­ed block­ing struc­ture that I showed you in the pre­vi­ous exper­i­ment, we did give them sequen­tial sen­tences, because we thought it would be too much for chil­dren. And in doing so what we may have done is actu­al­ly the learn­ing demands of the task too easy and not being able to observe these kinds of mem­o­ry ben­e­fits. That’s one idea potentially.

So I want to kind of sum­marise what I’ve told you so far, I’m in terms of the sci­ence so chil­dren and adults do find word learn­ing intrin­si­cal­ly. And we want to sug­gest that when tasks are chal­leng­ing enough that we find that reward is asso­ci­at­ed with green and we’re doing some future work on this. So we’re now look­ing at the neur­al basis of reward for word learn­ing in your typ­i­cal chil­dren and chil­dren with dyslex­ia. And we’re also start­ing to under­stand­ed the process­es of moti­va­tion going beyond the expe­ri­ence of reward alone. And this is work being car­ried out on whol­ly by my post­doc Desi.

15:10
But as you allud­ed to, I want­ed to talk a lit­tle bit about what is being online done for us. I want­ed to men­tion some pros and some cons and per­haps give you some very quick tips and tricks. So in terms of the pros, the reach of exper­i­ment, like I men­tioned that we test­ed some­thing like 345 data, chil­dren, for I exper­i­ment, there is no way I would have been able to do that, based on kind of strict lab set­tings, it just could not have hap­pened, I believe, to the fact that this was pos­si­ble in a pan­dem­ic mul­ti­ple times, there was again, no way that we could have gone into schools and got data in any oth­er way.

Design­ing these tests was real­ly, real­ly easy. So that ease of design and kind of get­ting under­grad­u­ate project stu­dents in more than doing this and so on was real­ly impor­tant. shar­ing and col­lab­o­ra­tion was eas­i­er. My  kind of col­lab­o­ra­tor was at NYU, but it was real­ly easy to kind of just reach out to him and say, here’s the pow­er, and you can look at it, you can play with it, give me your com­ments. And of course, goril­las can open mate­ri­als makes open sci­ence and shar­ing an opti­cal­ly rec­ol­lec­tion also much easier.

I think there are some cons, we while con­duct­ing this exper­i­ment online, we realised that we were at some point get­ting lots of bots. And that was prob­lem­at­ic, or you know, peo­ple who were just kind of pick­ing up our ads online and doing stuff, we did­n’t have a check for age in any way. So obvi­ous­ly, you’re test­ing a child in real life, you can see that they’re a child, one of the things that wor­ried us when maybe peo­ple are kind of par­tic­i­pat­ing, and there were adults, but they were just pre­tend­ing to be chil­dren to get vouch­ers from US gov­ern­ment. Obvi­ous­ly, one of the things that you get when you are watch­ing some­one do your task is a lot of insight into par­tic­i­pant behav­iour, we did­n’t quite have that.

And that we did have a tech prob­lem and one of the exper­i­ments and one of the par­tic­u­lar tasks that we were using need­ed a key­board response. But appar­ent­ly lots of chil­dren nowa­days use tablets to do every­thing. So they would come to this task and they would fail. And then basi­cal­ly, we need­ed to set up mul­ti­ple ver­sions, includ­ing a whole new day to exper­i­ment just in case they had failed this one par­tic­u­lar child­care cutouts.

And of course, there is a bit more cost to run­ning these things online then. So for exam­ple, if you run things in schools, you can just give peo­ple stick­ers. If it’s online, you need to tempt peo­ple with vouchers.

Tips and tricks, we sug­gest lots and lots of exper­i­men­tal con­trols we had in our exper­i­ment, accu­ra­cy checks, atten­tion checks, we kind of as I allud­ed to run a new ver­sion of just the day 2 exper­i­ment. And I think the main thing that I would say is be respon­sive to par­tic­i­pants and ide­al­ly, run stud­ies in small batch­es so that you can touch up email with your par­tic­i­pants and talk to them. That’s all Sor­ry for going over time, Jo. That’s me done thank you.

17:58
That was fab­u­lous. Lonely.

17:59
Sor­ry. I was try­ing to unmute myself and fail­ing to click the but­ton. That was absolute­ly bril­liant. I am so impressed with the work that you’ve done. I’m just going to have to change my view here because I can’t bear look­ing at myself when I’m speaking.

18:15
Gosh, that was a lot of kids that you’ve recruit­ed. Those are big sam­ples. How did you get them?

18:27
Oh, well, that was actu­al­ly recruit­ed even more than I pre­sent­ed here. So I think over­all, we prob­a­bly test­ed some­thing like 540 kids, and then actu­al­ly because of the kind of exper­i­men­tal con­trols that you have you drop peo­ple out. So lots and lots of kids, I guess, schools like we send a lot of emails to schools to be like, Please, can you put this up? And then lots schools are very hap­py to be like, you know, ask­ing us to devote any time. Peo­ple can just go do it online on this par­tic­u­lar link. That was great. What else did we do? Lots and lots of emails, lots and lots of social media kind of pres­sure and peo­ple. I mean, I think part of it is like when you do a reg­is­tered report you’ve com­mit­ted to doing the num­bers just

19:15
have to get out and get them. Yes, we’re get­ting the last five hard­er than get­ting the first five.

19:21
Actu­al­ly. So we realise at some point that like the, for some rea­son, we got lots of 16 to 18 euros, we got lots of 10 euros, but the kind of in between group that 12 to 14 euros, we had this lit­tle graph where we want­ed to see the num­bers going up and for some rea­son we just nev­er got 12 to 14 year olds. Get­ting those last few 12 to 14 euros was the most chal­leng­ing bit. But yeah, it works out real­ly beau­ti­ful­ly. And

19:52
so I think there are some ques­tions com­ing in the q&a. And if you’re in the audi­ence, and you’ve got a ques­tion, please do dump it in the q&a. Now, one more ques­tion from me. What? What types of checks? Did you put into, like san­i­ty checks on your data did you put into to to make sure you’re get­ting sen­si­ble data quality?

20:10
Yeah. So with Luck­i­ly, this was a reg­is­tered report. So like, a review is also rec­om­mend­ed, and your san­i­ty checks. There are kind of one san­i­ty check was part of our reg­is­tered report. And we basi­cal­ly said that if we’re real­ly see­ing chil­dren, and this prob­a­bly also answers, some of Matt Davis’s ques­tions in the q&a, is that we would expect the kind of extract­ing mean­ing would be increas­ing with age. And we find a very, very strong rela­tion­ship that so basi­cal­ly, accu­ra­cy in the m plus con­di­tion very strong­ly goes up by age. So for when you’re 10, you’re a lit­tle bit worse at answer­ing those ques­tions. When you’re 18, you’re much bet­ter answer­ing those ques­tions. And that is def­i­nite­ly what we were expect­ed. So that that’s kind of helpful.

And at some point, we hit this vein of peo­ple on Face­book, who were like, email­ing us. And you know, see­ing those emails is real­ly reas­sur­ing, because there’s a cer­tain way a par­ent com­mu­ni­cates. And you can def­i­nite­ly go yes, that def­i­nite­ly feels like an inter­ac­tion with a real client. There are some where we were just not sure. And then we have kind of a lot of them, luck­i­ly, fair kind of reten­tion checks. So we had stuff like we need­ed to have kids com­plete with a cer­tain lev­el of accu­ra­cy. We also have embed­ded atten­tion checks in there. We’ve also got under con­trol. So we’ve asked peo­ple about best sleep. And we’ve asked peo­ple about kind of we did this task Jason Yeat­man group called roar. So we have raw data. And again, on the raw, we can see, which is a lex­i­cal deci­sion task for those of you that means you can again, see the accu­ra­cy is increas­ing with age and reac­tion times kind of decreas­ing, real­ly, so kind of a good set of checks in that.

21:54
Very, very cool. Yes, that sounds like a com­pre­hen­sive set of san­i­ty checks. do you what do you want to give a more com­pre­hen­sive esti­mate? Or have you answered that suf­fi­cient­ly? The ques­tion was, do you think it’s pos­si­ble that the null effects of age can be explained by dis­hon­est age reports? I think you’ve just about cov­ered that. So I think, yeah, I

22:11
don’t think it would be dis­hon­est. I mean, I’m sure that like every sin­gle per­son in the data we can’t nec­es­sar­i­ly ver­i­fy. And one thing we should have real­ly done is kind of pos­si­bly enforce a mech­a­nism that peo­ple could only do the task ones because I was just like, oh, per­son with the same email, who’s done my task can be anoth­er lit­tle bit of kind of san­i­ty check kind of weed­ing out stuff, but I’m sure if like some­one was smarter and use like a sec­ond email, that’s a bit hard­er to claim

22:43
this thing. Did you con­sid­er try­ing to get like, and use the video record­ing zone just to get a screen­shot of the per­son? Or would you not have got that past ethics?

22:52
I think ethics would have been chal­leng­ing. I think a few peo­ple have been talk­ing about like get­ting audio record­ings dur­ing con­sent, because obvi­ous­ly, your voice is a real­ly good cue. If your agent if you say some­thing like I can send an obvi­ous­ly, you know that, you know, it’s a native Eng­lish speak­er who gives you hon­est­ly, like, you know, I mean, I know you don’t nec­es­sar­i­ly face some of these issues, but I would be inher­ent­ly extreme­ly sur­prised. And I’m sure we have a cou­ple of these sort of things in the main data set. But I’d be very sur­prised if that was the entire explanation.

23:24
Bril­liant. So only could you just go back to your side of top tips. And while cer­tain­ly he’s doing that, Neil, could you get your slides ready for shar­ing in a minute? And what I just want­ed to ask the audi­ence that we’ve got here today, which of the tips and tricks was most use­ful for you today? What What was the les­son you most need­ed to hear? Was it about learn­ing lots of exper­i­men­tal con­trols and sep­a­rat­ing accu­ra­cy checks and atten­tion sets? Or was it about need­ing to be respon­sive when things go wrong? And maybe cre­at­ing a dif­fer­ent ver­sion of exper­i­ment? That was some­thing you might not con­sid­er? Or was it the being respon­sive to par­tic­i­pants, or in these the advice on recruit­ing chil­dren? Bril­liant, thank you so much. It’s just won­der­ful. It’s real­ly great if you are will­ing to be gen­er­ous to Saloni and give the feed­back that’s been most help­ful to you, because it helps Saloni know, she’s done a good job. And it’s dif­fi­cult giv­ing a talk to a room of com­plete­ly blank screens. And that inter­ac­tion from you guys is just real­ly reward­ing to know that we are help­ing you learn, learn mes­sages that are help­ful that you’re going to take into your research and will make your lives bet­ter. Now with that Saloni thank you so much. You were won­der­ful. Thank you for start­ing us and set­ting the note the bar so high.