Build­ing an Engag­ing Child-Friend­ly Task for Remote Admin­is­tra­tion: Exam­ple of a Gam­i­fied Mouse-Track­ing Task on Paired Asso­ciate Learning

Simone Cal­abrich:
Hi, every­one. I’m Simone and I’m a PhD stu­dent at Ban­gor Uni­ver­si­ty. And today I’ll show you some of the things that we do to make our online exper­i­ments on paired asso­ciate learn­ing, more engag­ing and child-friend­ly as well. Need­less to say, the chil­dren and adults are two com­plete­ly dif­fer­ent pop­u­la­tions. So the way chil­dren engage with exper­i­ments and online test­ing is com­plete­ly dif­fer­ent from adults in many dif­fer­ent aspects. With few excep­tions, I would say most adults will com­plete our exper­i­ments with no major prob­lems, pro­vid­ed that they’re giv­en clear and detailed instruc­tions. Chil­dren, on the oth­er hand, will inevitably need more help and guid­ance and they might get bored faster as well. And anoth­er issue is that some­times par­ents or guardians might want to help the child dur­ing an exper­i­men­tal task, which I appre­ci­ate its some­thing that they do with the best inten­tion, but the data col­lect­ed under cer­tain cir­cum­stances will prob­a­bly not be a true reflec­tion of the child’s per­for­mance and we won’t be able to use their data in our reports.

Simone Cal­abrich:
I’m not sure if this is on top of the… So to cir­cum­vent that we found that hav­ing a research assis­tant guid­ing and moti­vat­ing the child through the exper­i­ments will pro­duce the best qual­i­ty data, and the appli­ca­tion which we have been using for that is Microsoft Teams. And what we do is we request the child to share their screen with us so that we can keep track of what they’re doing in the exper­i­ment as if we were with them in the lab. And with the par­ents or guardians per­mis­sion, we also some­times ask them to give us con­trol of their screen, to make sure that rel­e­vant details are entered cor­rect­ly, and also that the sev­er­al steps in our exper­i­ment are fol­lowed as expect­ed. What we also did was we pro­vid­ed our research assis­tants with lots of train­ing so that they could be able to admin­is­ter the tasks effectively.

Simone Cal­abrich:
And this was espe­cial­ly rel­e­vant because our par­a­digm required ver­bal respons­es to be scored online, and we also had mul­ti­ple lists. So it was impor­tant that they were on top of every­thing. And we also made sure that the research assis­tants them­selves pilot­ed the task as if they were par­tic­i­pants. And because we also admin­is­tered a bat­tery of cog­ni­tive and lit­er­a­cy tests, we made sure that the research assis­tants spent some time admin­is­ter­ing the tasks to each oth­er as well. And we believe that these steps allowed us to antic­i­pate and even fix some of the issues pri­or to start­ing data col­lec­tion with the chil­dren. And in addi­tion to hav­ing a research assis­tant sup­port­ing and encour­ag­ing the chil­dren, one thing which we also intro­duced to our task was gam­i­fi­ca­tion, which I know that it’s going to be dis­cussed exten­sive­ly lat­er today, but put sim­ply, gam­i­fi­ca­tion is the appli­ca­tion of game mech­a­nisms in non-game environments.

Simone Cal­abrich:
And one of the main goals of gam­i­fi­ca­tion is to enhance moti­va­tion lev­els. And there are many dif­fer­ent game ele­ments which can poten­tial­ly be applied in behav­ioral sci­ence. But this has some­thing to do with one of the ques­tions in the chat ear­li­er today, but it’s real­ly impor­tant that we eval­u­ate and decide which of these ele­ments are the most suit­able for the pur­pos­es of our inves­ti­ga­tion. And I think the gam­i­fi­ca­tion ele­ment, which most peo­ple prob­a­bly think about first, is the intro­duc­tion of some sort of reward based on the par­tic­i­pan­t’s per­for­mance. But we know that this is not always fea­si­ble for some inves­ti­ga­tions because some­times when we pro­vide feed­back to par­tic­i­pants or when we tell them how well or how bad­ly they’re doing, we might end up intro­duc­ing some sort of bias to their per­for­mance, which could neg­a­tive­ly affect our find­ings. So this was the case for us, to some extent, we want­ed to run a mouse­track­ing exper­i­ment in Goril­la, which would have the clas­si­cal ele­ments of most mouse­track­ing tasks.

Simone Cal­abrich:
We had two response options at the top of the screen. We had a but­ton at the bot­tom, which upon click­ing on it, it would play an audio file. But apart from a col­or­ful but­ton, which we added to make it slight­ly more child-friend­ly, we felt that there was not much that could be added to the exper­i­men­tal task itself because any major mod­i­fi­ca­tions such as pro­vid­ing the chil­dren with scores and things like that, could poten­tial­ly influ­ence our find­ings. So what we did was we embed­ded our exper­i­men­tal tasks in the con­text of a fic­tion­al sto­ry, which is a very, very easy to imple­ment gam­i­fi­ca­tion ele­ment. And we think that by intro­duc­ing char­ac­ters to our exper­i­ments, we could then use these char­ac­ters at sev­er­al dif­fer­ent points through­out the experiment.

Simone Cal­abrich:
So for exam­ple, the char­ac­ters in our case pro­vid­ed the chil­dren with instruc­tions, they gave the chil­dren words of encour­age­ment after a cer­tain num­ber of tri­als, as if this was an actu­al game, but that was actu­al­ly done at reg­u­lar inter­vals, regard­less of the chil­dren’s per­for­mance. And the nar­ra­tive we intro­duced also had a log­i­cal plot line. So we had a begin­ning and end to that sto­ry so that the chil­dren could have some­thing to look for­ward to as they did the task. And in our exper­i­ment, we were inves­ti­gat­ing chil­dren’s abil­i­ty to learn nov­el visu­al phono­log­i­cal asso­ci­a­tions, but we told the chil­dren that the sym­bols ans pseu­do-words that they would be exposed to, came from an alien lan­guage and that by play­ing the game, they would learn some words from that lan­guage and the decode a mes­sage at the end of the experiment.

Simone Cal­abrich:
And the main point with this was just to make the chil­dren for­get, to some extent, that they were par­tic­i­pat­ing in an exper­i­ment and get them inter­est­ed and curi­ous about the task as much as pos­si­ble. So we pri­mar­i­ly use attri­bu­tion-free web­sites to select pic­tures, short ani­ma­tions, and also sound effects for our tasks. And some of these web­sites even allow us to mod­i­fy the stim­uli and adapt them to suit the pur­pos­es of our task. So we cre­ative­ly used those screens and also the audio zones in Goril­la to gen­er­ate a stop motion-like ani­ma­tion in order to demon­strate how the exper­i­men­tal task was sup­posed to be done. So we record­ed a nar­ra­tive and then we sep­a­rat­ed the audio files into small­er clips. And when we com­bined that with some of the illus­tra­tions, we were able to tell a sim­ple but effec­tive sto­ry to the chil­dren with­out hav­ing to rely on a lot of tech­ni­cal exper­tise, we did­n’t even have to code for that.

Simone Cal­abrich:
We also used a speech soft­ware pack­age to manip­u­late some audio record­ings. We used specif­i­cal­ly the change gen­der func­tion in Praat in order to make the instruc­tions of our task styles a lit­tle bit more whimsical.

Zop:
Hi, My name is Zop and I come from a galaxy far, far away called Bip.

Simone Cal­abrich:
I’m not sure if you could hear this. But what we did was we tried to do our best to make every step of the way fun, col­or­ful, intrigu­ing. If we had a start but­ton or a rel­e­vant trans­la­tion between the screens, we added sound effects like this one. [crosstalk 00:07:33]

Simone Cal­abrich:
What­ev­er we found that was rel­e­vant. And these were all very sim­ple changes, which we intro­duced to pro­vide the chil­dren with, to some extent, a mul­ti­sen­so­ry expe­ri­ence as much as pos­si­ble, even though we were test­ing them remote­ly. And we tried to do that with­out chang­ing the par­a­digm of our exper­i­men­tal task. And just one last thing which I would like to empha­size is that a very impor­tant step is pilot­ing our tasks with adults, because adults can tell us about poten­tial tech­ni­cal issues they found in the task or even prob­lems that they think they could emerge, or they can tell us about a poten­tial lack of clar­i­ty in the instruc­tions and things like that. But even more impor­tant than that, is that we should pilot the task with an actu­al child pri­or to data col­lec­tion, because kids are gen­er­al­ly hon­est and they will let you know exact­ly what they think about your task.

Simone Cal­abrich:
They’re also very cre­ative. So if you ask them, they can give you some tips on some aspects, which you could per­haps add to the exper­i­ment to make it more engag­ing from the point of view of a child. And because they are your tar­get group, it’s prob­a­bly a good idea to lis­ten to them.

Simone Cal­abrich:
This is all I have to share for today. I would like to thank my super­vi­sor and one of our col­lab­o­ra­tors for their input. And thank you every­one for your attention.

Speak­er 3:
Thank you very much. Thank you. So just a reminder, please do put your ques­tions in the Q&A sec­tion and the speak­ers will be pick­ing up on those through­out the rest of the after­noon. But just quick­ly, do you have any expe­ri­ence or any thoughts about the best way of adapt­ing your real­ly nice online par­a­digms for nor­mal­ly devel­op­ing chil­dren? For if you were work­ing with dif­fer­ent groups of chil­dren, you might have issues around learn­ing disability.

Simone Cal­abrich:
Sor­ry, I think I’m hav­ing some dif­fi­cul­ties here with my Zoom.

Speak­er 3:
That’s okay. Can you hear me? Did you hear the question?

Simone Cal­abrich:
Hel­lo? Can you hear me?

Speak­er 3:
Hi. Can you hear me?

Simone Cal­abrich:
Okay, I can now.

Speak­er 3:
Excel­lent. Very quick­ly. A love­ly study and a real­ly nice exam­ple of how to mod­i­fy things for good online exper­i­men­ta­tion with chil­dren. And the first ques­tion that’s come through is about what… Do you have any hints or tips or ideas about how you could mod­i­fy the sort of thing you’re already doing for work with chil­dren with learn­ing dis­abil­i­ties or learn­ing challenges?

Simone Cal­abrich:
To be hon­est, I haven’t done any stud­ies with chil­dren with learn­ing dis­abil­i­ties, but I think prob­a­bly the only thing that I would say is not to have a long exper­i­ment because it’s prob­a­bly very exhaust­ing for them to be star­ing at a screen for such a long time. And I know that we’re try­ing to do exper­i­ments remote­ly, and the idea is to do that dif­fer­ent­ly from what we used to do in the lab, but def­i­nite­ly hav­ing some­one with them, talk­ing them through the exper­i­ment, I know it’s prob­a­bly very time-con­sum­ing and it’s the oppo­site of what we’ve been doing with online research, but hav­ing a research assis­tant with them is prob­a­bly the best thing to do for chil­dren with some sort of disability.

Speak­er 3:
That’s won­der­ful. Thank you very much.