Tap­ping into Tablet Tots and Mid­dleschool Minecrafters: Test­ing chil­dren online using Ani­mal-AI and Goril­la Game Builder

Lucy Cheke 0:00  

So I’m going to talk about a cou­ple of research projects we’ve been doing over the last cou­ple of years, try­ing to intro­duce some gam­i­fi­ca­tion in to cog­ni­tive research. So I’m going to tell you about two stud­ies we’ve got run­ning with using online games of quite dif­fer­ent sorts using quite dif­fer­ent approach­es to gam­i­fi­ca­tion. One using Goril­la game builder, which is new and shiny, and one, using the ani­mal AI envi­ron­ment, which is a sort of home built thing that I’ll tell you a lit­tle bit more about. Just a caveat, before I start, I am not an expert on online research, by any means or gam­i­fi­ca­tion, I’m just shar­ing some cur­rent research that we’re doing. But I’m hop­ing that not only there might be some use­ful things for oth­er peo­ple in here, but also, maybe we can get some help from some of you guys.

 

So the thing that I start­ed off think­ing with try­ing to gam­i­fy is that we need to meet kids, when we’re talk­ing about try­ing to do research with kids, we need to meet kids where they already are. Kids are play­ing com­put­er games, if they were allowed to get away with it all day, every day. If we want kids to engage with our research tasks, we can learn a lot from what they are spon­ta­neous­ly choos­ing to do with their time already. So the first kind of sub­set I’m going to talk about is what I’ve called tablet tots, that is to say very small chil­dren. And we’re used to research­ing with very small chil­dren in the lab, because there’s lots of rea­son, you know, with kind of phys­i­cal han­dleable tasks, because up until very recent­ly, this is not a group, you know, we’re talk­ing to sort of one to five year olds, this is not a group that we can usu­al­ly eas­i­ly do com­put­er tasks with. But with the advent of tablets, we now have and you know prop­er tablets that they have at home all the time, not the old fash­ioned kind, we now have an oppor­tu­ni­ty to real­ly exploit the kind of expe­ri­ence that chil­dren young chil­dren now have. And again, I’m speak­ing here as a par­ent of small chil­dren. These are my two small chil­dren here to age two and four in that pic­ture play­ing on their tablets. Who would if if allowed to play tablets all day long. And what we can learn as a par­ent from just watch­ing this is that there’s a few things that par­tic­u­lar­ly for the younger kids seem to be quite reas­sur­ing, as would would be games developers.

 

First­ly, these games are not sophis­ti­cat­ed. They don’t need a clever back­sto­ry, they don’t need any­thing real­ly fan­cy. They young chil­dren year 2, 3, 4 year olds have very sim­ple tastes. Repet­i­tive is fine, repet­i­tive is good. They like repet­i­tive. They can’t read. So many games on the tablets, let alone so many exper­i­ments have text instruc­tions, you need to have audio instruc­tions, they will click on the most obvi­ous thing on the screen. And they don’t know gam­ing and inter­net con­ven­tions. If you have a sym­bol that uni­ver­sal­ly means some­thing, unless it’s lit­er­al­ly the play sym­bol with a tri­an­gle, they don’t know what it means.

 

And dex­ter­i­ty for the young kids, par­tic­u­lar­ly the dex­ter­i­ty isn’t great. So drag­ging and drop­ping is fine for like three and a half up. But my two year old quite strug­gles with it. So I want­ed to cre­ate a game that might allow us, par­tic­u­lar­ly dur­ing the pan­dem­ic to start test­ing real­ly lit­tle kids online. And I came to Jo on Goril­la gen­er­al­ly, with my needs in that I can­not code I have very lit­tle time. And I have very few resources. And Jo said, Well, we’ve got this new thing. And it’s gori­la game builder. And I promise you, they’re not pay­ing me. But this was quite a find. So with goril­la game builder, I’m sure that you’re going to be tak­en through it in more detail. But basi­cal­ly, you get a sort of screen like this, where you can

 

3:50  

you pick kind of cre­ate with game builder here. And then you kind of you build a screen with objects in the screen, you can move them around and quick and click­able and so on. And then you have lots of dif­fer­ent pages on the exper­i­ment. But then you can also kind of ani­mate to change the nature of the what’s on the screen as peo­ple go through the task. So I want­ed to start very sim­ple, because I did­n’t know what I was doing. So I cre­at­ed a very sim­ple visu­al search task.

 

So this task, the pirate game is a very sim­ple visu­al search task. Essen­tial­ly, you get giv­en a tar­get item. So you get the pirate and they say Please can you find this, so this is sil­ver coin. And then after that you get an array. So this is an exam­ple of a pop out array. That is to say that the tar­get is dif­fer­ent from the, notably dif­fer­ent from the dis­trac­tors. So the idea is that this was sort of pop out at you. Or you can have what I have referred to as a camo array, which is where the items them­selves don’t stand out ter­ri­bly well from the back­ground and are quite sim­i­lar to one anoth­er. And this involves a lot much more search effort. So what we var­ied here was set size, ie the num­ber of dis­trac­tors, the obvi­ous­ness of the tar­get, and the posi­tion. So I’m just going to show you a quick lit­tle video of what this task looks like. If some­one could tell me whether or not the sound works on this, that would be real­ly great.

 

5:29  

Could you hear that 

 

Jo Ever­shed

the sound isn’t com­ing through to us.

 

Lucy Cheke 5:33  

Okay, the sound, the only thing with the sound is that all of the instruc­tions also read out as well as being

 

Jo Ever­shed

love­ly, that’s such a nice thing to do for young kids.

 

Lucy Cheke

So this is kind of fun this and then you have an array, and the child has to go and tap or click on the tar­get. So these are the prac­tice tri­als. And these are nice, big obvi­ous targets.

 

6:04  

That’s the end of the prac­tice ses­sion. And then we have more tri­als, but just with much more var­ied stim­uli. So these are kind of slight­ly slight­ly obvi­ous exam­ples, and then you have these ones that are the real­ly dif­fi­cult ones.

 

Okay. So that’s the game is incred­i­bly sim­ple. And actu­al­ly, the gam­i­fi­ca­tion isn’t great either for in terms of my design, because you know, we did­n’t do much in terms of like, cool ani­ma­tions that hap­pen when you get the right answer, and so forth. And that was most­ly because I was so short of time, I designed a whole ani­ma­tion to do but I did­n’t have time to do it. But it’s total­ly doable with with the sys­tem. So I’m just going to show you a lit­tle bit about the kind of data qual­i­ty and stuff of the data we’ve got so far, this is work in progress, we’ve so far only got nine four to sev­en year olds, and 16, eight to 12 year olds. This is, as Jo men­tioned, a study on cog­ni­tion in kids with long COVID, where we’re test­ing those who have had COVID, and those who have not had COVID. If you have a child aged four to 12, please feel free to sign up, we’re still recruit­ing, you can just fol­low that QR code to do so. And I’ll be at the end of the talk as well. So with a bear­ing in mind that this is work in progress. And I’m not going to show you the COVID long COVID data because the sam­ple is too small for that so far. But what I want­ed to check was to whether or not this was show­ing the same sort of the right sorts of pat­terns that we’d expect to see with a visu­al search task.

 

So you can see here this is you’ve got chil­dren, dif­fer­ent age groups, and the check the per­for­mance in terms of reac­tion time, and accu­ra­cy across these dif­fer­ent lev­els of dif­fi­cul­ty. So here, we can see that as you might expect, the more obvi­ous the tar­get, the quick­er kids are and the more accu­rate. And that’s actu­al­ly with the old­er kids per­for­mance. main reac­tion time remained pret­ty steady, pend­ing on how many dis­trac­tors there were with the younger chil­dren, that went up quite sig­nif­i­cant­ly. And this is what we were real­ly look­ing for this effect of the inter­ac­tion between set size and pop out, which is some­thing you real­ly expect to see with a visu­al search task. So here we can see that when the dis­trac­tor, then the tar­get is obvi­ous in the pop out con­di­tion, you see this flat reac­tion time. Where­as in the camo con­di­tion, when the item is quite sub­tly dif­fer­ent from the dis­trac­tors, you get an increase in reac­tion time as the set size increas­es. And that sug­gests we’re pick­ing up on the two kinds of visu­al search the pop out effect, where you just see it, and the ser­i­al, you know, hav­ing to look at every sin­gle item in turn sort of effect. And that’s real­ly reassuring.

 

The oth­er thing I was look­ing for was a left hand bias, we tend to have a left hand bias in visu­al search, most­ly because in Eng­lish, we read from left to right. And also because of the left hand right hand brain issue that were both most­ly right hand­ed. And again, because I was able to dic­tate exact­ly where on the screen each of these coins were, and they each had a loca­tion, I can then map that. And I found that I’ve got this nice cen­tral to left hand side bias. So that was just an exam­ple of the first go, I’ve had doing things in game builder. I made a func­tion­al visu­al search task in about two weeks with no cod­ing. We can put audio instruc­tions onto the lit­tle kids don’t have to read it’s playable on tablets, lap­tops, com­put­ers, and phones, but I opt­ed out of that. And it was by far the most pop­u­lar of all the tasks that were in that larg­er study that that took that was part in. I’ll def­i­nite­ly be mak­ing more tasks this way. And with more time, I think I can make some­thing a lot bet­ter. But that’s what I could make in less than two weeks, the time I had spare in two weeks, which I think says a lot.

 

9:51  

The oth­er thing I’m going to talk to you through is the oth­er kind of set of research that I’ve been doing the last cou­ple of years, and this is poten­tial­ly for slight­ly old­er kids. Some cog­ni­tive abil­i­ties don’t real­ly lend them­selves to be test­ed on this sort of 2d tablet style game that I’ve just showed you. These are bet­ter test­ed with games using embod­i­ment or some form of embod­i­ment where you are a first per­son indi­vid­ual explor­ing an envi­ron­ment inter­act­ing with objects. And these are real­ly pop­u­lar with kids. As we can see, with the pop­u­lar­i­ty of games such as Minecraft and Fort­nite. They’re nor­mal­ly first per­son shoot­ers, but you don’t actu­al­ly there’s no law that says you have to have a whole have a gun in the game.

 

So I want to intro­duce you to the ani­mal AI envi­ron­ment. This is an envi­ron­ment cre­at­ed by a research team that I work with, cre­at­ed in Uni­ty and orig­i­nal­ly designed to assess cog­ni­tion in AI, using tasks that are designed for test­ing cog­ni­tion in ani­mals. And that can be or could, have been phys­i­cal­ly imple­ment­ed with ani­mals and small chil­dren in real life. So to see kind of a lit­tle bit what it looks like the the aim of the game is to col­lect these green balls or yel­low balls as well. So it there’s very sim­ple aims retrieve food, which of green or yel­low balls, avoid poi­son, which are red balls, and avoid lava, which is a red or orange floor. And so, we’ve did a kind of a proof of con­cept study look­ing to see look­ing at com­par­ing deep rein­force­ment learn­ing AIs, and six to 10 year old chil­dren on a sub­set of the tasks that we pre­sent­ed in a com­pe­ti­tion and AI com­pe­ti­tion a cou­ple of years ago. There were 900 tasks in the mid­dle com­pe­ti­tion, kids can’t do it, the 900 tasks. So we chose 40 tasks from 10 domains that kind of span, the kind of sim­plest kind of com­mon sense kind of cog­ni­tive tasks that peo­ple do with ani­mals. And this is, Kozzy Voudouris, who is my PhD stu­dents who did this work.

 

So how do AIs com­pared to six to 10 year olds, well, a sum­ma­ry of the data is not well, chil­dren out­per­formed AIs on every­thing. There was no sig­nif­i­cant dif­fer­ence between the age groups. But inter­est­ing­ly, expe­ri­ence with sim­i­lar com­put­er games did make a dif­fer­ence to the results. So that’s just a thing to note that if you are gam­i­fy­ing, in a way that match­es some­thing kids might be doing, then they will have some exper­tise that will dif­fer­en­ti­ate between dif­fer­ent chil­dren. And because he did a clus­ter analy­sis show­ing that you could iden­ti­fy which indi­vid­u­als were AIs and which were chil­dren, regard­less of the age of the child, or the flavour of the AI. And one thing we looked at par­tic­u­lar­ly, for exam­ple, was object per­ma­nence. This is some­thing that kids past the age of about 12 months in real life sce­nar­ios, we saw that with the agents, we could see that we can watch what they did. So not only look at the score, but watch what they did. And if you look back, not only do we know that, for exam­ple, the top agent only scored 25%, we know that they got that score by using the rule always go left. So actu­al­ly, if we com­pare that chil­dren to even the best AIS, chil­dren out­per­formed them on every sin­gle thing.

 

There were one set of things that for exam­ple, the chil­dren did­n’t do so well on. This is a tool, an exam­ple of a tool use task. And this is a nine year old who passed it. And we realised after­wards. And this is one of the impor­tance of kind of doing these proof of con­cept and pre­lim­i­nary stuff, that these were just way too dif­fi­cult. It sounds obvi­ous to pull a hook to get a reward. But actu­al­ly, the phys­i­cal manip­u­la­tion of the hook was actu­al­ly very dif­fi­cult for chil­dren and AIs both. But what was inter­est­ing is that the dif­fer­ence between the chil­dren the AI is was com­plete­ly observ­able even when both failed. So you can see here on the left, and AI basi­cal­ly just going around in cir­cles. Where­as the child’s takes a real­ly con­sid­ered approach, we can watch back exact­ly what the child did. And then through dif­fer­ent ways of cod­ing the data, we can start to get maps of not just per­for­mance, but a pat­tern of per­for­mance and type of behaviour.

 

We’re tak­ing this for­ward with the Ani­mal AI envi­ron­ment. First­ly, I’m work­ing on some par­tic­u­lar­ly large bat­ter­ies. So Kozzy’s work­ing on an object per­ma­nence bat­tery, and this is Denia, who’s work­ing on a space and object­hood bat­tery. We’re also work­ing on new fea­tures. So we were build­ing in reward dis­pensers, which are these ripen­ing and decay­ing rewards and new playable char­ac­ters to make it cuter and fun­ner. So we’re basi­cal­ly try­ing to make it more ver­sa­tile and attrac­tive. It is cur­rent­ly avail­able for oth­er researchers to use online, but it’s real­ly tricky and fid­dly at the moment. So we’re try­ing to work on mak­ing it more avail­able as well as to this to this end, we will be hir­ing some­one. So please do watch this space. And if you’ve got game dev expe­ri­ence, par­tic­u­lar­ly with uni­ty and putting things online, get in touch.

 

14:30  

And the point of this point, I realise I’m run­ning out of time, but the point of this par­tic­u­lar approach is that we’re real­ly try­ing to build this for trans­la­tion­al poten­tial. The ani­mal AI tasks are designed to be because they’re in this kind of phys­i­cal envi­ron­ment that could be real. It could be a room in a lab. They are playable as com­put­er games, yes, but they are direct­ly relat­ed to those tasks that can and are used phys­i­cal­ly in the lab, both with ani­mals and chil­dren. So this facil­i­tates trans­la­tion between ani­mal mod­els, lab work with babies, tod­dlers, chil­dren too young for com­put­ers, and online games for old­er kids and adults and com­pu­ta­tion­al mod­els, because remem­ber, this was designed for and it’s being used for assess­ing AI mod­els. So we can make com­pu­ta­tion­al mod­els of cog­ni­tion of impair­ment of devel­op­ment, and direct­ly test the­o­ret­i­cal accounts, all with­in the same envi­ron­ment. So to sum­marise, Ein­stein appar­ent­ly said that hav­ing fun is the best way to learn. Hav­ing fun is I think, also the best way to test what has been learned. And I’m cer­tain­ly going to be mov­ing more towards gam­i­fi­ca­tion in every­thing I do from now on. So just a quick thank you to the, you know, the many, many peo­ple that have been involved in both of those research projects. And thank you for to goril­la and to Jo for invit­ing me to speak today. If you want to take part in the one to try out the the pirate game on your own kid, you can fol­low this link down down here at the bot­tom here, or scan this QR code, and it’ll take you to it. Thank you very much.

 

Jo Ever­shed 15:58  

Thank you, Lucy, that was amaz­ing. I am going to ask the audi­ence in the chat to put any ques­tions you have for Lucy into the into the q&a now. So I can come back to those in a minute. I’m just try­ing to think what I want to ask you, I just find that I actu­al­ly just find that all so fas­ci­nat­ing. What I don’t under­stand is how have you been able to do all of this, you’re like a mum, you’ve got two kids about the same age as mine

 

Lucy Cheke 16:31  

Thank you. Okay, so I did, I did the visu­al search task. And that was most­ly because with the COVID project, it’s fun­ny, I have two hats on, I am a researcher in the psy­chol­o­gy depart­ment doing the effects of kind of health on mem­o­ry and cog­ni­tion. And that basi­cal­ly has no fund­ing and is run­ning on vol­un­teers and stu­dents and like me try­ing to do stuff myself. And then I’m also I work in an AI, an AI insti­tute called the Cen­tre for the future of intel­li­gence. And there, I’m the direc­tor of the kinds of intel­li­gence projects and we’ve got mul­ti­ple, incred­i­bly tal­ent­ed post­docs doing amaz­ing things and PhD stu­dents as well. So this is well resourced, This is not. So on this side, I did the game builder thing myself, because I did­n’t think it was fair to expect any­one else to kind of work out this new tool. There is a moth on my face. And with the ani­mal AI thing that’s been devel­oped with, it’s most­ly fund­ed by peo­ple who are very inter­est­ed in under­stand­ing the cog­ni­tive abil­i­ties of AIs at the moment, which is a big deal for the con­flict of eco­nom­ic pol­i­cy, secu­ri­ty and so forth rea­sons. But I’m keep­ing it so that it can still test kids so that we can bring it more fur­ther into psy­chol­o­gy. And I’d real­ly love to build a com­mu­ni­ty around peo­ple using this to test kids because because I think it has loads of poten­tial. But yeah, there’s a there’s a there’s a big team.

 

Jo Ever­shed 17:51  

that. So that’s amaz­ing. I total­ly agree with you. I think the idea that AIs are going to reach adult lev­el cog­ni­tion with­out first devel­op­ing child­like cog­ni­tion is unlike­ly. And the data, like first, right, yeah, so we’re gonna have to work this way back and lay­er them in. And that was so inter­est­ing to see how the AIs were per­form­ing against what was it six year olds, six to 10 year olds. And still very basic, but if we can get that data from younger chil­dren, that’s going to help that that jour­ney. But of course, at the moment, you can’t get younger chil­dren to do this stuff online, they’re too com­pli­cat­ed. But if we can sim­pli­fy them, then we might be able to start get­ting to a point where the AI can out­per­form a very young child, if we’re con­fi­dent they’re using the con­trols. So I think that’s, that’s tremen­dous­ly inter­est­ing ques­tion has come in. Thank you. Do you ever allow adults to play with their kids eg as a fun activ­i­ty to do togeth­er? Because then adults could try to encour­age kids kids to beat them?

 

Lucy Cheke 18:48  

That’s a good, that’s a good point. It’s a good idea. And we haven’t done that. And with the pirate game. I’m not sure how I mean, yeah, there’s prac­tice tri­als and so on. But I think with game builder, you prob­a­bly heard with ani­mal AI, cer­tain­ly, like there’s basi­cal­ly we had prac­tice tri­als. And there’s, and it’s very open, which is why it takes actu­al­ly a lot of skill and devel­op­er time. So it’s cur­rent­ly tricky. But we, we had that in the study that we ran, we had chil­dren ask­ing request­ing to be able to come back and just keep play­ing and mess­ing around with it. And so we had to kind of have a like a two tier sys­tem, like we’re col­lect­ing data from you ver­sus like, you’ve come back for the 16th time and are try­ing to beat your own record. So def­i­nite­ly, there was lots of scope for that. And the tasks we were doing were bor­ing for adults, but that’s because it was real­ly like here’s a reward can you get it? But they don’t need to be bor­ing for adults and we’ve made some real­ly hard ones now. So hope­ful­ly we should be when we are plan­ning on test­ing adults as well. So I think that’s that’s real­ly great idea a way of engag­ing kids.

 

Jo Ever­shed 19:53  

Yeah, and if you were using game builder, there are two ways you could do that. You could have like anoth­er grown up take her has a go and now the child has a go so you could inter­leave them that could work real­ly well. Or we do have mul­ti­play­er com­ing to goril­la so you could set up so that a par­ent and a child could in fact play togeth­er and respond to the same task. I guess the bit that’s always tricky though is you don’t want the par­ents coach­ing the child because you actu­al­ly want to gen­uine­ly cap­ture the child’s behav­iour. That’s always the tricky bit to com­pen­sate for. I hope I have tweet­ed your link that you need more par­tic­i­pants. If any­body on the call today has got chil- young chil­dren, please do con­sid­er reach­ing out to Lucy.