Mea­sur­ing skilled motor behav­iors in the age of COVID

I’m Ryan More­head. I’m a co-direc­­tor of the Immer­sive Cog­ni­tion Lab here at the Uni­ver­si­ty of Leeds, and I’m not sure how chop­py this video is for you guys, but I want­ed to put it on here because it’s some­body play­ing a first-per­­son shoot­er, AME train­er, on their per­son­al com­put­er at home. And it kind of rep­re­sents the best that you can get in terms of skilled motor behav­ior with a mouse and key­board and a home PC com­put­er. This is some­thing that over time, I think we want to try to get to with brows­er-based exper­i­ments. So I come from a back­ground of motor con­trol, human com­pu­ta­tion­al motor con­trol, that JP just talked a lit­tle bit about the kind of equip­ment that we use. And in par­tic­u­lar, we use equip­ment that has high spa­tial and tem­po­ral fideli­ty. So it’s often sam­pling health where you are, where you’re at the posi­tion of your limb at 200 Hertz, or even up to a thou­sand Hertz. And you’re get­ting visu­al feed­back as fast as 240 Hertz.

The sit­u­a­tion on the inter­net is a lit­tle bit dif­fer­ent. Now, the ICON lab here at Leeds we are using Uni­ty WebGL, and we’re host­ing it on Ama­zon Web Ser­vices and using some oth­er soft­ware such as C‑sharp and JavaScript to do the exper­i­ments that we’re doing. But we have the fun­da­men­tal lim­i­ta­tion that all inter­net-based research has, which is that we don’t know the equip­ment peo­ple are using and it can actu­al­ly be pret­ty poor equip­ment. So often that means the dis­play is at best refresh­ing at 60 Hertz, but occa­sion­al­ly up to 240 Hertz. And the hand track­ing that we use, whether a mouse or key­board press­es, is maybe 60 Hertz and poten­tial­ly up to a thou­sand Hertz. So that means two things, one is that often this is less pre­cise than what we’re used to, but also it’s high­ly vari­able from per­son to person.

If you’re focus­ing just on key­board­ing tasks, which is some­thing that a post­doc in our lab, Emi­ly Williams is doing, then this pri­mar­i­ly just affects the sam­pling inter­val that you can detect a key press. So 60 Hertz means a 16.67 mil­lisec­onds is about every time that you’re going to get a data point. And it’s also when you can present infor­ma­tion back to the par­tic­i­pant. Anoth­er ele­ment of key­boards online is this, n‑key rollover? You can actu­al­ly know if some­body has a sort of high per­for­mance gam­ing key­board where you can detect six or eight or how­ev­er many simul­ta­ne­ous key press­es, and so it’s pos­si­ble that in some cas­es you can detect real­ly rel­e­vant key press­es that are errors that peo­ple are making.

That’s pret­ty much the case for key­boards, but for mouse move­ments, it’s a lit­tle bit more com­pli­cat­ed. And I’ll talk about sev­er­al dif­fer­ent ele­ments of this dur­ing the talk for your sort of vanil­la, just reach­ing from a start loca­tion to a tar­get­ed exper­i­ments, such as the kind that Matthew War­bur­ton, a PhD stu­dent, in the lab is doing. This means that you might be using an opti­cal mouse, or you might be using a track pad, and I think it’s impor­tant to keep in mind that these are dif­fer­ent bio­me­chan­i­cal­ly and also in terms of the mech­a­nisms of how they work. And while we were prepar­ing for Matthew’s exper­i­ment, we got some data from Anisha Chandy and Jonathan Tsay at the Ivry Lab at UC Berke­ley and ana­lyzed the vari­abil­i­ty in reached direc­tion and also the accu­ra­cy. And what I’m show­ing you here is the suc­cess for dif­fer­ent tar­get direc­tions for 24 dif­fer­ent tar­gets around 360 degrees, both when you have visu­al feed­back of where you’re mov­ing to, and when you don’t have visu­al feed­back, shown in red. And you can clear­ly see that it’s dif­fer­ent across direc­tions, but also dif­fer­ent across the two devices.

And so that’s some­thing that you want to keep in mind when you’re run­ning exper­i­ments like this, is that you may find dif­fer­ences based on the equip­ment that the par­tic­i­pan­t’s using. So for the rest of the talk I’m going to focus on a spe­cif­ic task, which is an inter­cep­tive tim­ing task that John Pick­a­vance, a PhD stu­dent, is doing. The tasks that he had at the begin­ning of quar­an­tines and lock­downs was to trans­late this lab based exper­i­ment with a high refresh rate on both the input and out­put of his equip­ment into an online task. And typ­i­cal­ly in inter­cep­tive tim­ing tasks, there’s a lit­tle tar­get mov­ing across the screen, and you’re try­ing to inter­cept it with a cur­sor that you can only move in one dimen­sion and you do that by slid­ing a han­dle along a rail.

So John devel­oped this task for use with chil­dren in schools. And so he kind of gam­i­fied it and used instead of just a black block, he used a uniden­ti­fied fruit object, and that moves across the screen on a fixed path. And then there’s a fruit bat down in a cave that you have to fly out and try to inter­cept the fruit. And you want to try to do this with­in a 100 to 300 mil­lisec­onds. An impor­tant thing to point out here is that you can move your mouse lat­er­al­ly on this, but the bat itself will only move ver­ti­cal­ly. And so it’s, it’s only mov­ing along the ver­ti­cal path for this task. And hope­ful­ly let’s see, I think I have to play this one. Hope­ful­ly you guys can see this video, let it loop through a few times. This is what the task looks like. You just move the bat out, try to hit the tar­get, and if you do hit it, you get this kind of Got­ti splat popped up on the screen and also a tone place.

So an impor­tant thing to keep in mind dur­ing this actu­al­ly, sor­ry, we col­lect this data on Ama­zon and Pro­lif­ic. But an impor­tant thing to keep in mind is that the equip­ment that we’re using to record this data in the first place has lim­its, and so JP was try­ing to look and see if you have some­body mak­ing a move­ment across dif­fer­ent peo­ple as one per­son wants to make a big move­ment and oth­er per­son­’s mak­ing a small move­ment, or if they have dif­fer­ences in gain on their mouse, or dif­fer­ences in qual­i­ty of mouse, how is that going to affect things? And so he used some equip­ment to mea­sure if you make dif­fer­ent mouse move­ments of increas­ing veloc­i­ties, will your mouse be able to track it? And it turns out that for all mice, includ­ing very high end gam­ing mice, if you start to move a lit­tle bit faster than a meter per sec­ond with them, that they start to bug out and give you real­ly bad data.

And what this led us to do is actu­al­ly intro­duce a screen at the begin­ning of his task, where he can have peo­ple try to move the lit­tle bat from a start posi­tion to a fin­ish with one lit­tle move­ment that’s indi­cat­ed by a gift on the screen for what they’re try­ing to do, whether they’re using a track pad or an opti­cal mouse. This allows us to kind of stan­dard­ize and ame­lio­rate the dif­fer­ences across par­tic­i­pants’ com­put­ers. So what JP or John Pick­a­vance want­ed to look at with this task is not just the inter­cep­tive tim­ing, but actu­al­ly stop­ping your­self from mov­ing in the con­text of an inter­cep­tion tasks. So he had a sub­set of tri­als where the screen’s back­ground changed col­ors indi­cat­ing that it was dawn. And when it’s dawn, the screen may stay like this, state orange, and if it stays orange then you’re free to move out and inter­cept the tar­get the same way that you would if it’s night­time, but on a sub­set of tri­als, the screen will actu­al­ly change color.

And hope­ful­ly it’ll not be too chop­py for you guys. The screen will actu­al­ly change col­or and if you move the bat out­side of the cave, when it’s day­time, you’ll actu­al­ly get sun­burned and lose the tri­al. So these are stop tri­als or no-go tri­als. You don’t actu­al­ly want to move on them. And the fact that the screen changes col­or will indi­cate whether this is a tri­al where you’re cer­tain that you can move freely and inter­cept the tar­get, or if it’s a tri­al where you don’t know whether it’s going to be safe to go or not. And these are allot­ted in the exper­i­ment, 50% are cer­tain tri­als, 50% are uncer­tain and out of every block of uncer­tain, sor­ry, these are ran­dom­ly inter­leaved, but out of the uncer­tain tri­als, 5 of the 15 are on our stock tri­als. So that’s 33% of your stock tri­als, sor­ry, 33% of your uncer­tain tri­als or stop tri­als, which is actu­al­ly real­ly impor­tant for these designs. And then we have eight blocks of tri­als. And for all the data I’m going to show you for the most part, we’re show­ing 52 peo­ple that we col­lect­ed off of Prolific.

Impor­tant­ly for this task, for being a Stop Sig­nal Reac­tion Time task, there’s some basic things that we need to make sure are going on, which is that I’m intro­duc­ing this screen change, where the tells you to stop mov­ing and not move out of the cave. Actu­al­ly it is chal­leng­ing for you. And so what we do is, dur­ing the tri­al, the tar­get appears and starts to move. And there’s a time where we think you should start mov­ing and we’re going to present the stop sig­nal ini­tial­ly, actu­al­ly, right about when we think you want to start mov­ing. And then we stair­case the stop sig­nal back and forth to deter­mine a time where it’s actu­al­ly pre­sent­ed before you would need to start mov­ing. And you could only stop your­self on 50% of those trials.

So we’re always stair­cas­ing this, for each indi­vid­ual, 50% of their tri­als, they’re mak­ing it when the stop sig­nal is pre­sent­ed and 50% they’re fail­ing. And then what we do is find the actu­al reac­tion time. So how long before you were start­ing, you were going to start mov­ing? Are you able to stop your­self? What we find in this task, it’s about 200 mil­lisec­onds, which is con­sis­tent with oth­er stop sig­nal tasks. So that’s good. We’re kind of meet­ing the cri­te­ria there. And how­ev­er, what we’re real­ly inter­est­ed in here are proac­tive stop­ping, which are mea­sures where, you know it’s an uncer­tain tri­al and you’re actu­al­ly doing some­thing dif­fer­ent on these uncer­tain tri­als than what you would do on the cer­tain tri­als. And we have a few dif­fer­ent mea­sures of this. Impor­tant­ly, we’re only look­ing at tri­als where you actu­al­ly made a movement.

And so, but we’re com­par­ing uncer­tain tri­als where you move to cer­tain tri­als where you moved. So one of the mea­sures here is Move­ment Time. And what you can see is that there’s a clear dif­fer­ence in the cer­tain and uncer­tain Move­ment Times where peo­ple are mov­ing faster on the uncer­tain tri­als. Also the Ini­ti­a­tion Time, so when peo­ple start to move is faster on uncer­tain, and for both of these, they get faster over the block. And then also for anoth­er mea­sure called Tim­ing Error, we see a dif­fer­ence here as well, where peo­ple are actu­al­ly lat­er on the uncer­tain and to make this Tim­ing Error a lit­tle bit more intu­itive or pal­pa­ble for you guys, it’s kind of tan­ta­mount to where you’re hit­ting the tar­get with the bat. So, here I’m plot­ting on cer­tain tri­als on these graphs at the bot­tom here where peo­ple hit on the actu­al fruit UFO here at the top, and you can see that they most­ly hit the front right cor­ner, which is actu­al­ly the ide­al spot to hit to max­i­mize your success.

And on the uncer­tain tri­als start­ing out there ini­tial­ly, and sort of creep­ing back over time as the exper­i­ment goes on, which you can see over here. And so for both of these mea­sures, this Ini­ti­a­tion Time, sor­ry, these are all sig­nif­i­cant, but for both of these mea­sures, Ini­ti­a­tion Time and Tim­ing Error, we were inter­est­ed in whether peo­ple were doing this con­scious­ly or whether it was just some­thing that emerged out of the task. And so what we did is put a task at the end of the exper­i­ment where we had peo­ple watch a video of the UFO going by. And we told them to note when they would have tried to start mov­ing dur­ing the actu­al task. And then after­wards we let them posi­tion the UFO at that loca­tion. And you can see that they clear­ly are putting the cer­tain, the UFO on a cer­tain back­ground ear­li­er than they are uncertain.

So they’re aware that they’re ini­ti­at­ing lat­er. And we also had them click where they were try­ing to hit the UFO with the bat on cer­tain ver­sus uncer­tain tri­als. And we found a sim­i­lar thing where they indi­cat­ed fur­ther back on the UFO for where they are try­ing to hit so that we think this is impor­tant because these proac­tive mea­sures are not just some­thing that’s implic­it­ly emerg­ing through some uncon­scious learn­ing process. But this is some­thing that they active­ly know they’re doing. There’s one final point that I just want to make here. That’s actu­al­ly a method­olog­i­cal point. And this has to do with the lag, because we know that there are dif­fer­ences across peo­ple’s com­put­ers. So what I’m plot­ting on these plots is the intend­ed posi­tion of the tar­get. So at where the tar­gets sup­posed to be mov­ing, giv­en the amount of time that’s elapsed in the tri­al ver­sus where we actu­al­ly record the tar­get is.

And for some tri­als, there’s very lit­tle dif­fer­ence between these two things. But on some peo­ple’s com­put­ers, their com­put­er chugs a lit­tle bit drops a frame, and they have some trou­ble with the tar­get appear­ing where it’s sup­posed to be. And those are kind of high­light tri­als. And this is some­thing that actu­al­ly affects some of our mea­sures. So here on the left is the pro­por­tion of times that they hit the tar­get on cer­tain tri­als. And you see that there there’s about a 10% reduc­tion in per­for­mance when peo­ple have a lot of lag­gy tri­als, and then they also have longer Stop Sig­nal Reac­tion Times to the same thing. And John noticed that this actu­al­ly seemed to be dis­pro­por­tion­ate­ly affect­ing peo­ple that were using a track pad. So he added a lit­tle screen to the begin­ning of the task, that if some­one indi­cat­ed that they had a track pad that told them to go plug in their lap­top before they start­ed the task.

And so in an ini­tial pilot with this, he had to throw out over half the par­tic­i­pants, 11 of which were for low hit per­cent­age. And after adding in this lit­tle screen, he has to throw out far few­er for bad per­for­mance. How­ev­er, you still actu­al­ly still see that that peo­ple using a track pad have more lag­gy tri­als than peo­ple using an opti­cal mouse. So in gen­er­al, lap­tops aren’t as high per­form­ing com­put­ers as a desk­top PC. So for a quick sum­ma­ry here, when you’re get­ting data from either an opti­cal or a track pad mouse, you need to take into con­sid­er­a­tion that these are bio­me­chan­i­cal­ly dif­fer­ent and may result in dif­fer­ent suc­cess rates for reach­es in dif­fer­ent directions.

There are upper bounds on how fast you can move a mouse and so that can affect the move­ments that you can record. And you should try to ame­lio­rate that if you can and lag itself on because of vari­able equip­ment across par­tic­i­pants can affect the per­for­mance that you see in these tasks. How­ev­er, we can still get design tasks that get the stan­dard effects that we see in the field, and also find some inter­est­ing new find­ings with these tech­niques. So my gen­er­al thought here is that online exper­i­ments are cool. And I’d like to thank every­body in the lab that con­tributed to this. So thank you.