Mea­sur­ing skilled motor behav­iors in the age of COVID

Ryan More­head, Uni­ver­si­ty of Leeds


By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

Full Tran­script:

I’m Ryan More­head. I’m a co-direc­­tor of the Immer­sive Cog­ni­tion Lab here at the Uni­ver­si­ty of Leeds, and I’m not sure how chop­py this video is for you guys, but I want­ed to put it on here because it’s some­body play­ing a first-per­­son shoot­er, AME train­er, on their per­son­al com­put­er at home. And it kind of rep­re­sents the best that you can get in terms of skilled motor behav­ior with a mouse and key­board and a home PC com­put­er. This is some­thing that over time, I think we want to try to get to with brows­er-based exper­i­ments. So I come from a back­ground of motor con­trol, human com­pu­ta­tion­al motor con­trol, that JP just talked a lit­tle bit about the kind of equip­ment that we use. And in par­tic­u­lar, we use equip­ment that has high spa­tial and tem­po­ral fideli­ty. So it’s often sam­pling health where you are, where you’re at the posi­tion of your limb at 200 Hertz, or even up to a thou­sand Hertz. And you’re get­ting visu­al feed­back as fast as 240 Hertz.

The sit­u­a­tion on the inter­net is a lit­tle bit dif­fer­ent. Now, the ICON lab here at Leeds we are using Uni­ty WebGL, and we’re host­ing it on Ama­zon Web Ser­vices and using some oth­er soft­ware such as C‑sharp and JavaScript to do the exper­i­ments that we’re doing. But we have the fun­da­men­tal lim­i­ta­tion that all inter­net-based research has, which is that we don’t know the equip­ment peo­ple are using and it can actu­al­ly be pret­ty poor equip­ment. So often that means the dis­play is at best refresh­ing at 60 Hertz, but occa­sion­al­ly up to 240 Hertz. And the hand track­ing that we use, whether a mouse or key­board press­es, is maybe 60 Hertz and poten­tial­ly up to a thou­sand Hertz. So that means two things, one is that often this is less pre­cise than what we’re used to, but also it’s high­ly vari­able from per­son to person.

If you’re focus­ing just on key­board­ing tasks, which is some­thing that a post­doc in our lab, Emi­ly Williams is doing, then this pri­mar­i­ly just affects the sam­pling inter­val that you can detect a key press. So 60 Hertz means a 16.67 mil­lisec­onds is about every time that you’re going to get a data point. And it’s also when you can present infor­ma­tion back to the par­tic­i­pant. Anoth­er ele­ment of key­boards online is this, n‑key rollover? You can actu­al­ly know if some­body has a sort of high per­for­mance gam­ing key­board where you can detect six or eight or how­ev­er many simul­ta­ne­ous key press­es, and so it’s pos­si­ble that in some cas­es you can detect real­ly rel­e­vant key press­es that are errors that peo­ple are making.

That’s pret­ty much the case for key­boards, but for mouse move­ments, it’s a lit­tle bit more com­pli­cat­ed. And I’ll talk about sev­er­al dif­fer­ent ele­ments of this dur­ing the talk for your sort of vanil­la, just reach­ing from a start loca­tion to a tar­get­ed exper­i­ments, such as the kind that Matthew War­bur­ton, a PhD stu­dent, in the lab is doing. This means that you might be using an opti­cal mouse, or you might be using a track pad, and I think it’s impor­tant to keep in mind that these are dif­fer­ent bio­me­chan­i­cal­ly and also in terms of the mech­a­nisms of how they work. And while we were prepar­ing for Matthew’s exper­i­ment, we got some data from Anisha Chandy and Jonathan Tsay at the Ivry Lab at UC Berke­ley and ana­lyzed the vari­abil­i­ty in reached direc­tion and also the accu­ra­cy. And what I’m show­ing you here is the suc­cess for dif­fer­ent tar­get direc­tions for 24 dif­fer­ent tar­gets around 360 degrees, both when you have visu­al feed­back of where you’re mov­ing to, and when you don’t have visu­al feed­back, shown in red. And you can clear­ly see that it’s dif­fer­ent across direc­tions, but also dif­fer­ent across the two devices.

And so that’s some­thing that you want to keep in mind when you’re run­ning exper­i­ments like this, is that you may find dif­fer­ences based on the equip­ment that the par­tic­i­pan­t’s using. So for the rest of the talk I’m going to focus on a spe­cif­ic task, which is an inter­cep­tive tim­ing task that John Pick­a­vance, a PhD stu­dent, is doing. The tasks that he had at the begin­ning of quar­an­tines and lock­downs was to trans­late this lab based exper­i­ment with a high refresh rate on both the input and out­put of his equip­ment into an online task. And typ­i­cal­ly in inter­cep­tive tim­ing tasks, there’s a lit­tle tar­get mov­ing across the screen, and you’re try­ing to inter­cept it with a cur­sor that you can only move in one dimen­sion and you do that by slid­ing a han­dle along a rail.

So John devel­oped this task for use with chil­dren in schools. And so he kind of gam­i­fied it and used instead of just a black block, he used a uniden­ti­fied fruit object, and that moves across the screen on a fixed path. And then there’s a fruit bat down in a cave that you have to fly out and try to inter­cept the fruit. And you want to try to do this with­in a 100 to 300 mil­lisec­onds. An impor­tant thing to point out here is that you can move your mouse lat­er­al­ly on this, but the bat itself will only move ver­ti­cal­ly. And so it’s, it’s only mov­ing along the ver­ti­cal path for this task. And hope­ful­ly let’s see, I think I have to play this one. Hope­ful­ly you guys can see this video, let it loop through a few times. This is what the task looks like. You just move the bat out, try to hit the tar­get, and if you do hit it, you get this kind of Got­ti splat popped up on the screen and also a tone place.

So an impor­tant thing to keep in mind dur­ing this actu­al­ly, sor­ry, we col­lect this data on Ama­zon and Pro­lif­ic. But an impor­tant thing to keep in mind is that the equip­ment that we’re using to record this data in the first place has lim­its, and so JP was try­ing to look and see if you have some­body mak­ing a move­ment across dif­fer­ent peo­ple as one per­son wants to make a big move­ment and oth­er per­son­’s mak­ing a small move­ment, or if they have dif­fer­ences in gain on their mouse, or dif­fer­ences in qual­i­ty of mouse, how is that going to affect things? And so he used some equip­ment to mea­sure if you make dif­fer­ent mouse move­ments of increas­ing veloc­i­ties, will your mouse be able to track it? And it turns out that for all mice, includ­ing very high end gam­ing mice, if you start to move a lit­tle bit faster than a meter per sec­ond with them, that they start to bug out and give you real­ly bad data.

And what this led us to do is actu­al­ly intro­duce a screen at the begin­ning of his task, where he can have peo­ple try to move the lit­tle bat from a start posi­tion to a fin­ish with one lit­tle move­ment that’s indi­cat­ed by a gift on the screen for what they’re try­ing to do, whether they’re using a track pad or an opti­cal mouse. This allows us to kind of stan­dard­ize and ame­lio­rate the dif­fer­ences across par­tic­i­pants’ com­put­ers. So what JP or John Pick­a­vance want­ed to look at with this task is not just the inter­cep­tive tim­ing, but actu­al­ly stop­ping your­self from mov­ing in the con­text of an inter­cep­tion tasks. So he had a sub­set of tri­als where the screen’s back­ground changed col­ors indi­cat­ing that it was dawn. And when it’s dawn, the screen may stay like this, state orange, and if it stays orange then you’re free to move out and inter­cept the tar­get the same way that you would if it’s night­time, but on a sub­set of tri­als, the screen will actu­al­ly change color.

And hope­ful­ly it’ll not be too chop­py for you guys. The screen will actu­al­ly change col­or and if you move the bat out­side of the cave, when it’s day­time, you’ll actu­al­ly get sun­burned and lose the tri­al. So these are stop tri­als or no-go tri­als. You don’t actu­al­ly want to move on them. And the fact that the screen changes col­or will indi­cate whether this is a tri­al where you’re cer­tain that you can move freely and inter­cept the tar­get, or if it’s a tri­al where you don’t know whether it’s going to be safe to go or not. And these are allot­ted in the exper­i­ment, 50% are cer­tain tri­als, 50% are uncer­tain and out of every block of uncer­tain, sor­ry, these are ran­dom­ly inter­leaved, but out of the uncer­tain tri­als, 5 of the 15 are on our stock tri­als. So that’s 33% of your stock tri­als, sor­ry, 33% of your uncer­tain tri­als or stop tri­als, which is actu­al­ly real­ly impor­tant for these designs. And then we have eight blocks of tri­als. And for all the data I’m going to show you for the most part, we’re show­ing 52 peo­ple that we col­lect­ed off of Prolific.

Impor­tant­ly for this task, for being a Stop Sig­nal Reac­tion Time task, there’s some basic things that we need to make sure are going on, which is that I’m intro­duc­ing this screen change, where the tells you to stop mov­ing and not move out of the cave. Actu­al­ly it is chal­leng­ing for you. And so what we do is, dur­ing the tri­al, the tar­get appears and starts to move. And there’s a time where we think you should start mov­ing and we’re going to present the stop sig­nal ini­tial­ly, actu­al­ly, right about when we think you want to start mov­ing. And then we stair­case the stop sig­nal back and forth to deter­mine a time where it’s actu­al­ly pre­sent­ed before you would need to start mov­ing. And you could only stop your­self on 50% of those trials.

So we’re always stair­cas­ing this, for each indi­vid­ual, 50% of their tri­als, they’re mak­ing it when the stop sig­nal is pre­sent­ed and 50% they’re fail­ing. And then what we do is find the actu­al reac­tion time. So how long before you were start­ing, you were going to start mov­ing? Are you able to stop your­self? What we find in this task, it’s about 200 mil­lisec­onds, which is con­sis­tent with oth­er stop sig­nal tasks. So that’s good. We’re kind of meet­ing the cri­te­ria there. And how­ev­er, what we’re real­ly inter­est­ed in here are proac­tive stop­ping, which are mea­sures where, you know it’s an uncer­tain tri­al and you’re actu­al­ly doing some­thing dif­fer­ent on these uncer­tain tri­als than what you would do on the cer­tain tri­als. And we have a few dif­fer­ent mea­sures of this. Impor­tant­ly, we’re only look­ing at tri­als where you actu­al­ly made a movement.

And so, but we’re com­par­ing uncer­tain tri­als where you move to cer­tain tri­als where you moved. So one of the mea­sures here is Move­ment Time. And what you can see is that there’s a clear dif­fer­ence in the cer­tain and uncer­tain Move­ment Times where peo­ple are mov­ing faster on the uncer­tain tri­als. Also the Ini­ti­a­tion Time, so when peo­ple start to move is faster on uncer­tain, and for both of these, they get faster over the block. And then also for anoth­er mea­sure called Tim­ing Error, we see a dif­fer­ence here as well, where peo­ple are actu­al­ly lat­er on the uncer­tain and to make this Tim­ing Error a lit­tle bit more intu­itive or pal­pa­ble for you guys, it’s kind of tan­ta­mount to where you’re hit­ting the tar­get with the bat. So, here I’m plot­ting on cer­tain tri­als on these graphs at the bot­tom here where peo­ple hit on the actu­al fruit UFO here at the top, and you can see that they most­ly hit the front right cor­ner, which is actu­al­ly the ide­al spot to hit to max­i­mize your success.

And on the uncer­tain tri­als start­ing out there ini­tial­ly, and sort of creep­ing back over time as the exper­i­ment goes on, which you can see over here. And so for both of these mea­sures, this Ini­ti­a­tion Time, sor­ry, these are all sig­nif­i­cant, but for both of these mea­sures, Ini­ti­a­tion Time and Tim­ing Error, we were inter­est­ed in whether peo­ple were doing this con­scious­ly or whether it was just some­thing that emerged out of the task. And so what we did is put a task at the end of the exper­i­ment where we had peo­ple watch a video of the UFO going by. And we told them to note when they would have tried to start mov­ing dur­ing the actu­al task. And then after­wards we let them posi­tion the UFO at that loca­tion. And you can see that they clear­ly are putting the cer­tain, the UFO on a cer­tain back­ground ear­li­er than they are uncertain.

So they’re aware that they’re ini­ti­at­ing lat­er. And we also had them click where they were try­ing to hit the UFO with the bat on cer­tain ver­sus uncer­tain tri­als. And we found a sim­i­lar thing where they indi­cat­ed fur­ther back on the UFO for where they are try­ing to hit so that we think this is impor­tant because these proac­tive mea­sures are not just some­thing that’s implic­it­ly emerg­ing through some uncon­scious learn­ing process. But this is some­thing that they active­ly know they’re doing. There’s one final point that I just want to make here. That’s actu­al­ly a method­olog­i­cal point. And this has to do with the lag, because we know that there are dif­fer­ences across peo­ple’s com­put­ers. So what I’m plot­ting on these plots is the intend­ed posi­tion of the tar­get. So at where the tar­gets sup­posed to be mov­ing, giv­en the amount of time that’s elapsed in the tri­al ver­sus where we actu­al­ly record the tar­get is.

And for some tri­als, there’s very lit­tle dif­fer­ence between these two things. But on some peo­ple’s com­put­ers, their com­put­er chugs a lit­tle bit drops a frame, and they have some trou­ble with the tar­get appear­ing where it’s sup­posed to be. And those are kind of high­light tri­als. And this is some­thing that actu­al­ly affects some of our mea­sures. So here on the left is the pro­por­tion of times that they hit the tar­get on cer­tain tri­als. And you see that there there’s about a 10% reduc­tion in per­for­mance when peo­ple have a lot of lag­gy tri­als, and then they also have longer Stop Sig­nal Reac­tion Times to the same thing. And John noticed that this actu­al­ly seemed to be dis­pro­por­tion­ate­ly affect­ing peo­ple that were using a track pad. So he added a lit­tle screen to the begin­ning of the task, that if some­one indi­cat­ed that they had a track pad that told them to go plug in their lap­top before they start­ed the task.

And so in an ini­tial pilot with this, he had to throw out over half the par­tic­i­pants, 11 of which were for low hit per­cent­age. And after adding in this lit­tle screen, he has to throw out far few­er for bad per­for­mance. How­ev­er, you still actu­al­ly still see that that peo­ple using a track pad have more lag­gy tri­als than peo­ple using an opti­cal mouse. So in gen­er­al, lap­tops aren’t as high per­form­ing com­put­ers as a desk­top PC. So for a quick sum­ma­ry here, when you’re get­ting data from either an opti­cal or a track pad mouse, you need to take into con­sid­er­a­tion that these are bio­me­chan­i­cal­ly dif­fer­ent and may result in dif­fer­ent suc­cess rates for reach­es in dif­fer­ent directions.

There are upper bounds on how fast you can move a mouse and so that can affect the move­ments that you can record. And you should try to ame­lio­rate that if you can and lag itself on because of vari­able equip­ment across par­tic­i­pants can affect the per­for­mance that you see in these tasks. How­ev­er, we can still get design tasks that get the stan­dard effects that we see in the field, and also find some inter­est­ing new find­ings with these tech­niques. So my gen­er­al thought here is that online exper­i­ments are cool. And I’d like to thank every­body in the lab that con­tributed to this. So thank you.


Get on the Registration List

BeOnline is the conference to learn all about online behavioral research. It's the ideal place to discover the challenges and benefits of online research and to learn from pioneers. If that sounds interesting to you, then click the button below to register for the 2023 conference on Thursday July 6th. You will be the first to know when we release new content and timings for BeOnline 2023.

With thanks to our sponsors!