MouseView.js: A no-eyes, mouse-based, alter­na­tive to eye track­ing online. What you can use it for, and how to use it on Gorilla.sc

Alex Anwyl-Irvine:
Thanks for join­ing every­one. Wel­come to the spe­cial­ist sem­i­nar on our MouseView.js library. I’m Alex Anwyl-Irvine, I’m just com­ing to the end of my PhD at the Uni­ver­si­ty of Cam­bridge and I cur­rent­ly work as an R&D intern at Cam­bridge cog­ni­tion and I was also a part-time devel­op­er at Gorilla.

Alex Anwyl-Irvine:
Myself, Tom and Edwin made this library, and most of the soft­ware devel­op­ment has been done by myself, which is why I’m going to give this talk today about the library and much of what we’ll dis­cuss is also detailed in our pre-print on site archive, which you can check out on our website.

Alex Anwyl-Irvine:
So just to give a brief out­line, this talk will be in three main parts. So the first will be a bit behind why we thought this tool would be the answer to some of our research chal­lenges. The sec­ond will be about how the data gath­ered using Mou­se­View com­pares to a typ­i­cal free view­ing eye track­ing task. And then third­ly, I’ll talk about how you actu­al­ly add this to an exper­i­ment using Goril­la Zone as an example.

Alex Anwyl-Irvine:
So first­ly, why did we decide to make this? As the pan­dem­ic closed mul­ti­ple labs this year and last year, eye track­ing became dif­fi­cult to do in a typ­i­cal way and the obvi­ous alter­na­tive was to use web­cams to do this, which gen­er­al­ly speak­ing is a great option. There are plen­ty tools out there like WebGaz­er, which has been built into plat­forms such as Goril­la and jsPysch and oth­ers. How­ev­er, we found solu­tions like this tend to ampli­fy the issues that your expe­ri­ence with eye track­ing in the lab, and these are things like los­ing the eyes or issues by find­ing land­marks on the face or con­trast prob­lems with the cam­era and eye­wear, and fur­ther to the list, you get lots of attri­tion which we have to com­bat with large of samples.

Alex Anwyl-Irvine:
And then even after that, we end up with rel­a­tive­ly low res­o­lu­tion data, which for cer­tain research ques­tions is inter­est­ing, but for us and our col­lab­o­ra­tors, this was quite dif­fi­cult to work with. And so we thought, would­n’t it be great if we could improve upon these accu­ra­cy issues with a dif­fer­ent tool. And would­n’t it be even bet­ter if we could avoid the awk­ward cal­i­bra­tion stages and those errors too, these are the source of much of the attri­tion that you can expe­ri­ence when run­ning these types of exper­i­ments online.

Alex Anwyl-Irvine:
And the solu­tion we came up with is mouse track­ing with an occlud­ing lay­er and this is a pret­ty sim­ple idea, and it has none of the fid­dly tech­ni­cal issues that you have with web­cam eye track­ing. And this works in the fol­low­ing main steps. So first, we cre­ate an over­lay atop any web­site, and this could either be sol­id or a par­tial cover.

Alex Anwyl-Irvine:
Sec­ond, we then detect the mouse posi­tion or touch posi­tion on the touch screen and then we record those coor­di­nates. And at the same time, an aper­ture is cut in the over­lay, which reveals the con­tent under­neath. We then iter­ate between points two and three, which moves the aper­ture to the mouse loca­tion or fin­ger loca­tion on the touch screen, every time the screen refresh­es, and this forces the user to move the mouse to any area that they’re inter­est­ed in. And over time we can build up an atten­tion­al map, which is anal­o­gous to those that we make with eye-track­ers. And to dri­ve this point home, here’s an exam­ple of maps we pro­duced using eye track­ers on the top, and then the equiv­a­lent map, just using a MouseView.js, even though this is just col­lapsed over time, you can see that they’re pret­ty sim­i­lar topologically.

Alex Anwyl-Irvine:
And this is what Mou­se­View looks like in prac­tice. So the end result is here and you can see the user explore the pic­ture with an aper­ture and reveal­ing the face of the dog and the cat. And then in a sec­ond, you’ll see a heat map which reveals the attend­ed loca­tions of this image. And so that’s essen­tial­ly how it works.

Alex Anwyl-Irvine:
In addi­tion to this base func­tion­al­i­ty, Mou­se­View gives you lots of con­fig­u­ra­tion options. So first­ly, you can vary the aspects of that over­lay that cov­er the screen. You can alter the trans­paren­cy, so here its going from about 50% to 0% trans­par­ent. You can change the col­or to any­thing you want. And a big strength is the abil­i­ty to blur any­thing under­neath that over­lay, even if there’s a trans­paren­cy of zero. And this allows par­tic­i­pants to see the low-lev­el fea­tures of the whole scene and use that to guide their atten­tion and sim­u­lates some­thing like the non-foveat­ed area of our visu­al field.

Alex Anwyl-Irvine:
How­ev­er, this is an exper­i­men­tal fea­ture and can take one to four sec­onds to ini­ti­ate, so isn’t appro­pri­ate for things like videos. We do pro­vide a call­back func­tion, which allows you to hide the con­tents of the web­site or exper­i­ment from view, whilst it gen­er­ates, but things like Goril­la Zone actu­al­ly already do this for you so you don’t real­ly need to wor­ry. The sec­ond part of the con­fig­u­ra­tion is you can alter the size of the aper­ture, so you can change it from being very small to very big. And you can also express it as a per­cent­age of the page, width and height. And impor­tant­ly, you can also apply a Gauss­ian blur to the edge of this, which soft­ens the aper­ture and again, this sim­u­lates some ele­ments of foveat­ed vision, so the edge of that foveat­ed blur.

Alex Anwyl-Irvine:
So it’s well of me telling you about this tool and its set­tings, but we want­ed to find out how well this tool could repli­cate eye track­ing results. And I’m going to give a brief out­line here because oth­ers are going to talk more about this lat­er, but I’ll show you the basis of our val­i­da­tion paper.

Alex Anwyl-Irvine:
So our empir­i­cal ques­tion was based on a paper pub­lished by Tom and Edwin pre­vi­ous­ly, which was a pas­sive view­ing eye track­ing task and it used an in-lab eye track­er. In this task, peo­ple viewed a series of pairs of images for 10, 20 sec­onds, and these were either a neu­tral image paired with a pleas­ant stim­uli or a neu­tral image paired with a dis­gust­ing stim­uli. I’ve used emo­jis here to avoid hav­ing to show you the images, but I promise you’re not miss­ing out on any­thing because they quite dis­gust­ing, but you might see some of them lat­er, so that’s exciting.

Alex Anwyl-Irvine:
So when we look at the lab eye track­ing results of these two types of tri­als, we find this pat­tern. We have an ear­ly invol­un­tary look to the evoca­tive pic­ture, regard­less of what it is and then lat­er, vol­un­tary bias after this. And this bias goes in the direc­tion of the pleas­ant image, so peo­ple are look­ing at the thing that they’re inter­est­ed in, and this is quite nice and then it’s flipped for the unpleas­ant or dis­gust­ing images.

Alex Anwyl-Irvine:
And when we used Mou­se­View to repli­cate this exper­i­ment, we found this pat­tern. We found no invol­un­tary look, but we sus­pect this may be because mouse track­ing only tracks top-down atten­tion­al con­trol. And I’m sure Tom will touch on this lat­er, but we did repli­cate the lat­er com­po­nent, the vol­un­tary bias, peo­ple dwelt on the pleas­ant image, but avoid­ed the dis­gust­ing image quite strong­ly. And as avoid­ance strength­ens over time, which was also the case in the lab eye track­ing study. We found that this ele­ment was reli­able and valid, but you can read more about that in the pre-print.

Alex Anwyl-Irvine:
Okay, so like I said, we’re going to get more infor­ma­tion on what peo­ple have done with this tool lat­er, but I’m also here to talk about, how to get exper­i­ments work­ing your­self. And thanks to the many open source in gen­er­al pur­pose nature of this tool, there are many options avail­able to you and these range from full on app devel­op­ment to no pro­gram­ming at all, which is what this rep­re­sents. The [inaudi­ble 00:07:45] JS ver­sion can be includ­ed in any Apple web­site. And we have a node pack­age man­ag­er repos­i­to­ry and a GitHub set up and I’ve tried to make this as straight­for­ward for devel­op­ers to use as I can. But to do that, you do need some cod­ing expe­ri­ence, in the mid­dle between cod­ing and no cod­ing. You’ve got exper­i­ment builders like Psy­cho­JS and Psy­choPy and jsPsych. How­ev­er, you still do need to do a bit of pro­gram­ming to under­stand this, but Goril­la is more towards the no cod­ing thing, so its appro­pri­ate for a small tuto­r­i­al, when­ev­er you got 15 minutes.

Alex Anwyl-Irvine:
So in Goril­la, you can use the code imple­men­ta­tion which allows you to cus­tomize things a bit more, but you can use their Task Builder, and they’ve recent­ly cre­at­ed a drag and drop zone that involves absolute­ly no cod­ing at all. And that’s what I’m going to show you how to use today.

Alex Anwyl-Irvine:
So we have a Goril­la Zone. So this is an exam­ple of a sim­ple tri­al, iden­ti­cal to the type of free view­ing exper­i­ment I spoke about ear­li­er, and this is the Goril­la screens in the task. You have three screens here, so you’ve got a fix­a­tion cross with a timer to get a delay between each tri­al. And then you have a screen with a but­ton, which is very impor­tant because it ensures when they press that but­ton that the cur­sor or fin­ger is at the cen­ter of the screen and they’re not see­ing one image more than the oth­er. And then final­ly the stim­uli them­selves, side-by-side or shar­ing for 20 sec­onds. And it’s this screen that we’re going to put our Goril­la Zone into.

Alex Anwyl-Irvine:
And you select this view­ing screen and this brings up this lay­out, which shows the left image and the right image. And this is a lit­tle tim­ing wid­get. You press the edit lay­out but­ton and it shows the lay­out of the screen with these gray box­es. And we want to add a new zone in, so we press that but­ton and this cre­ates a new zone, which we then select with our mouse and brings up this menu where we go to the advanced tab and if you’re on the beta pro­gram, you’ll see that lit­tle Mou­se­View box there. And this is what we press to add in, and then it’s added in.

Alex Anwyl-Irvine:
And you can see it here in the cen­ter. And this will have been added with our default set­tings, which you can change by find­ing the cor­rect sec­tion in the con­fig­u­ra­tion set­tings. So you could scroll down, here’s the mouse track­ing set­tings. So specif­i­cal­ly here in the mode, record, which I’ll use for this pur­pose, just records the mouse coor­di­nates and does the Mou­se­View over­lay on this screen. You can use upload, which in com­bi­na­tion with this set­ting here allows you to apply a Mou­se­View to your whole exper­i­ment so you’re con­tin­u­ous­ly track­ing over mul­ti­ple screens.

Alex Anwyl-Irvine:
This then has a drop­down box with the Mou­se­View spe­cif­ic set­tings. And in this drop­down box, you have the things I spoke about ear­li­er, so the over­lay col­or, the alpha which deals with trans­paren­cy, the Gauss­ian blur, stan­dard devi­a­tion and pix­els and the aper­ture size, and these are all con­fig­urable. Now, once we’ve done that, we’re ready to run the exam­ple exper­i­ment. So play this. So when you pre­view the task and launch it, you will receive the instruc­tions and then when they press next, the but­ton appears, they click on it and here’s our blur with the aper­ture under.

Alex Anwyl-Irvine:
And it real­ly is as sim­ple as that, and this data’s being record­ed and you’ll be able to down­load it in a spread­sheet that has all of the coor­di­nates, but I don’t have time to tell you how to ana­lyze that, but hope­ful­ly we’ll be releas­ing some codes to help peo­ple do that on our website.

Alex Anwyl-Irvine:
Okay, so I think that’s about it and thanks for lis­ten­ing to me, you can vis­it mouseview.org, which has detailed doc­u­men­ta­tion, links to our pre-print and exam­ples of var­i­ous dif­fer­ent lev­els of imple­men­ta­tion I spoke about. The Zone on Goril­la is in closed beta, so you can get it added to your account by con­tact­ing them, using a form. I’d sug­gest doing this, if you just want to play around with dif­fer­ent exper­i­ment designs, it’s free to make an exper­i­ment, so it’s a pret­ty cool way of work­ing out what you can do. And for updates in the future, then fol­low me on Twit­ter, but a shame­less pro­mo­tion, and also fol­low any­one in this sem­i­nar because I think we’ll all be doing things in the future.

Alex Anwyl-Irvine:
I’d like to thank Edwin and Tom who have been great at mak­ing this project hap­pen, but also Will at Goril­la who led pro­gram­ming the drag and drop zone and also Thomas Pronk and Rebec­ca Hirst, Psy­choPy, for imple­ment­ing it there, and mak­ing our exam­ples. Okay. Thanks.