MouseView.js: A no-eyes, mouse-based, alter­na­tive to eye track­ing online. What you can use it for, and how to use it on Gorilla.sc

Alex Anwyl-Irvine, Cam­bridge Uni­ver­si­ty
@AlexanderIrvine

YouTube

By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

Full Tran­script:

Alex Anwyl-Irvine:
Thanks for join­ing every­one. Wel­come to the spe­cial­ist sem­i­nar on our MouseView.js library. I’m Alex Anwyl-Irvine, I’m just com­ing to the end of my PhD at the Uni­ver­si­ty of Cam­bridge and I cur­rent­ly work as an R&D intern at Cam­bridge cog­ni­tion and I was also a part-time devel­op­er at Gorilla.

Alex Anwyl-Irvine:
Myself, Tom and Edwin made this library, and most of the soft­ware devel­op­ment has been done by myself, which is why I’m going to give this talk today about the library and much of what we’ll dis­cuss is also detailed in our pre-print on site archive, which you can check out on our website.

Alex Anwyl-Irvine:
So just to give a brief out­line, this talk will be in three main parts. So the first will be a bit behind why we thought this tool would be the answer to some of our research chal­lenges. The sec­ond will be about how the data gath­ered using Mou­se­View com­pares to a typ­i­cal free view­ing eye track­ing task. And then third­ly, I’ll talk about how you actu­al­ly add this to an exper­i­ment using Goril­la Zone as an example.

Alex Anwyl-Irvine:
So first­ly, why did we decide to make this? As the pan­dem­ic closed mul­ti­ple labs this year and last year, eye track­ing became dif­fi­cult to do in a typ­i­cal way and the obvi­ous alter­na­tive was to use web­cams to do this, which gen­er­al­ly speak­ing is a great option. There are plen­ty tools out there like WebGaz­er, which has been built into plat­forms such as Goril­la and jsPysch and oth­ers. How­ev­er, we found solu­tions like this tend to ampli­fy the issues that your expe­ri­ence with eye track­ing in the lab, and these are things like los­ing the eyes or issues by find­ing land­marks on the face or con­trast prob­lems with the cam­era and eye­wear, and fur­ther to the list, you get lots of attri­tion which we have to com­bat with large of samples.

Alex Anwyl-Irvine:
And then even after that, we end up with rel­a­tive­ly low res­o­lu­tion data, which for cer­tain research ques­tions is inter­est­ing, but for us and our col­lab­o­ra­tors, this was quite dif­fi­cult to work with. And so we thought, would­n’t it be great if we could improve upon these accu­ra­cy issues with a dif­fer­ent tool. And would­n’t it be even bet­ter if we could avoid the awk­ward cal­i­bra­tion stages and those errors too, these are the source of much of the attri­tion that you can expe­ri­ence when run­ning these types of exper­i­ments online.

Alex Anwyl-Irvine:
And the solu­tion we came up with is mouse track­ing with an occlud­ing lay­er and this is a pret­ty sim­ple idea, and it has none of the fid­dly tech­ni­cal issues that you have with web­cam eye track­ing. And this works in the fol­low­ing main steps. So first, we cre­ate an over­lay atop any web­site, and this could either be sol­id or a par­tial cover.

Alex Anwyl-Irvine:
Sec­ond, we then detect the mouse posi­tion or touch posi­tion on the touch screen and then we record those coor­di­nates. And at the same time, an aper­ture is cut in the over­lay, which reveals the con­tent under­neath. We then iter­ate between points two and three, which moves the aper­ture to the mouse loca­tion or fin­ger loca­tion on the touch screen, every time the screen refresh­es, and this forces the user to move the mouse to any area that they’re inter­est­ed in. And over time we can build up an atten­tion­al map, which is anal­o­gous to those that we make with eye-track­ers. And to dri­ve this point home, here’s an exam­ple of maps we pro­duced using eye track­ers on the top, and then the equiv­a­lent map, just using a MouseView.js, even though this is just col­lapsed over time, you can see that they’re pret­ty sim­i­lar topologically.

Alex Anwyl-Irvine:
And this is what Mou­se­View looks like in prac­tice. So the end result is here and you can see the user explore the pic­ture with an aper­ture and reveal­ing the face of the dog and the cat. And then in a sec­ond, you’ll see a heat map which reveals the attend­ed loca­tions of this image. And so that’s essen­tial­ly how it works.

Alex Anwyl-Irvine:
In addi­tion to this base func­tion­al­i­ty, Mou­se­View gives you lots of con­fig­u­ra­tion options. So first­ly, you can vary the aspects of that over­lay that cov­er the screen. You can alter the trans­paren­cy, so here its going from about 50% to 0% trans­par­ent. You can change the col­or to any­thing you want. And a big strength is the abil­i­ty to blur any­thing under­neath that over­lay, even if there’s a trans­paren­cy of zero. And this allows par­tic­i­pants to see the low-lev­el fea­tures of the whole scene and use that to guide their atten­tion and sim­u­lates some­thing like the non-foveat­ed area of our visu­al field.

Alex Anwyl-Irvine:
How­ev­er, this is an exper­i­men­tal fea­ture and can take one to four sec­onds to ini­ti­ate, so isn’t appro­pri­ate for things like videos. We do pro­vide a call­back func­tion, which allows you to hide the con­tents of the web­site or exper­i­ment from view, whilst it gen­er­ates, but things like Goril­la Zone actu­al­ly already do this for you so you don’t real­ly need to wor­ry. The sec­ond part of the con­fig­u­ra­tion is you can alter the size of the aper­ture, so you can change it from being very small to very big. And you can also express it as a per­cent­age of the page, width and height. And impor­tant­ly, you can also apply a Gauss­ian blur to the edge of this, which soft­ens the aper­ture and again, this sim­u­lates some ele­ments of foveat­ed vision, so the edge of that foveat­ed blur.

Alex Anwyl-Irvine:
So it’s well of me telling you about this tool and its set­tings, but we want­ed to find out how well this tool could repli­cate eye track­ing results. And I’m going to give a brief out­line here because oth­ers are going to talk more about this lat­er, but I’ll show you the basis of our val­i­da­tion paper.

Alex Anwyl-Irvine:
So our empir­i­cal ques­tion was based on a paper pub­lished by Tom and Edwin pre­vi­ous­ly, which was a pas­sive view­ing eye track­ing task and it used an in-lab eye track­er. In this task, peo­ple viewed a series of pairs of images for 10, 20 sec­onds, and these were either a neu­tral image paired with a pleas­ant stim­uli or a neu­tral image paired with a dis­gust­ing stim­uli. I’ve used emo­jis here to avoid hav­ing to show you the images, but I promise you’re not miss­ing out on any­thing because they quite dis­gust­ing, but you might see some of them lat­er, so that’s exciting.

Alex Anwyl-Irvine:
So when we look at the lab eye track­ing results of these two types of tri­als, we find this pat­tern. We have an ear­ly invol­un­tary look to the evoca­tive pic­ture, regard­less of what it is and then lat­er, vol­un­tary bias after this. And this bias goes in the direc­tion of the pleas­ant image, so peo­ple are look­ing at the thing that they’re inter­est­ed in, and this is quite nice and then it’s flipped for the unpleas­ant or dis­gust­ing images.

Alex Anwyl-Irvine:
And when we used Mou­se­View to repli­cate this exper­i­ment, we found this pat­tern. We found no invol­un­tary look, but we sus­pect this may be because mouse track­ing only tracks top-down atten­tion­al con­trol. And I’m sure Tom will touch on this lat­er, but we did repli­cate the lat­er com­po­nent, the vol­un­tary bias, peo­ple dwelt on the pleas­ant image, but avoid­ed the dis­gust­ing image quite strong­ly. And as avoid­ance strength­ens over time, which was also the case in the lab eye track­ing study. We found that this ele­ment was reli­able and valid, but you can read more about that in the pre-print.

Alex Anwyl-Irvine:
Okay, so like I said, we’re going to get more infor­ma­tion on what peo­ple have done with this tool lat­er, but I’m also here to talk about, how to get exper­i­ments work­ing your­self. And thanks to the many open source in gen­er­al pur­pose nature of this tool, there are many options avail­able to you and these range from full on app devel­op­ment to no pro­gram­ming at all, which is what this rep­re­sents. The [inaudi­ble 00:07:45] JS ver­sion can be includ­ed in any Apple web­site. And we have a node pack­age man­ag­er repos­i­to­ry and a GitHub set up and I’ve tried to make this as straight­for­ward for devel­op­ers to use as I can. But to do that, you do need some cod­ing expe­ri­ence, in the mid­dle between cod­ing and no cod­ing. You’ve got exper­i­ment builders like Psy­cho­JS and Psy­choPy and jsPsych. How­ev­er, you still do need to do a bit of pro­gram­ming to under­stand this, but Goril­la is more towards the no cod­ing thing, so its appro­pri­ate for a small tuto­r­i­al, when­ev­er you got 15 minutes.

Alex Anwyl-Irvine:
So in Goril­la, you can use the code imple­men­ta­tion which allows you to cus­tomize things a bit more, but you can use their Task Builder, and they’ve recent­ly cre­at­ed a drag and drop zone that involves absolute­ly no cod­ing at all. And that’s what I’m going to show you how to use today.

Alex Anwyl-Irvine:
So we have a Goril­la Zone. So this is an exam­ple of a sim­ple tri­al, iden­ti­cal to the type of free view­ing exper­i­ment I spoke about ear­li­er, and this is the Goril­la screens in the task. You have three screens here, so you’ve got a fix­a­tion cross with a timer to get a delay between each tri­al. And then you have a screen with a but­ton, which is very impor­tant because it ensures when they press that but­ton that the cur­sor or fin­ger is at the cen­ter of the screen and they’re not see­ing one image more than the oth­er. And then final­ly the stim­uli them­selves, side-by-side or shar­ing for 20 sec­onds. And it’s this screen that we’re going to put our Goril­la Zone into.

Alex Anwyl-Irvine:
And you select this view­ing screen and this brings up this lay­out, which shows the left image and the right image. And this is a lit­tle tim­ing wid­get. You press the edit lay­out but­ton and it shows the lay­out of the screen with these gray box­es. And we want to add a new zone in, so we press that but­ton and this cre­ates a new zone, which we then select with our mouse and brings up this menu where we go to the advanced tab and if you’re on the beta pro­gram, you’ll see that lit­tle Mou­se­View box there. And this is what we press to add in, and then it’s added in.

Alex Anwyl-Irvine:
And you can see it here in the cen­ter. And this will have been added with our default set­tings, which you can change by find­ing the cor­rect sec­tion in the con­fig­u­ra­tion set­tings. So you could scroll down, here’s the mouse track­ing set­tings. So specif­i­cal­ly here in the mode, record, which I’ll use for this pur­pose, just records the mouse coor­di­nates and does the Mou­se­View over­lay on this screen. You can use upload, which in com­bi­na­tion with this set­ting here allows you to apply a Mou­se­View to your whole exper­i­ment so you’re con­tin­u­ous­ly track­ing over mul­ti­ple screens.

Alex Anwyl-Irvine:
This then has a drop­down box with the Mou­se­View spe­cif­ic set­tings. And in this drop­down box, you have the things I spoke about ear­li­er, so the over­lay col­or, the alpha which deals with trans­paren­cy, the Gauss­ian blur, stan­dard devi­a­tion and pix­els and the aper­ture size, and these are all con­fig­urable. Now, once we’ve done that, we’re ready to run the exam­ple exper­i­ment. So play this. So when you pre­view the task and launch it, you will receive the instruc­tions and then when they press next, the but­ton appears, they click on it and here’s our blur with the aper­ture under.

Alex Anwyl-Irvine:
And it real­ly is as sim­ple as that, and this data’s being record­ed and you’ll be able to down­load it in a spread­sheet that has all of the coor­di­nates, but I don’t have time to tell you how to ana­lyze that, but hope­ful­ly we’ll be releas­ing some codes to help peo­ple do that on our website.

Alex Anwyl-Irvine:
Okay, so I think that’s about it and thanks for lis­ten­ing to me, you can vis­it mouseview.org, which has detailed doc­u­men­ta­tion, links to our pre-print and exam­ples of var­i­ous dif­fer­ent lev­els of imple­men­ta­tion I spoke about. The Zone on Goril­la is in closed beta, so you can get it added to your account by con­tact­ing them, using a form. I’d sug­gest doing this, if you just want to play around with dif­fer­ent exper­i­ment designs, it’s free to make an exper­i­ment, so it’s a pret­ty cool way of work­ing out what you can do. And for updates in the future, then fol­low me on Twit­ter, but a shame­less pro­mo­tion, and also fol­low any­one in this sem­i­nar because I think we’ll all be doing things in the future.

Alex Anwyl-Irvine:
I’d like to thank Edwin and Tom who have been great at mak­ing this project hap­pen, but also Will at Goril­la who led pro­gram­ming the drag and drop zone and also Thomas Pronk and Rebec­ca Hirst, Psy­choPy, for imple­ment­ing it there, and mak­ing our exam­ples. Okay. Thanks.

 

Get on the Waitlist

BeOnline is the conference to learn all about online behavioral research. It's the ideal place to discover the challenges and benefits of online research and to learn from pioneers. If that sounds interesting to you, then click the button below and sign up for our newsletter. You will be the first to know when we release new content and open applications for BeOnline 2022.

With thanks to our sponsors!

MouseView.js: A no-eyes, mouse-based, alter­na­tive to eye track­ing online. What you can use it for, and how to use it on Gorilla.sc