Intro­duc­tion to Goril­la Task Builder 2.0

Jo Ever­shed 0:00
which is Nick Hodges from goril­la who’s going to share how some of the new fea­tures that are avail­able in Task Builder 2, the new, new offer­ing from Goril­la. And then after that talk, we’re going to have a break. So stick around to dis­cov­er how the researchers Neil, Saloni, and Myles have cre­at­ed their stud­ies, and how in the future, it’s going to be even eas­i­er to cre­ate these stud­ies and deploy them online. Saloni, Neil, miles, thank you, you’ve all been absolute­ly bril­liant. Do stick around.

Nick Hodges 0:29
Oh, thank you, right. Let me just show this. Right. Can you see my slides? And hear me? Cool. Won­der­ful, thank you. So six years ago, a researcher, prob­a­bly just like many of you watch­ing, had been try­ing for months to get her stair­cas­ing tasks to work. And as you can see here, quite clear­ly got to the end of her teth­er. I’m sure many of you have felt like this at times. Six years lat­er, come on slides. Fast for­ward six years lat­er. And she’s built the same task in goril­la in around two hours. And Jade is now a col­league of ours here at Goril­la. And it was at this point where she man­aged to put the task togeth­er so quick­ly that we knew we were onto some­thing with our designs.

So hel­lo, every­one. Good morn­ing, I’m Nick. And I’m going to show you the next gen­er­a­tion of our tools, task buider 2. Now, our vision here at Goril­la is to give you a sin­gle plat­form with a com­mon visu­al lan­guage for ques­tion­naires for tasks for human com­put­er inter­ac­tion games, mul­ti­play­er, shops, and so on. And the big idea that we’re sort of run­ning with at the moment is that every­thing is nice­ly inte­grat­ed. And every­thing is, is self aware of all the dif­fer­ent com­po­nents that it’s work­ing with.

Now, that all sounds very high lev­el, but fun­da­men­tal­ly, I want to turn you from this poor per­son on the right to this much more, c’mon, relaxed per­son on the left, that’s what we’re shoot­ing for. So the first thing with task builder 2 that we real­ly want to do is just make it that much faster and eas­i­er to iter­ate on your task and add in dif­fer­ent ideas. So pic­ture this sce­nario where it does­n’t mat­ter whether you’re a PhD stu­dent or a post­doc, you’re a PI. I’m sure you’re all famil­iar with this sce­nario. So you’re build­ing imag­ine you’re build­ing a task. It’s a sim­ple lit­tle puz­zle, I’m sure some of you have seen this exam­ple before. It’s a lit­tle three by three grid with a miss­ing pic­ture in the bot­tom right. And you’ve got to say which one the right one is. So you’ve got some instruc­tions, and then we go through some tri­als. So I think that one’s prob­a­bly the cat. Not quite sure, let’s have a look. Let’s go for the lamp per­haps. And this one could be the burger.

So you build instruc­tions, you put some tri­als togeth­er, and then you go to your go to review it with your col­leagues, whether that’s your PR, your super­vi­sor, your review board, and some­one will prob­a­bly say oh, we should prob­a­bly do some prac­tice tri­als, which tells you whether you’re cor­rect or not. So we can see whether peo­ple are get­ting it. Great. So that’s nice and sim­ple, we could do that. Let’s go ahead and just add that straight in. So we’re going to add an object in here, we’ll grab the feed­back, we’ll put it straight in, move it around a bit, done.

We can actu­al­ly pre­view our tri­al straight in here. So we can see if this is work­ing. So if we click on the cat that’s on to the right answer, and we can see that that’s cor­rect in the stream down the right. And the snail is com­ing through, I think Cor­rect. Great. So when you go back, and you chose to go put in the feed­back now and say, Oh, this is real­ly cool. But I think we should add a screen after the prac­tice tri­als, sothey can do them again if they want. Okay, let’s do that. So here is the spread­sheet that’s dri­ving that task. So you’ve got instruc­tions, you’ve got the free prac­tice tri­als. And I’ve intro­duced a lit­tle prac­tice com­plete screen at the end. So here it is, is just a sim­ple bit of text with a yes, no. And all I’m going to add in here is abil­i­ty to just jump to a dif­fer­ent row of the spread­sheet. So at the end, we say, well, if they respond to they respond, Yes, then we want to jump back to the row where our prac­tice chart starts, which is still in our spreadsheet.

So if we go ahead and look at that, we can see this hap­pen­ing. So there we go. Here’s our instruc­tion stream. Again, we go through our prac­tice tri­als, we can see the feed­back is com­ing through now. And then oh, we made a bit of a hash of this that so yes, let’s do this again. Yes. So we go back, we do them again. Oh, I see. There we go. And there’s a chan­de­lier. And this one must be that one is the red burg­er. Okay, great. And then we say no, don’t do that again. And then it goes on through the main trials.

You go back to your super­vi­sor say, oh, that’s real­ly good. But actu­al­ly think­ing about this. Let’s not give them a choice. If they get if they get any­thing wrong, they just have to do it again. So again, we’ve made this nice easy to do we want to save, first we need to do is make sure we keep track of any mis­takes. So we’re going to add this save accu­ra­cy com­po­nent in here and say, we’re going to store the num­ber of incor­rect respons­es in a field called mis­takes. And then all we can do here is change our cri­te­ria. So instead of it being when they say yes, we’re going to say well, when­ev­er that mis­takes field

4:44
hits a cer­tain cri­te­ria, so when­ev­er that mis­takes field is greater than zero, then we’ll jump back now, and I don’t know about you like I can’t afford to spend my time doing this in code for every task doing this in the UI like this is so much faster and quick­er. And I think the key point for me is that these are thing these aren’t these are key impor­tant sci­en­tif­ic deci­sions that you want to be able to make about your par­tic­i­pants expe­ri­ence. You don’t want imple­ment­ing them to be bur­den­some, you want them to be quick and easy. You just want to just turn it on and away you go.

Now we’re going to hear more, I think lat­er today about improv­ing the expe­ri­ence of your par­tic­i­pants and how that gets you much bet­ter qual­i­ty data. But the key thing for me is it just needs to be real­ly easy to add all these things in. I was quite inspired watch­ing Myles talk just now par­tic­u­lar­ly well, all of it. But cer­tain­ly the bits goes down where he’s say­ing, well, now that we found these find­ings, he wants to try all this oth­er stuff with the syn­chro­nis­ing with the bal­le­ri­na with all these oth­er kinds of things. This is the kind of thing I’m talk­ing about where you can have ideas and get them into this thing real­ly nice and quickly.

So that’s a, that’s an exam­ple of how how we have reimag­ined your work­flow in our in our new tool­ing. I want to show you some of the ideas I’ve showed you in more detail and how they actu­al­ly inter­op­er­ate. So the first one is live pre­view. And the first point to make is that that pre­view pan­el in the cen­tre with the four response options, and the bit in the in the top, that is the actu­al live task engine run­ning for there in real time right there in the tool­ing. So when you’re mov­ing stuff around, so you can obvi­ous­ly you can lay stuff out visu­al­ly, you can, you can see exact­ly what the per­son is going to see in that win­dow. Every­thing is reac­tive and visu­al. So you can drag things around, you can resize them, we’ve got a grid to help you lay stuff out nice­ly, you can change the scale of that grid, if you want a bit more fine grained con­trol. If you want to be real­ly, real­ly exact, you can just turn the grid off com­plete­ly. And it will switch to a com­plete­ly pix­el based coor­di­nate sys­tem. But and then you can always switch back if you change your mind and put your read­just your stim­uli to put them where you want them.

As well as that you can see how this looks on dif­fer­ent sized screens. So you can just flick through these pre­views here and auto­mat­i­cal­ly dynam­i­cal­ly scale that pan­el to those screen res­o­lu­tions. Because it’s real­ly easy to see how it’s going to shake out in dif­fer­ent screens. On the debug pan­el over here, you can actu­al­ly look at indi­vid­ual rows of your spread­sheet. So you can jump into spe­cif­ic tri­als to see how they work. And this makes it real­ly easy to trou­bleshoot things.And final­ly, you can play the tri­als for real your­self, no need to have to go into pre­view and gets the right thing. And you can see on the right we’re see­ing the actu­al log­ic of what’s going on here. So if I choose this now you can see it’s been marked as incor­rect. As we go through that.

The next place this applies is also when it comes to spread­sheet ran­dom­iza­tion. So I’m sure many of you famil­iar with using the spread­sheet to ran­domise your tasks. So here is a spread­sheet for my task where I’ve got my instruc­tions and debrief. And I’ve got six tri­als going down there with their puz­zle and their response options all con­fig­ured up. So now, if you I want to ran­domise this, I’ve added a col­umn — col­umn here to just so all the tri­als I want to ran­domise have got a one in them and then and I can turn on ran­dom­iza­tion, adding ran­domised tri­als and then spec­i­fy the col­umn I want to ran­domise on. I can click the lit­tle ran­dom­iza­tion Pre­view but­ton, the lit­tle dice there. And I can just click it over and over again to see the ran­dom­iza­tion log­ic run­ning for real. And this gives me a pre­view of what’s going to hap­pen when my par­tic­i­pants and take part and how it might ran­domise them.

If I want­ed to do more com­plex ran­dom­iza­tion, so for exam­ple, the first one, I did is just ran­domise that, that set of tri­als. But here I’m going to I’m going to do it based on the same field. But I just want­ed to take a ran­dom sub­set, I’d want to take three of those tri­als, and then ran­domise them, so I can just reorder those. So we’re going to take three of that, that sec­tion, and do them in a ran­dom order. And again, we can see that hap­pen­ing. And we can see those tri­als com­ing through as a ran­dom sub­set of those in a ran­dom order. So all of this is live, all of this is reac­tive. And order, this is show­ing you exact­ly what’s going to hap­pen all com­plete­ly with­in the tooling.

The sec­ond big idea here is what we call the com­po­nent sys­tem. So now in a tra­di­tion­al sys­tem, each ele­ment on the screen is just a sin­gle thing. So just an image or a but­ton or some­thing like that. Now, the prob­lem with that approach is that you quick­ly end up with a sort of com­bi­na­to­r­i­al explo­sion of behav­iours, you need an image, a click­able image or drag­gable image and a click­able text and a drag­gable text and so on. And this that ends up being sort of a lim­it­ed and some­what brit­tle. So the solu­tions, this is com­po­nents. So instead of each of being one thing, each object is made up of sep­a­rate com­po­nents, and you can kind of con­nect them togeth­er like Lego bricks. So in the sim­plest case, you might want some­thing that is an image. So we add an image com­po­nent. We want you to be able to gen­er­ate response when you click on it. So we add a click response com­po­nent. If we want you to be able to drag it, we add a drag­gable com­po­nent. If you want it to start out invis­i­ble and then appear after 2000, 2000 mil­lisec­onds, you can add a trig­ger vis­i­ble com­po­nent and you set up one trig­ger so on the start of the screen, you can see that we’re going to set the vis­i­bil­i­ty to invis­i­ble and then a sec­ond trig­ger on time elapsed after two sec­onds, we’re going to set it to be vis­i­ble again.

9:45
And then for all of this behav­iour, if actu­al­ly lat­er on you change your mind and actu­al­ly notice sor­ry, this isn’t gonna be an image it’s going to be a video. All you need to do is swap out the image for a video and all the oth­er stuff behaves exact­ly the same way. All these oth­er com­po­nents don’t real­ly care what the the main thing you’re play­ing with is over here. And they all inter­op­er­ate real­ly, real­ly seam­less­ly. So we can see this in action now. So here’s, I’ve got an image here with a stim­u­lus on it. And let’s go through that trig­ger vis­i­ble one here. So you can see here, we can just the all of the UI here, reacts to the set­tings you’re putting in. So it’s all nice­ly aware, it only shows you the set­tings that you need. And it makes it real­ly, real­ly easy to con­fig­ure all these things visu­al­ly. So you haven’t had to touch any code to do all of this stuff. This is all hap­pen­ing here. Because the after two sec­onds delay, then my image appears.

Now, the next thing I was taught was bind­ing. Now bind­ing is a key con­cept in Goril­la, we had this in task builder one, but we’ve extend­ed it and made it more pow­er­ful and eas­i­er to use. The basic con­cept of bind­ing is you’ve seen is that when you have your your tri­al, which in this case, say has a fix­a­tion, a stim­u­lus and a response, well, the fix­a­tion is the same. And maybe the response items are the same, but we want to change the stim­u­lus with each tri­al. So what we want to do is basi­cal­ly inject the val­ue from our spread­sheet into that into that stim­u­lus box where we where we define what our stim­u­lus is going to be. Now, it could come from your spread­sheet, but it can come from oth­er places, you can also do it in the exper­i­ment tree. So if you’re ran­domis­ing peo­ple to two con­di­tions, you might want to spec­i­fy here at the manip­u­la­tion lev­el, well, group A is going to have a time lim­it, there’s bound to a manip­u­la­tion and in one con­di­tion, they get 10 sec­onds, now they get 20. And so this is all done in here so that when­ev­er so in this case, here’s the bind­ing that stim­u­lus to the to the spread­sheet, so you can just add it as a fixed one as a sta­t­ic one, or you click this lit­tle bind­ing but­ton, lit­tle chain links. And then you can choose the col­umn that you want to bind it to. So no hav­ing, no hav­ing to type in mag­ic things, it’s already aware of all the columns that you have, if you spec­i­fy a new one, it appears, the same thing goes for manip­u­la­tions. And final­ly the store. So the store is a new ver­sion of embed­ded data. And you can do the same kind of thing. So when you’re sav­ing stuff out to embed­ded data to store it lat­er, that all kind of hap­pens through this UI. And ide­al­ly, when you want to bind it back in again, to use it.

You’ve now got much more con­trol over respons­es. So the response pro­cess­ing has tak­en a big way up. Now, in most tri­als, one way or anoth­er, we present some­thing on the screen and wait for the par­tic­i­pant to do some­thing and record their response. Now, by default, you wait for response from the par­tic­i­pant and advance on the first one. But that’s not always what we want. The sim­plest thing you might want to do is score it. So you can add in a score at your screen. And you can mark it cor­rect if it match­es a par­tic­u­lar answer, you might want to only advance on the first cor­rect response, not just any response. So they can get it wrong as many times as they like and you only advance when they final­ly get it right. You might want to only advance on a spe­cif­ic response.

More get­ting more com­plex you might want to put in, you’re able to force them to do a cer­tain amount of effort, they might have to do 20 respons­es in order to get through to the next one. And this is a way in which you can essen­tial­ly take these indi­vid­ual respons­es and com­bine them into one sort of big­ger one that you then process lat­er. So and you can do the same thing with com­pound respons­es. And I’m going to show you an exam­ple of this in a moment, where we essen­tial­ly want to take a bunch of respons­es and to join them togeth­er as if they’re spelling out a word or some­thing like this. But again, the impor­tant point here is that all of these things are decou­pled and mod­u­lar. So you can com­bine them togeth­er in a way that works for you.

So here’s a good a quick exam­ple of a dig­it span, this is the screen where you actu­al­ly enter the num­bers at the end. So we’ve shown the par­tic­i­pant a sequence of three num­bers, and you’ve got to remem­ber them. So you can see if I just hit 123, we can see the response build­ing up in that pan­el on the right. And you can you can use a back­space there if they want to delete an answer. And we can see that that will come through in the pan­el on the right.Now each of those indi­vid­ual but­tons is just a reg­u­lar but­ton with a response on it. And it’s this com­pound respons­es piece that is doing the join­ing togeth­er. So what a com­pound response com­po­nent is doing is is say­ing okay, well, we’re going to wait until we’ve got at least three and only then do we pass it on through the pipeline to what we’ve got there is the score, which can then mark it against our cor­rect answer. You can use the same thing to do things like trail mak­ing. So this is just a this is just a sim­ple exam­ple. But you can imag­ine this could have a map or some­thing behind it, where they’re click­ing the things in order in order to build a trail. And again, you can see the respons­es com­ing through on the side. And it’s the same com­pound respons­es as join­ing all those things togeth­er. In order to basi­cal­ly amal­ga­mate all those indi­vid­ual clicks into one sort of canon­i­cal response.

14:21
So with all this togeth­er, there’s a huge amount of oth­er things that you can do here. Where you can do that we can do dif­fi­cul­ty ratch­et­ing. So where you have mul­ti­ple spread­sheets of dif­fer­ent dif­fi­cul­ty might have a spread­sheet of easy tri­als, medi­um tri­als and hard tri­als. And again, we can do all this real­ly, real­ly eas­i­ly in the UI you sim­ply store the count of cor­rect and incor­rect answers then you add a cri­te­ria here for well for mov­ing between the spread­sheets that lets you do dif­fi­cul­ty ratcheting.

You could do stair­cas­ing and this is the one that Jade was build­ing ear­li­er. We do the same trick of count­ing their cor­rect and incor­rect answers. And then we add cri­te­ria for step­ping our tar­get val­ue in this case, how long they get to see a stim­u­lus for we step that tar­get val­ue up and down. We they get less time to see it, the bet­ter they do it You can do with­in task branch­ing. So jump­ing between rows and spread­sheets. So again, based on the cri­te­ria you want to have, you can jump to a dif­fer­ent row, that was how we did the going back to your, your prac­tice tri­als again. And just branch­ing with­in dif­fer­ent screens, again, with the same cri­te­ria, you can say, Well, based on what they say here, or what hap­pens here, we want to jump to this oth­er screen.

And so that’s where we are at the moment. This is live now on Goril­la, and you can all go and have a go with it. And com­ing soon is script­ing. So the main things we want to add in here, first of all, the abil­i­ty for you to build your own com­po­nents. Now, the cru­cial thing about this is that when you build a com­po­nent your­self, you can expose the set­tings that it needs as UI ele­ments in the actu­al tool­ing. So when you’ve built your com­po­nent, you’ll be able to add a lit­tle con­fig­u­ra­tion menu, just like all the ones that you’ve seen. So peo­ple try to con­fig­ure how they want your com­po­nent to work. And you’ve got access to all of the bind­ing tech and every­thing else that’s in there. So oth­er peo­ple using your code, they don’t need to copy in your script and mess around with it, they can, it will come up for them just like any oth­er any oth­er piece in the tools.

And all of the things that I’ve shown you whether they’re visu­al things, like images, or videos or any­thing like that, you can build those any­thing to do with behav­iour, whether they show or hide whether they trig­ger or this oth­er kind of stuff, any of the response pro­cess­ing things. And indeed any oth­er ran­dom­iza­tion things. They’re all just com­po­nents, and you script them all in the same kind of way with the same set of inter­faces. So it’s all real­ly, real­ly con­sis­tent and easy to get into.

And then final­ly, the oth­er bit that’s com­ing soon is our new evo­lu­tion of open mate­ri­als. So obvi­ous­ly at the moment, in open mate­ri­als, you can share exper­i­ments and tasks, but the evo­lu­tion we want to put in there as you can also post com­po­nents to that as well. So when­ev­er you build some­thing that’s real­ly use­ful and reusable, you can post that to open mate­ri­als. And it’ll actu­al­ly come through in the UI for peo­ple to put into their own tasks.

I can see we’re run­ning a bit over time, there’s one oth­er thing I want­ed to show you quick­ly. So this is a very sim­ple demo we made for doing your UI and UX research. So here we are, I’ve got the goril­la web­site, mount­ed in a tablet frame. And what I’ve done is I’ve added lit­tle hotspots, which is essen­tial­ly a click­able region of the screen over the top of the image to see how they nav­i­gate through this UI. And what I’m inter­est­ed in is whether they go through the signup flow where they go and look at the tools. And again, this is done real­ly sim­ply, you can see in the tool­ing here, we’ve got here’s the land­ing page, you see there you can see that checker­board is where the where the hotspot is. And then if you go into the signup one, we’ve got two more there one if they click on the signup flow. And one if they click on the head­er, and in the screen, there’s my branch­ing, which basi­cal­ly lis­tens for a response and then jumps to a par­tic­u­lar screens based on where they’ve clicked.

So all of that lets you do things like so what we’re real­ly try­ing to do is expand out all these oth­er dif­fer­ent kinds of areas of research that you can do in these tools. And we are on time, so I was going to I think mul­ti­play­er we’ve done before, but we can come back to that lat­er. And the last thing I want­ed to share with every­one is that obvi­ous­ly Task Builder 2 is out now. Now we are we’re see­ing lots and lots take up for it, which is real­ly excit­ing to see what peo­ple are build­ing with it. And final­ly, I just want­ed to tease that we are also build­ing ques­tion­naire builder 2. I know there’s been a lot of appetite for this amongst Goril­la users. So all the good­ies that you want and expect from a pow­er­ful ques­tion­naire builder is going to be there. So whether respons­es are option­al or required, con­di­tion­al, con­di­tion­al ques­tions, branch­ing ‚scor­ing, being able to accept your URL para­me­ters in from oth­er sources and indeed the script same kind of script­ing sup­port that I’ve talked about for task builder 2 that’s going to be in ques­tion­aire builder 2 as well, and that is all com­ing soon as well. And so that is every­thing that we’ve been build­ing at the moment and we’re real­ly excit­ed to see where you can take it.

Jo Ever­shed 18:55
Fab­u­lous, thank you Nick for shar­ing the improve­ments to Goril­la task builder, and where all of that work is going. That was absolute­ly brilliant.