The online behav­iour­al Research Con­fer­ence 

Where ambi­tious behav­iour­al sci­en­tist come and learn

Speak­ers

Online research is #trend­ing, but is it good sci­ence?
Jo Ever­shed, Caul­dron Sci­ence
@EvershedJo

Caul­dron Sci­ence’s remit is to sup­port behav­iour­al researchers by pro­vid­ing online soft­ware that lib­er­ates behav­iour­al sci­ence from the lab so that it can be done in front-line set­tings such as schools, hos­pi­tals and homes. To accel­er­ate online behav­iour­al research we cre­at­ed Goril­la, a flex­i­ble and pow­er­ful plat­form for cre­at­ing behav­iour­al exper­i­ments and deploy­ing them online. In this talk, I will share with how online research and the tools to sup­port it fit into the wider dia­logue of the repli­ca­tion cri­sis, Open Sci­ence and impact on society.

[C]lick your screen: prob­ing the sens­es online
Andy Woods
@AndyTWoods

We are at the cusp of some far-reach­ing tech­no­log­i­cal advances that will be of tremen­dous ben­e­fit to research. With­in a few short years we will be able to test thou­sands of peo­ple from any demo­graph­ic with ‘con­nect­ed’ tech­nol­o­gy every bit as good as we use in our labs today — indeed more so. Here I dis­cuss on-web ver­sus in-lab, pre­dict­ed tech­no­log­i­cal advances and issues with online research.

Ensur­ing data qual­i­ty when you can’t see your par­tic­i­pants
Jen­ni Rodd, UCL
@JenniRodd

Researchers are often ner­vous about col­lect­ing exper­i­men­tal data online because unlike lab-based exper­i­ments they can­not meet their par­tic­i­pants and can’t direct­ly watch their behav­iour. I will sum­marise some of the issues that arise from ‘invis­i­ble’ par­tic­i­pants, and sug­gest pos­si­ble solu­tions that (i) main­tain data qual­i­ty and (ii) allow prin­ci­pled deci­sion about when online data col­lec­tion is (and is not) appro­pri­ate. I will also sug­gest that this prin­ci­pled approach to data qual­i­ty con­trol is often miss­ing in lab-based set­tings and that the lessons we have learned due to run­ning exper­i­ments online can be broad­ly applied to a range of exper­i­men­tal approaches.

The state of the art: a tech­ni­cal overview of what works today
Nick Hodges, Goril­la
@nikoraisu

If you were start­ing an online research project today, what could you build and how would you build it? In this talk, I’ll guide you through the dif­fer­ent com­po­nents that go into run­ning an online study and how they fit togeth­er. We’ll explore the wider land­scape of browsers, apps, servers and the cloud, and cov­er the details of screen refresh rates, periph­er­al polling rates, frame syn­chro­ni­sa­tion, brows­er tim­ing pre­ci­sion, auto­play and more.

Recruit­ing Par­tic­i­pants for Online Stud­ies: Chal­lenges and Solu­tions
Katia Damer, Pro­lif­ic
@ekadamer

Pro­lif­ic helps researchers around the world find par­tic­i­pants for online stud­ies. It is tai­lored for sci­en­tif­ic research, high­ly scal­able, and can flex­i­bly allow researchers to recruit any tar­get demo­graph­ics. In this talk, cofounder Katia will dis­cuss the var­ied chal­lenges when recruit­ing par­tic­i­pants online, and intro­duce Pro­lif­ic, which offers solu­tions to many of these chal­lenges. She will also dis­cuss some dif­fer­ences between run­ning lab and online stud­ies and how to design online stud­ies wise­ly in order to col­lect high qual­i­ty data and keep par­tic­i­pants hap­py. Final­ly, she will reflect on how the above con­sid­er­a­tions inform Prolific’s mis­sion to make online data col­lec­tion trustworthy.

Test­ing over 4 mil­lion par­tic­i­pants via a mobile app: Sea Hero Quest
Hugo Spiers, UCL
@HugoSpiers

Sea Hero Quest is an app designed to be a poten­tial diag­nos­tic for the ear­ly detec­tion of Alzheimer’s demen­tia. The app tests the users abil­i­ty to nav­i­gate a vir­tu­al boat through dif­fer­ent envi­ron­ments in search of sea crea­tures. It records nav­i­ga­tion per­for­mance and sends this data to sci­en­tists for analy­sis. The tasks were devel­oped by sci­en­tists, the work fund­ed by T‑mobile and built by the game devel­op­ers Glitch­ers Ltd. Ini­tial analy­sis of the results has revealed the chang­ing pat­tern of nav­i­ga­tion skill over the life-time between gen­ders and between coun­tries of the world. Future research with the data will seek to apply machine learn­ing to iden­ti­fy pat­terns of nav­i­ga­tion per­for­mance and relate these to demographics.

Mak­ing cog­ni­tive and lan­guage assess­ments avail­able online to prac­ti­tion­ers
Mairéad Mac­Sweeney, UCL
@MaireadMacS

Over 10 years, the UCL Deaf­ness, Cog­ni­tion and Lan­guage Research Cen­tre (DCAL) estab­lished many stan­dard­ised assess­ments of cog­ni­tion and lan­guage that were designed specif­i­cal­ly for peo­ple who were deaf. To ensure that these assess­ments were eas­i­ly avail­able to prac­ti­tion­ers work­ing with deaf chil­dren and adults, we devel­oped an online ‘por­tal’ to host the tests. We have been through, and are still going through, many iter­a­tions of the DCAL Por­tal to make it as user friend­ly as pos­si­ble. In this talk, I will share with you what we have learnt along the way.

Using online resources to improve children’s lan­guage skills: Prospects and Chal­lenges
Charles Hulme, Uni­ver­si­ty of Oxford

Oral lan­guage skills are a crit­i­cal foun­da­tion for the devel­op­ment of lit­er­a­cy skills and for for­mal edu­ca­tion more broad­ly. Our group is engaged in devel­op­ing lan­guage inter­ven­tions and lan­guage assess­ment tools. Online App based assess­ments hold huge promise for allow­ing class teach­ers to iden­ti­fy chil­dren with poor lan­guage skills who would ben­e­fit from tar­get­ed lan­guage inter­ven­tions. Such tools can also poten­tial­ly help teach­ers to under­stand more about the nature and preva­lence of oral lan­guage dif­fi­cul­ties. I will describe a sys­tem that we are cur­rent­ly devel­op­ing (ATLAS- Auto­mat­ed Test­ing of LAn­guage Skills). Chil­dren with poor lan­guage skills require suit­able edu­ca­tion­al inter­ven­tions. To date there is lim­it­ed evi­dence that com­put­erised reme­di­a­tion pro­grammes work (I will describe one that doesn’t). An impor­tant issue is the extent to which effec­tive child friend­ly tools can be devel­oped to improve the deliv­ery of lan­guage and oth­er edu­ca­tion­al interventions.

The Uni­ty game engine for a large-cohort, devel­op­men­tal study.
Alex Irvine, MRC Cog­ni­tion and Brain Sci­ences Unit, Cam­bridge
@alexanderirvine

The Uni­ty game engine is a cross-plat­form devel­op­ment envi­ron­ment. It has great poten­tial as a tool for col­lect­ing large amounts of behav­iour­al data, both in the lab, the class room and remote­ly — e.g. over the inter­net and on user’s own devices. I present an ear­ly ver­sion of an iPad app which uses a series of mea­sures to track cog­ni­tive, social and emo­tion­al devel­op­ment in pri­ma­ry school chil­dren. Describ­ing the process of devel­op­ing this soft­ware in-house, and the strengths of the Uni­ty envi­ron­ment for cre­at­ing scal­able and high­ly usable exper­i­men­tal applications.

Online data col­lec­tion in health psy­chol­o­gy research: what I have learned
Suzan­na For­wood, Anglia Ruskin Uni­ver­si­ty
@SuzannaWood

This talk will dis­cuss some of the key insights I have gained using online meth­ods to col­lect data in the area of health psy­chol­o­gy. I will take you on a per­son­al tour of the suc­cess­es and fail­ures of online research over the past 7 years, and some key things I wish I had known before I start­ed. This will touch on using online test­ing for large sam­ple sizes, using mar­ket research agen­cies to get epi­demi­o­log­i­cal­ly valid sam­ples, how to design an online task to ensure your data are as good as pos­si­ble, and what approach I now advo­cat­ed based on my experience.

Doing mul­ti-per­son inter­ac­tive social cog­ni­tion stud­ies in psy­chi­a­try: suc­cess­es, chal­lenges and frus­tra­tions
Vaugh­an Bell, Divi­sion of Psy­chi­a­try, UCL
Nichola Rai­hani, Exper­i­men­tal Psy­chol­o­gy, UCL
@vaughanbell
@nicholaraihani

Most social cog­ni­tion research nev­er tests peo­ple in actu­al social sit­u­a­tions. We con­duct research where peo­ple inter­act with each oth­er in con­trolled exper­i­ments, mean­ing online plat­forms that coor­di­nate inter­ac­tion are essen­tial for our stud­ies. This allows us to do large-scale, replic­a­ble, open sci­ence com­pat­i­ble stud­ies. How­ev­er, we have encoun­tered sev­er­al chal­lenges: devel­op­ment and host­ing costs for cus­tom soft­ware, short­com­ings of estab­lished plat­forms in terms of inter­ac­tion and open sci­ence com­pat­i­bil­i­ty, and a strong prej­u­dice in psy­chi­a­try jour­nals against online stud­ies. We’ll dis­cuss some poten­tial solu­tions and some remain­ing frustrations.

Online large-scale stud­ies with chil­dren out of the lab: the promise and the chal­lenge
Kate Nation, Uni­ver­si­ty of Oxford
@ReadOxford

Chil­dren vary enor­mous­ly in their cog­ni­tive and lin­guis­tic abil­i­ties. It is impor­tant that our exper­i­ments cap­ture this vari­abil­i­ty and that our the­o­ries try to explain it. This requires (a) large sam­ples and (b) reli­able met­rics. Time and resources for data col­lec­tion in schools are pre­cious and lim­it­ed. An ide­al solu­tion there­fore is to devel­op online exper­i­ments that can be com­plet­ed by chil­dren at school, or in the home. In this talk I will con­sid­er our ini­tial for­ages into the world of online test­ing with pri­ma­ry school chil­dren, focus­ing on some of the par­tic­u­lar issues that arise when work­ing with chil­dren, schools and families.

Online audi­to­ry train­ing and learn­ing stud­ies: promise and pit­falls
Fred Dick, Birkbeck

Our audi­to­ry envi­ron­ment is full of com­plex com­mu­nica­tive sig­nals, alerts, and behav­ioral­ly rel­e­vant acoustic back­grounds, all of which need to be learned de novo by each per­son over devel­op­ment and into adult­hood. What is more, the audi­to­ry sys­tem has to be flex­i­ble enough to learn to rep­re­sent and parse entire­ly new sound envi­ron­ments — for instance, dis­tin­guish­ing the myr­i­ad sources of traf­fic sounds that are impor­tant for one’s sur­vival while cycling through cen­tral Lon­don. The dura­tion of such learn­ing — over days, weeks, and months — makes it awk­ward, expen­sive, and often entire­ly imprac­ti­cal to study in tra­di­tion­al lab set­tings, so the pos­si­bil­i­ty of doing extend­ed, larg­er N, and more dynam­ic audi­to­ry learn­ing stud­ies online is very exciting.

We have been work­ing on a num­ber of such exper­i­ments (as have sev­er­al oth­er audi­to­ry groups) — I’ll dis­cuss some of our ini­tial par­a­digms and find­ings, as well as touch­ing on some of the ongo­ing chal­lenges that arise in design­ing and car­ry­ing out online audi­to­ry learn­ing and train­ing studies.