The cost of not exper­i­ment­ing? Learn­ings from the FCA

Dr Alexan­dra Chester­field:
Thanks, Jo, thank you for invit­ing me. My pre­sen­ta­tion on finan­cial reg­u­la­tion is going to feel like a come­down, I think. After swim­ming pools and night clubs and col­or exper­i­ments. So apolo­gies in advance. As I said, I’m Alex Chester­field. I cur­rent­ly lead a behav­ioral risk team in inter­nal audit at one of the UKs biggest banks. I think it’s one of the first behav­ioral sci­ence teams in inter­nal audit. And the goal of our team is real­ly try and pro­tect our employ­ees and cus­tomers. And help, I guess pre­vent any prob­lems. I love the job of the team. It’s super inter­est­ing. I’m actu­al­ly here today to talk about some of the research ideas with for­mer col­leagues, at the UK’s finan­cial reg­u­la­tor, the FCA, who real­ly pio­neered using exper­i­men­tal meth­ods to inform finan­cial regulation.

Dr Alexan­dra Chester­field:
And again, for those who were think­ing, what the hell is reg­u­la­tion? Reg­u­la­tion ulti­mate­ly tries to make finan­cial ser­vices mar­kets, work bet­ter every­day for con­sumers? And then you think a lot of our lives, every time we buy some­thing, our cur­rent accounts, maybe if we own a house. Or the mort­gage. When we buy things on a cred­it card. So, it’s one of those non-trust ser­vices, on those mar­kets. It’s in the back­ground, it’s very much a hygiene fac­tor, but it does make a lot of things in our lives day-to-day pos­si­ble. So the reg­u­la­tor exists to try and make the mar­kets work, as well as pos­si­ble for con­sumers. So I’ll be talk­ing main­ly about my expe­ri­ences at the FCA, as most of the research that my for­mer col­leagues and some of myself did, is pub­lished. So lis­ten­ers and watch­ers, now can go and read up on those stud­ies in much more detail.

Dr Alexan­dra Chester­field:
And I should also say as an ex reg­u­la­tor, and some­one cur­rent­ly work­ing in inter­nal audit, this ses­sion is very much my per­son­al reflec­tions, and don’t reflect the views of any cur­rent or pri­or orga­ni­za­tion, that I work for. A bit more for­mal­ly, I stud­ied Cog­ni­tive and Deci­sion Sci­ence at UCA, so I think I saw a glimpse [inaudi­ble 00:02:01] on the Bed­ford Way, which I have very fond mem­o­ries of. I’m also actu­al­ly co-author­ing a book, on why we divide and how we can bring peo­ple back togeth­er. It’s called Poles Apart. This is a plug. It’s pub­lished in Sep­tem­ber with Pen­guin Ran­dom House. But back to the ses­sion. I was going to touch on four things. So first of all, one is my role at the FCA. Sec­ond, why the FCA runs exper­i­ments. What chose to do that? Third, what types of exper­i­ments were run, when I was there, and the cur­rent team is still very much doing exper­i­ments and they’re growing.

Dr Alexan­dra Chester­field:
It’s a fan­tas­tic team. I can high­ly rec­om­mend work­ing there. Jobs, do come up at the FCA. And then, four, what we learned in the process. I’m not going to use slides. No, no slides. It’s just going to be me. Just going to be me talk­ing. So I hope that’s okay. So, first of all, num­ber one is my role at the FCA. I was hired to help bring in a more real­is­tic view of human behav­ior, and into the behav­ioral eco­nom­ics team at the FCA. So a lot of reg­u­la­tors are staffed by super love­ly, and very smart peo­ple, often with eco­nom­ics or legal back­grounds, to write reg­u­la­tion, or super­vise firms. Behav­ioral sci­en­tists, whether that’s eco­nom­ics, psy­chol­o­gy, neu­ro­science, social soci­ol­o­gy, are very much in the minor­i­ty. And that means that they lack an impor­tant per­spec­tive, espe­cial­ly giv­en their role is ulti­mate­ly to try and under­stand why firms behave the way they do.

Dr Alexan­dra Chester­field:
And then why con­sumers behave the way they do. And I pre­vi­ous­ly worked at Which, the con­sumer cham­pi­on, and used lots of mixed meth­ods. So qual­i­ta­tive and more quan­ti­ta­tive. To under­stand where con­sumers were fac­ing prob­lems, and chal­lenges in dif­fer­ent mar­kets. And how gov­ern­ment, or reg­u­la­tors, or busi­ness might be able to solve those prob­lems. So I worked at the FCA for three years. It is an incred­i­bly diverse team. There’re exper­i­men­tal econ­o­mists, behav­ioral econ­o­mists, neu­ro­sci­en­tist, biol­o­gists, and lin­guis­tic. Peo­ple with lin­guis­tics back­grounds. So it’s a real­ly diverse team. And I think that was a real­ly impor­tant part of mak­ing it effec­tive. And the goal of the team was to use meth­ods and insights, from behav­ioral eco­nom­ics, or behav­ioral sci­ence more broad­ly, to help inform the deci­sions of the orga­ni­za­tion. So a bit more specif­i­cal­ly, that means it’s informed that the reg­u­la­tions, or rules that were being made about how firms should behave. About how the FCA super­vise firms.

Dr Alexan­dra Chester­field:
And then also it late­ly, the team was apply­ing behav­ioral sci­ence to some of its own inter­nal oper­a­tions. So for exam­ple, how to pro­mote diver­si­ty, and inclu­sion in your orga­ni­za­tion. So that’s the first part, my role at the FCA. Sec­ond. I was going to touch on, was why the FCA runs exper­i­ments. It real­ly did pio­neer, and that’s much cred­it to Ste­fan Hunt and Paul Adams, who real­ly set the team up back in 2013. There’s a few things, just to zoom out for a moment, there was a few things that hap­pened, which meant banks and the wider ecosys­tem, so reg­u­la­tors, gov­ern­ment, et cetera, need to start think­ing about behav­ioral sci­ence, and why the Behav­ior eco­nom­ics team was set up at the FCA. So first of all, start at the finan­cial cri­sis of 2008. That was a pre­cur­sor for the UK gov­ern­ment, their deci­sion to split the finan­cial regulator.

Dr Alexan­dra Chester­field:
It was pre­vi­ous­ly the FSA into two. So one part of the reg­u­la­tor, or one part of the reg­u­la­tions, which was look­ing after, what’s called the sta­bil­i­ty of finan­cial mar­kets. So mak­ing sure that all the mon­ey’s mov­ing in the right places. And that the mar­kets as a whole, weren’t going to col­lapse. When that moves to, what is known as the Pru­den­tial reg­u­la­to­ry author­i­ty, under the Bank of Eng­land. And then the FCA, that’s the part that I was in. The behav­ioral eco­nom­ics team, and that would focus on the oth­er part of reg­u­la­tions. So what’s called con­duct reg­u­la­tion. So con­duc­t’s basi­cal­ly how your funds behave, and that super­vised more than 56,000 finan­cial ser­vices firms. And over 140,000, what are called approved indi­vid­u­als. That’s peo­ple like giv­ing finan­cial advice, for exam­ple. So the cre­ation of the FCA in 2013, which focus­es very much on con­duct, that’s the C of FCA, led to new objec­tives from par­lia­ment, includ­ing the need to pro­mote competition.

Dr Alexan­dra Chester­field:
So com­pe­ti­tion across finan­cial ser­vices mar­kets. So peo­ple switch­ing again, and I guess the the­o­ry is, is that more peo­ple switch, that would dri­ve up val­ue, and qual­i­ty, and low­er prices for cus­tomers. It’s a good thing. You want more com­pe­ti­tion. So that was one of the new reg­u­la­tors objec­tives in 2013, plus a renewed focus on con­sumers, so pro­tect­ing con­sumers. And if you want to pro­tect con­sumers, and pro­mote com­pe­ti­tion, then you’ve got to under­stand both how con­sumers behave, and how that affects com­pe­ti­tion in mar­kets. So on that basis, the FCA had a clear man­date to start think­ing, and using behav­ioral sci­ence. But as well as the finan­cial cri­sis, I think in par­al­lel, you had the launch of Nudge, for exam­ple. I think that was pub­lished back in 2008. And one of the biggest authors, Cass Sun­stein, who was appoint­ed as Oba­ma’s reg­u­la­tion chief in 2009. The books oth­er author, Thaler, advised the UK’s new prime min­is­ter, Cameron.

Dr Alexan­dra Chester­field:
[inaudi­ble 00:07:36] the behav­ioral insights team. So there’s lots of noise about behav­ioral sci­ence, hap­pen­ing at the same time, or short­ly after the finan­cial cri­sis hap­pened. So I think that’s broad­er pop­u­lar­iza­tion of behav­ioral sci­ence. Obvi­ous­ly, it’s been decades and decades of behav­ioral sci­ence, but the pop­u­lar­iza­tion of it, the fact that gov­ern­ments are start­ing to use behav­ioral sci­ence, com­bined with the finan­cial cri­sis, led pol­i­cy­mak­ers, aca­d­e­mics with­in the great and the good, could have asked that ques­tion, why did the finan­cial cri­sis hap­pen? Despite the pletho­ra of rules, and reg­u­la­tion, and gov­er­nance and com­pli­ance? Why did we get this finan­cial cri­sis, that hard­ly any­one actu­al­ly pre­dict­ed? So those two things com­bined, I think, led to a much more focus on behav­ior, with­in finan­cial orga­ni­za­tions. Also, cul­ture, orga­ni­za­tion­al cul­ture, and then how that influ­ences busi­ness mod­els, prod­ucts, cus­tomer jour­neys, and inter­ac­tions with cus­tomers. The two peo­ple that set up the behav­ior eco­nom­ics team, knew it was­n’t just enough, to bring in a the­o­ry about what might work, despite best intentions.

Dr Alexan­dra Chester­field:
And despite the best behav­ior eco­nom­ics the­o­ry, you still don’t real­ly know whether a rule will work in prac­tice, whether it will solve the prob­lem, and that it’s designed to do and to help cus­tomers. That led to the real­iza­tion that exper­i­ments, or ran­dom­ized con­trolled tri­als, so field tri­als, can help the FCA to be a more effec­tive reg­u­la­tor, and ulti­mate­ly help cus­tomers. So that’s where it start­ed. So third, third part of my ses­sion, I was going to touch on what types of exper­i­ments the FCA ran, and I guess to sum it up many. One of the first ones, this was before I joined. One of the first ones that the team did. I think this shows the jour­ney that the team went on, but one of the first ones, first ran­dom­ized con­trolled trials.

Dr Alexan­dra Chester­field:
So field tri­als that the team ran. This is by the founders, Paul Adams and Ste­fan Hunt, was on opti­miz­ing com­mu­ni­ca­tions that firms were send­ing to con­sumers. And these com­mu­ni­ca­tions that firms were send­ing, were try­ing to put things right that had gone wrong. So for exam­ple, if prod­ucts have been missed sold, these let­ters were telling cus­tomers, that actu­al­ly they could have a poten­tial claim. So, poten­tial rights to get com­pen­sa­tion, or address­es is more for­mal­ly now in finan­cial ser­vices. But the chal­lenge was, that firms were send­ing these let­ters, telling cus­tomers that they were poten­tial­ly owed mon­ey, no com­pen­sa­tion, but the response rate was very, very low. So the team, eco­nom­ics team, saw an oppor­tu­ni­ty to do a ran­dom­ized con­trol tri­al. So the focus of the research is very much on how to encour­age con­sumers, who may be, due redress.

Dr Alexan­dra Chester­field:
You may be due the oppor­tu­ni­ty to get mon­ey back, to respond to these let­ters. So how do we increase the response rate to these let­ters? The team worked on a real case, with a firm, that was vol­un­tar­i­ly writ­ing, to almost 200,000 cus­tomers, about the fail­ing that it had, had in its sales process. So sev­en treat­ments were designed. These were real­ly basic. This was done back in, prob­a­bly 2014. This was kind of in the ear­ly stages, I guess, of Nudge, and behav­ioral sci­ence. There’s sev­en treat­ments designed. So one, for exam­ple, was adding a mes­sage to an enve­lope, to try and encour­age peo­ple to open the let­ter in the first place, sim­pli­fy­ing the texts, send­ing a reminder, et cetera, and bul­lets was anoth­er one. So again, that was a way of sim­pli­fy­ing the text.

Dr Alexan­dra Chester­field:
I think they changed the name of the respon­dent. So the chief exec ver­sus the sales man­ag­er. So it almost chang­ing the mes­sen­ger again, to see what would work empir­i­cal­ly. In terms of what the results were, the con­trol let­ter, so that was the one that the firm was send­ing out his­tor­i­cal­ly, that had the very low response rate. The con­trol rate in the tri­al received a 1.5% response rate. So only 1.5% of all cus­tomers that the let­ter was being sent to you. So almost 200,000 were respond­ing. When I guess look­ing across the treat­ments, and what worked, and what did­n’t. The team found that salient bul­lets, salient bul­let points, which was mak­ing that text stand out more. So more promi­nent, to the read­er, was real­ly effec­tive. So that alone increased the response rates. So from 1.5% in the con­trol group to near­ly 4%.

Dr Alexan­dra Chester­field:
So again, that’s still low, at an absolute lev­el, but rel­a­tive­ly, a jumped from 1.5% to 4% is quite effec­tive. Just for adding some bul­let points. But, when the bul­let points were com­bined with oth­er treat­ments, that boost­ed the response rate to 12%. So again, arguably an over­all absolute lev­el, 12% is still quite low, but at a rel­a­tive lev­el from 1.5% to 12%, that’s real­ly effec­tive, That’s equiv­a­lent to about an addi­tion­al 20,000 peo­ple respond­ing, to get mon­ey that they’re owed. So here, the learn­ing from that first ran­dom­ized con­trolled tri­al with a firm, was that small changes do work. And then inter­nal­ly, in the orga­ni­za­tion, your reports are get­ting that ear­ly use case, a quick win, we can call it. To get buy-in to doing more field tri­als and get­ting more resource, and get­ting more peo­ple into the team to do more of these trials.

Dr Alexan­dra Chester­field:
And if any­one’s inter­est­ed in read­ing on about that in more detail, if you just Google occa­sion­al FCA, occa­sion­al paper, num­ber two, you can read all about the tri­al, and the results, and the stats, in more and more detail. Actu­al­ly, just before I left a cou­ple of years ago, I ran a sim­i­lar tri­al with col­leagues, with pawn­broking cus­tomers, at one of the UKs biggest pawn­broking firms. So pawn­broking is, it’s actu­al­ly a very old method of lend­ing mon­ey. Sim­ply read lots of Dick­ens. It fea­tures in there, so it’s a way of peo­ple give a watch, and they get a loan essen­tial­ly for that good. And then if you can’t repay the loan, then the shop, the pawn­broking shop keeps your watch or your item jew­el­ry, as kind of col­lat­er­al. So any­way, but there’s a sim­i­lar prob­lem. So here, with the pawn­broking from mon­ey was on the table.

Dr Alexan­dra Chester­field:
Pawn­broking cus­tomers were owed mon­ey, and these cus­tomers are quite vul­ner­a­ble as well. So the actu­al absolute amounts that were owed, were quite low, but again, for the cus­tomers, it was quite a lot of mon­ey, but they weren’t claim­ing the mon­ey. So from a behav­ioral per­spec­tive, that again was real­ly chal­leng­ing, but also inter­est­ing that there’s mon­ey on the table, that peo­ple are owed, but why aren’t they get­ting it? And again, if peo­ple are inter­est­ed to look at this, if you just Google FCA occa­sion­al paper num­ber 59, you can read all about this tri­al in more detail. But here, I want to tell you about this one, it’s inter­est­ing, because here, we inte­grat­ed design prin­ci­ples into the start. So way before we even got to the ran­dom­ized con­trolled tri­al with the firm, we worked with a design aca­d­e­m­ic at Cardiff Mets, to under­stand more qual­i­ta­tive­ly, why these cus­tomers weren’t respond­ing to the com­mu­ni­ca­tions that our firm is send­ing out.

Dr Alexan­dra Chester­field:
We want to under­stand these peo­ple’s lives. None of us had used pawn broking firms before we could come up with some bar­ri­ers, but they were like­ly to be wrong. So we want­ed to speak and under­stand peo­ple’s lives in con­text, in broad­er con­text, before design­ing an inter­ven­tion, to try and get them to respond, and get the mon­ey that they were owed. So we worked with design­ers, doing some real­ly deep ethnog­ra­phy obser­va­tion work. And then we also co-designed with them, what might work to then test in the field tri­al. And anoth­er learn­ing was, I think it’s around hav­ing humil­i­ty, so again, I was work­ing in the FCA with some real­ly good behav­ioral data sci­en­tists, who were amazing.

Dr Alexan­dra Chester­field:
We had some of the super­vi­sors on board, again, who worked with the firm. We had peo­ple at the firm involved. And whilst behav­ioral sci­en­tists are good at many things, we’re not always good at design­ing com­mu­ni­ca­tions that, peo­ple will open, click, read, under­stand, take the required action. So I think hav­ing that humil­i­ty, and real­iz­ing that often you need oth­er dis­ci­plines involved in the mix, or the meth­ods, to try and solve your prob­lem, as it was a real­ly impor­tant learn­ing. From that first tri­al done years ago, to where we got to. And the result of that pawn­broking tri­al was that the behav­ioral­ly informed let­ter, which was, you can tell we have design­ers involved. It dou­bled col­lec­tion rates. Again, that was real­ly pow­er­ful, for the cus­tomers that we were help­ing. There was actu­al­ly a sec­ond inter­ven­tion that we tri­aled. That was much more focused on the inter­face that staff in the pawn­broking stores were using with customers.

Dr Alexan­dra Chester­field:
We did a lot of data ana­lyt­ics in the back­ground. That’s actu­al­ly in field. So when that is fin­ished, that will come out in good time. But as we learn more, so as the team pro­gress­es, as the team grew, from using the field tri­als, oth­er exper­i­men­tal meth­ods were used, includ­ed online exper­i­ments, Qua­si-exper­i­men­tal meth­ods. So where, doing a field tri­al, online exper­i­ments was­n’t pos­si­ble. What kind of met­ric meth­ods can we use to infer causal­i­ty, from the data that we get from firms. And also inte­grat­ing oth­er tools and insights, pri­or to get­ting to a field tri­al. Or to run an exper­i­ment to real­ly inform the design of the inter­ven­tion. Now, I was going to fin­ish. Jo, I can see you on, I was going to fin­ish on my five learn­ings. And I have anoth­er exam­ple of that, but I’ll just do the five learn­ings and leave it there.

Dr Alexan­dra Chester­field:
Okay. Right. So what did we learn in the process? Five learn­ings from doing exper­i­ments, at the FCA. One was, inte­grate exper­i­men­tal meth­ods with oth­er meth­ods, par­tic­u­lar­ly qual­i­ta­tive at the start, to under­stand why peo­ple are behav­ing in that par­tic­u­lar con­text, before you then test inter­ven­tions. First. Sec­ond is online exper­i­ments are so much cheap­er, and so much faster, and a real alter­na­tive to field tri­als, if the deci­sion con­text is replic­a­ble. So for exam­ple, in an online invest­ment con­text, that makes an ide­al sit­u­a­tion, rather than doing a field tri­al, to actu­al­ly doing sim­u­lat­ing that deci­sion online. Also, it does­n’t nec­es­sar­i­ly have to take the place of field tri­als. We also use online exper­i­ments. This is pub­lished as well. Pri­or to a field tri­al, to whit­tle down all the pos­si­ble inter­ven­tions that we could do, again to save time and mon­ey once we’re actu­al­ly in field work­ing banks.

Dr Alexan­dra Chester­field:
Learn­ing three, was to man­age the expec­ta­tions of senior lead­ers. This is like the biggest learn­ing, I think. There’s so much uncer­tain­ty ahead of going into an exper­i­ment. You have got no idea what works. I worked with some of the, this is not me. Some of my col­leagues were so bril­liant, so qual­i­fied, and even with the best col­leagues work­ing, the best aca­d­e­mics, a lot of the time stuff still does­n’t work, or it back­fires. And, actu­al­ly that is not a fail­ure. That is not a fail­ure. You still learn some­thing. So an exper­i­ment nev­er fails. You always learn some­thing. Unless the exper­i­men­t’s that bad­ly designed. But that’s hard. So that I think giv­ing man­ag­ing expec­ta­tions upfront to stake­hold­ers is real­ly, real­ly impor­tant. I’m phras­ing it as a, we are de-risk­ing this process. I found that helps.

Dr Alexan­dra Chester­field:
Num­ber four is the val­ue of an inter­dis­ci­pli­nary team. So again, you need that com­bi­na­tion of very quan­ti­ta­tive peo­ple. Who know the exper­i­men­tal meth­ods, who can ana­lyze data, who can push for­wards, but you also need peo­ple that can man­age inter­nal rela­tion­ships, stake­hold­ers, and the cre­ative skills designed to get treat­ments. And com­mu­ni­cate those to stake­hold­ers. And then num­ber five, despite best inten­tions treat­ments, don’t always work. So that ini­tial effect that you get, which can have such a great impact. And don’t always work in the longer term. So look at longer term out­comes, as well as that short term effect on deci­sions. That would be my fifth learning.

Jo Ever­shed:
That’s fan­tas­tic. Thank you very much. I, for one am very grate­ful, that there are reg­u­la­tors in the UK, who are sit­ting there, sort of act­ing on behalf of con­sumers. Mak­ing sure that com­pa­nies have to make the repa­ra­tions that they’ve been required to make. So thank you for that work there. One ques­tion that I want­ed to ask you is, which of the projects that you’ve worked on, have you been the most proud of? Because, I can see you’re doing real good in the work, but do you ever have that moment where you’re like, yes!

Dr Alexan­dra Chester­field:
Yeah, prob­a­bly the pawn­broking one that I men­tioned. A, I think, per­son­al­ly and self­ish­ly, I learned a lot on that doing a real life world tri­als. Where I was doing bits of cod­ing, bits of data analy­sis, and work­ing with some great col­leagues, who are real­ly tech­ni­cal­ly qual­i­fied. But main­ly I think, sec­ond­ly, main­ly because the cus­tomers that we were help­ing, you had a tan­gi­ble effect on real­ly vul­ner­a­ble cus­tomers, who actu­al­ly that mon­ey made a real dif­fer­ence in their day-to-day lives.