Introduction
It’s a commonly held belief that we learn from failure. But recent research has found the opposite: that failure can undermine learning. The results of this research raise questions about “phish-test-train” strategies to increase staff understanding of phishing attacks.
The wisdom of learning from failure is everywhere. The Harvard Business Review based an entire edition on it. Even Yoda says, in his unique manner, “the greatest teacher, failure is”. And, yes, we can and do learn from failure. But the reality of learning from failure requires time and space to reflect, dialogue to examine the failure and understanding what went wrong and what to do next time.
Researchers at the University of Chicago designed an experiment to test whether success or failure-oriented feedback would produce better outcomes. Overwhelmingly, across five studies and more than 1600 participants, they found that success feedback was the more effective method.
“Participants answered binary-choice questions, following which they were told they answered correctly (success feedback) or incorrectly (failure feedback). Both types of feedback conveyed the correct answer. However, on a follow-up test, participants learned less from failure feedback than from success feedback. This effect was replicated across professional, linguistic, and social domains—even when learning from failure was less cognitively taxing than learning from success and even when learning was incentivised. Participants who received failure feedback also remembered fewer of their answer choices.”
In the words of the researchers: “Failure is ego threatening, which causes people to tune out and miss the information on offer.”
Phishing Simulations are commonplace and rely on delivering guidance to users when they fail a phishing test. An ideal teachable moment? This research suggests not.
Given that users “tune out and miss the information on offer”, the moment of failure in a phishing test may not, in fact, be a good time to impart learning. No matter how simply, or gently done, the recipients may not be receptive to learning. Worse they may feel tricked and embarrassed. We have interviewed a number of phish-test “victims” who could not even remember clicking a link; and so, would be unlikely to recall failure feedback.
But phish-test-train regimes have lowered click rates. Why? There could be several explanations: for example, the Hawthorne effect which “suggests that study subjects' behaviour or study results are altered by the subjects' awareness that they are being studied or that they received additional attention.” (Fernald, Coombs, DeAlleaume, West, & Parnes, 2012)
Or, perhaps, as one awareness practitioner noted “I think I’ve trained my staff to spot fake phishing emails… I’m not that clear on whether they can spot real ones!”
We aren’t alone in noting that phishing test and train solutions are not the be-all and end-all (see the NCSC view on phish-test-train). Phishing tests can have their place as part of the toolbox (setting a baseline, understanding susceptibility), but not as the whole toolbox. And ideally not for training.
A second experiment added observers. The researchers found that these observer “participants learned just as much from other people’s failure as from others’ success. Thus, when ego concerns are muted, people can tune in and learn from failure.” It turns out we can learn more effectively from other people’s failures than from our own!
That’s part of the reason why, when it comes to security awareness, short and sharp messages, containing a relatable narrative (perhaps about how people have fallen for phishing or other cyber threats) can be highly engaging.
Combine that with actionable steps people can take to protect themselves and you'll have an effective awareness strategy,