More Perils of Probability

British statistician, geneticist and probability guru, Peter Donnelly, recently gave a TED lecture about some of the common mistakes that people -- all people -- make when considering some fairly common probability scenarios. Peter is a very good speaker and present a self-deprecating wit that I find appealing. He is also very good at explaining what he does.

I was surprised, and delighted, to discover that he used as his primary example a scenario virtually identical to the one that I regularly employ to introduce lawyers to the perils of probability in tort litigation. The example involves a hypothetical person who receives a positive result to an HIV test. The underlying question is: "What are the chances the person is actually HIV positive?" My version of the example was derived from actual statistics concerning the incidence of HIV in a particular population (white women with no symptoms or risk factors). While Peter's example involves numbers that run in roughly the same ballpark, it is unclear whether he based them on real data or fabricated them for ease-of-use. Peter focuses his talk on the fact that such probabilities are hard to work with and that people are known to make particular kinds of mistakes when interpreting them. While I recognize such difficulties in my own presentation, I go on to focus on presentation strategies litigators can use to help juries correctly interpret probabilities. Anyway, it must be a really great example!

I am including here both Peter Donnelly's TED lecture and my own presentation slides on the topic. Peter gets to the common example about 40% of the way through his talk and you can find my treatment about 2/3 of the way through my slides (for those disinclined to watch both all the way through).

First Peter's talk.



Now, mine (No voice-over, I'm afraid. Feel free to invite me to present to your firm or lawyers group).

[Download presentation or view online]

The basic lesson to be learned here is that jurors are just not very good at math. Few have any experience with probability theory, especially anything related to very low-probability events. As I have discussed before, in relation to hindsight bias, tort cases typically stem from such extremely unlikely events. When jurors are faced with information processing tasks that are beyond their abilities, they typically resort to cognitive short-cuts. The most common of these is to use "intuition." Unfortunately, as Peter Donnelly so astutely illustrates, our intuitions about probabilities can often be completely off-the-mark. But the evaluation of reasonable care requires an accurate evaluation of the risks facing the care-taker. Litigators, then, have a Herculean task in getting jurors to understand the true underlying probabilities of a case.

I believe that two basic strategies are critical to getting jurors to appreciate the true risks faced by parties in a tort dispute. The first is reasoning by analogy. It is important to connect the choice problem faced by a decision-maker to something with which jurors are themselves familiar. Second, visual learning is key. Represent probabilities in a way that allows an average person to just "get it" by visual inspection. Ask yourself whether your exhibit passes the ol' "interocular impact test."

Finally, I want to point readers who, might not have watched Peter Donnelly's presentation all the way through, to a very interesting and disturbing example he presents at the end of his talk. In England, a woman was convicted of having murdered her two children, both of whom died of "crib death." The jury was largely convinced to convict on the testimony of an expert who testified that the chance she was innocent was simply the probability of any baby dying of crib death squared. That is, the expert chose to ignore the obvious dependence of the two events. If environmental or genetic factors made it more likely that one child would die in this way, those same factors would, of course, increase the likelihood that another child in the same family would also die in this way. That is, the fact that the defendant's two children died of crib death actually made it less likely that either resulted from foul play -- exactly the opposite of the expert's testimony. And the scariest thing is that not a single person in the courtroom called him on it. No one appreciated the enormous error in reasoning that was being committed, resulting in an innocent woman being sent to prison. Fortunately, her conviction was later reversed on appeal and the expert in question was discredited.

So, in your next case, make sure that (1) you have the probability theory right, (2) your expert does, too, (3) your expert is prepared to teach the jury how to evaluate such probabilities, and (4) you support such efforts with well-designed visual aids, both to reinforce your expert's testimony and also to guide jurors who might remain "confused" by the math.

Some Podcasts Fill a Niche, Others Miss the Mark

A couple of weeks ago, I posted a LinkedIn poll about work-related podcast listening habits. I had a few reasons for being curious about this question. First, as I am considering hosting a podcast on jury issues, I wanted to get a sense of my potential audience. Second, while podcasts clearly comprise part of the "social marketing" landscape, they have received much less attention in the press (and on blogs, from what I can tell) than blogs, twitter feeds and social networking sites. Finally, I wondered whether the demographic trends typically associated with technological adoption and Web 2.0 applied similarly to podcast usage, especially given the ubiquitousness of iPods and other digital audio devices.

Using a LinkedIn Poll

A LinkedIn poll has certain advantages and other disadvantages. The main advantage is that one can easily broadcast its existence to one's entire LinkedIn network. In my case, that is about 250 people. In addition, one can post the link to the poll on the discussion boards of any LinkedIn groups to which one belongs. (I did so for about 5 groups). LinkedIn users who just like checking out the ongoing polls can also find and answer any poll they choose. Regular LinkedIn updates about recent activity of connections can serve to remind (or annoy) people that the poll is ongoing. Finally, LinkedIn provides a unique URL for each poll so it is possible to invite people to see/take a poll from Twitter, a blog or a website.

On the downside, a LinkedIn poll is limited to a single question with a list of discreet answers. So, there is no way to ask a series of nested questions or even to collect poll-specific data about respondents. Also, only LinkedIn members can answer a free poll. For a fee, LinkedIn will expand the subject pool outside of the LinkedIn community. Back on the positive side, there is an opportunity for respondents to leave comments.

Work-related Podcast Usage



As you can see from the graph above, I received 23 responses. Not exactly a deluge. As many of you in my network can attest, I tried pretty hard to encourage people to take the poll. It is hard to know whether the majority of folks who chose not to participate are people with zero interest or experience with podcasts, or simply weren't willing to take the few minutes to complete the poll. Hey, time is money. In something like this, the temptation to free-ride on the efforts of others is pretty high. I plan to leave the poll active for a while longer to see whether this blog entry inspires a few more people to take the poll. (<-- Hint, Hint)

So, what do we make of these results? Well, the first thing that stood out for me is that over half of the respondents indicated that they rarely or never listened to work-related podcasts. And this is from the small set of people sufficiently interested in the topic to take the poll. This suggests that the universe of available podcasts is not serving well the needs of the professional community. Most of my professional contacts are connected in some way to the legal profession, so the results are likely most relevant in that arena. Perhaps doctors and painters and chimney sweeps all have great podcasts. But, despite the recent proliferation of law-related podcasts, they don't seem to have developed a loyal following (at least according to my very limited sample).

All is not lost. Almost a third of my respondents indicated that they listen to a podcast related to their jobs weekly. This probably means that they actually subscribe to at least one podcast, perhaps more. Some podcasts produce new episodes weekly, but it is more common for new episodes to come out monthly. So, in order to listen weekly, it is probably necessary to subscribe to a few different ones. One person said that s/he listened to a work-related podcast every day. Boy, would I like to know what business that person is in.

Given the bimodal nature of the distribution, it seems that the podcast market hasn't really evolved to serve most professionals' needs. That said, certain niches must be getting served pretty well. I wonder if, during this early period in the market's evolution, related podcasts operate more as complements than substitutes. That is, a critical mass of podcasts on a topic needs to emerge so that professionals in that field see podcasts as a viable method of keeping up-to-date on industry developments.

Demographic Differences

Given the small sample size, these results are really only suggestive. I wouldn't dream to generalize from what we have here. First, there did not seem to be any real difference with respect to gender. In my sample, the heavy podcast listener was a woman. On the other hand, somewhat more women than men reported never listening to podcasts. Things are a little more interesting with respect to age, especially given the conventional wisdom about generational differences in technology adoption.



Well, as predicted, our podcast addict was a youngster. On the other hand, the rest of our heavy podcast listeners were over 35. The 35 - 55 age group made up more than half of our sample and their listening habits were all over the map. One lesson seems to be that one should not target professional podcasts exclusively to gen-x or gen-y types. Professionals of all ages are savvy enough to make good use of this technology (keeping in mind that everyone who took this poll was already savvy enough to be on LinkedIn).

What next?

I guess I am a little surprised that more respondents didn't indicate that they were podcast fans. This begs the question of what more systematic market research has discovered regarding podcast usage. There is a good longitudinal study of podcast listeners conducted by Bridge Ratings here. They actually revised down their estimates of podcast listening growth (2006 estimates compared to 2005 ones). On the other hand, a nice longitudinal study of social media usage by Inc. 500 companies, conducted by two UMass researchers, shows that the percentage of these companies that produced podcasts increased from 11% in 2007 to 21% in 2008. A nice predictive study of podcast usage is summarized here, with many graphs and tables.

I think that the podcast market is suffering from a matching problem. It is still unclear which type of content is effectively conveyed through podcasting. As video podcasting develops (I predict that most will be video within two years), the answer to this question might change dramatically. Podcasting will probably evaporate in certain fields where the demand just seems too weak. On the other hand, certain fields will have full podcast immersion, with a wide variety of high quality programming to choose from. Got any ideas about which industries will be in which category? Then, submit a comment below and put your predictive talents to the test.

One thing is certain. A lot more market research about podcast listening habits needs to be done. Right now, it seems like everyone is surfing the wave of technology adoption, firing off podcasts because everyone else is doing it. Eventually, good data will have to replace conjecture about where the podcast market is heading.

Avoiding the Black Swan Effect at trial

Nicholas Nassim Taleb's book, The Black Swan: The Impact of the Highly Improbable, introduced a wide lay audience to the the "black swan effect," whereby people tend to overgeneralize based upon extremely rare, but impactful events. More generally, in psychology, this is referred to hindsight bias. The basic idea is that highly improbable events tend to get noticed. They stick in the brain and are easily accessed mentally. As such, people tend to underestimate just how improbable the event was. In addition, most people are really crummy at comprehending either really large or really small numbers. So, for example, an average person can't really distinguish the difference between a one-in-a-million chance and a one-in-ten-million-chance. Both are pretty much "never gonna happen." By contrast, people are pretty good at appreciating the difference between a coin flip and a one-in-twenty chance. Mathematically, the two comparisons are identical. The first event in each pairing is ten times more likely to happen than the second one.

Recently, Taleb has teamed up with Daniel G. Goldstein, and Mark W. Spitznagel to apply the black swan effect to corporate risk management in an article in the Harvard Business Review, enititled, "The Six Mistakes Risk Managers make in Risk Management." The focus of the article is to help managers recognize hindsight bias in their own decision-making. They caution these risk managers not to think they could have predicted these events from the past. Not should they use them to predict the future.

Given how much difficulty even seasoned risk managers have overcoming hindsight bias, it should not be surprising that ordinary people find the problem almost insurmountable. The problem is that many torts are the results of Black Swan events. A very improbable event takes place, adversely impacting someone in a profound way. That person sues and some company finds itself defending a failure to predict and prevent the unforeseeable. So, a risk manager also needs to understand this issue because it profoundly affects her company's performance in court.

I wrote an article on how (poorly) jurors handle issues of statistics and probabilities for The Jury Box, back when it was a newsletter. In addition, I have given several presentations on this topic, some slides for which can be found here. Given the timeliness of the topic (Really, it's timeless), I am reprinting that original article below. I hope you find it instructive.

A jury trial can be Risky Business (Don’t worry – not all issues will make reference to Tom Cruise). In addition to the inherent uncertainty associated with putting your case in the hands of a group of laypersons, a jury trial is complicated by the fact that jurors themselves are notoriously unreliable at evaluating risk. In this issue I focus on how juries handle probabilities, risks and cost-benefit analyses.

If it weren’t for bad luck, I’d have no luck at all
Most litigation involves a dispute over an unlikely event. A patient develops an allergic reaction, brakes fail, a grape rolls down an aisle to precisely the place where a little old lady is about to take her next step. These low-probability events turn into legal questions of foresee-ability, reasonable care and adequate notice, all of which turn, to one degree or another, on just how unlikely the jury believes the event to have been.

One common problem that jurors have evaluating probabilities is known as hindsight bias. When someone learns of a low-probability event having actually occurred, there is a tendency to treat it as if were much more likely than it was. If a juror places greater likelihood on the event, she will believe it to have been more easily anticipated and will assign greater urgency to guarding against it. The result is that defendants are often blamed for not anticipating and preventing truly freak accidents.

An alarming second order effect is that the more bizarre the circumstances, the greater the hindsight bias. This may be because really weird circumstances are more easily remembered and recalled by jurors.

One strategy for overcoming hindsight bias is to argue by analogy to something with which jurors are likely to be familiar. The problem usually confronts defense counsel, so it is also wise to choose an analogy for which jurors might feel some responsibility. For instance, if one argues that an allergic reaction to a medication is as common as an automobile accident caused by a sneezing driver, jurors may conclude that the situation was not very dangerous, given that they never pull over their cars just because they need to sneeze.

All costs with no benefits
The economic theory of reasonable care in tort goes back at least as far as Learned Hand’s opinion in Carroll Towing. A cost-benefit analysis showing that all economically efficient precautions were taken is supposed to be a defense to a charge of negligence. Many industry regulations make such calculations mandatory. One might imagine, then, that jurors would look favorably upon companies who perform cost-benefit analyses. One would be dead wrong. Typically, whatever appreciation that jurors might have for a company thinking hard about safety is overwhelmed by their discomfort in reducing human pain and suffering to a mathematical calculation, especially one involving money.

Several empirical studies have shown that defendants are almost always punished for performing cost-benefit analyses, regardless of how clearly the calculations support the measures taken. Plaintiffs’ attorneys are wise to play up the cold, callous, calculating nature of the defendant’s methods. By contrast, defense counsel has the difficult task of convincing the jury that her client cares about safety without the testimony being reduced to probabilities, statistics and dollar signs. Again, reasoning by analogy is often the best policy, alerting jurors to the many cost-benefit calculations they perform in their every day lives, with a focus on those costly precautions most people choose not to take. For instance, it is clearly safer for children to wear helmets on playgrounds but almost no parent makes her children wear them.

It is also worth noting that companies are actually punished for placing a higher value on human life in their cost-benefit analyses. While this high value might help marginally in avoiding liability, it creates a costly anchor when jurors are calculating damages.

The zero risk fallacy
Many jurors mistakenly believe that it is possible to make products, services and treatments absolutely safe. They conclude that any risk of loss or injury is unacceptable. They have essentially adopted a strict liability standard despite the law to the contrary. Others have simply decided that manufacturers or service providers, rather than consumers, should be responsible for all safety precautions because of perceived wealth or knowledge advantages.

Many jurors are troubled by the idea of bad things happening to innocent people. Some conclude that the world is unfair and that the poor victim is entitled to be compensated for her loss. The only source from which the jury can take money is the defendant, so liability is attached despite conclusions that the defendant did nothing wrong. The inadmissibility of evidence of insurance can exacerbate this problem since jurors often assume that the absence of any mention of insurance means that the plaintiff had none.

Plaintiffs face their own risks
Juror difficulties with risk and probabilities do not always benefit plaintiffs. Plaintiffs who are engaged in risky activities are sometimes entitled to compensation because defendant’s conduct unacceptably increased the risk. Jurors sometimes conclude that risk-takers implicitly assume all responsibility for their well-being. For instance, a juror might think, “Hey, skiing is a dangerous sport. If you get hurt, you have no-one to blame but yourself.” A plaintiff will have trouble convincing such a juror that the ski-binding manufacturer is liable for her injuries.

In most cases, both parties have behaved imperfectly. Some jurors implicitly adopt a contributory negligence rule, whereby any fault by the plaintiff bars compensation. I recommend that counsel make sure that the jury is given a very clear instruction on negligence. Ideally, the jury should be given the instruction in advance of opening arguments (an increasingly common practice, endorsed by the ABA). The idea is to get the jury to focus as quickly as possible on the defendant’s conduct.
I also recommend that plaintiff’s counsel consider a “de-fanging” strategy, whereby the plaintiff owns up to any personal mistakes. This will prevent defense counsel from raising the plaintiff’s failings in a manner that suggests to the jury that they somehow excuse the defendant’s conduct. If the plaintiff conveys confidence in the legitimacy of her claim despite a full appreciation of her own shortcomings, the jury is more likely to do so, as well.