More Perils of Probability

British statistician, geneticist and probability guru, Peter Donnelly, recently gave a TED lecture about some of the common mistakes that people -- all people -- make when considering some fairly common probability scenarios. Peter is a very good speaker and present a self-deprecating wit that I find appealing. He is also very good at explaining what he does.

I was surprised, and delighted, to discover that he used as his primary example a scenario virtually identical to the one that I regularly employ to introduce lawyers to the perils of probability in tort litigation. The example involves a hypothetical person who receives a positive result to an HIV test. The underlying question is: "What are the chances the person is actually HIV positive?" My version of the example was derived from actual statistics concerning the incidence of HIV in a particular population (white women with no symptoms or risk factors). While Peter's example involves numbers that run in roughly the same ballpark, it is unclear whether he based them on real data or fabricated them for ease-of-use. Peter focuses his talk on the fact that such probabilities are hard to work with and that people are known to make particular kinds of mistakes when interpreting them. While I recognize such difficulties in my own presentation, I go on to focus on presentation strategies litigators can use to help juries correctly interpret probabilities. Anyway, it must be a really great example!

I am including here both Peter Donnelly's TED lecture and my own presentation slides on the topic. Peter gets to the common example about 40% of the way through his talk and you can find my treatment about 2/3 of the way through my slides (for those disinclined to watch both all the way through).

First Peter's talk.



Now, mine (No voice-over, I'm afraid. Feel free to invite me to present to your firm or lawyers group).

[Download presentation or view online]

The basic lesson to be learned here is that jurors are just not very good at math. Few have any experience with probability theory, especially anything related to very low-probability events. As I have discussed before, in relation to hindsight bias, tort cases typically stem from such extremely unlikely events. When jurors are faced with information processing tasks that are beyond their abilities, they typically resort to cognitive short-cuts. The most common of these is to use "intuition." Unfortunately, as Peter Donnelly so astutely illustrates, our intuitions about probabilities can often be completely off-the-mark. But the evaluation of reasonable care requires an accurate evaluation of the risks facing the care-taker. Litigators, then, have a Herculean task in getting jurors to understand the true underlying probabilities of a case.

I believe that two basic strategies are critical to getting jurors to appreciate the true risks faced by parties in a tort dispute. The first is reasoning by analogy. It is important to connect the choice problem faced by a decision-maker to something with which jurors are themselves familiar. Second, visual learning is key. Represent probabilities in a way that allows an average person to just "get it" by visual inspection. Ask yourself whether your exhibit passes the ol' "interocular impact test."

Finally, I want to point readers who, might not have watched Peter Donnelly's presentation all the way through, to a very interesting and disturbing example he presents at the end of his talk. In England, a woman was convicted of having murdered her two children, both of whom died of "crib death." The jury was largely convinced to convict on the testimony of an expert who testified that the chance she was innocent was simply the probability of any baby dying of crib death squared. That is, the expert chose to ignore the obvious dependence of the two events. If environmental or genetic factors made it more likely that one child would die in this way, those same factors would, of course, increase the likelihood that another child in the same family would also die in this way. That is, the fact that the defendant's two children died of crib death actually made it less likely that either resulted from foul play -- exactly the opposite of the expert's testimony. And the scariest thing is that not a single person in the courtroom called him on it. No one appreciated the enormous error in reasoning that was being committed, resulting in an innocent woman being sent to prison. Fortunately, her conviction was later reversed on appeal and the expert in question was discredited.

So, in your next case, make sure that (1) you have the probability theory right, (2) your expert does, too, (3) your expert is prepared to teach the jury how to evaluate such probabilities, and (4) you support such efforts with well-designed visual aids, both to reinforce your expert's testimony and also to guide jurors who might remain "confused" by the math.

Community Attitude Surveys: Cheap, informative, effective

It was reported yesterday in the Modesto Bee that two-thirds of those surveyed already believed that Columbus Allen Jr. is guilty of killing California Highway Patrol Officer Earl Scott. Allen's defense team is attempting to use the survey in support of a change of venue motion, but the judge seems unlikely to grant the request. It is instructive to read the stream of vitriol posted by readers commenting on the article.

My god, they should of tried him the moment they arrested him... OFF TO THE GALLOWS BOY!

The evidence is clear. He killed Earl Scott. He is guilty, nothing else left to discuss, time for him to suffer and die a slow slow death.

This poor excuse for a life form was stupid enough to show up later with gunshot residue on his hands and arms. i am willing to bet he wasn't at the gun range practicing. guilty. fry him. this society is already paying for enough criminals and cannot afford them.

Of course everybody knows he's F*%&ing guilty! He murdered a CHP officer on 99 when everyone was commuting. I still remember that morning driving past that grim scene.
Why has this trial been put-off for so long? Get it the hell over with and sentence him!

Of course, this is a murder case, with lots of pretrial publicity. We should not be surprised to see strong public attitudes. In addition, the defense team has conducted the survey to achieve a very specific purpose: get the trial out of Stanislaus County. That said, a community attitude survey can be extremely helpful for civil litigation, as well.

Imagine you are defending a hospital in a medical malpractice case. Wouldn't it be useful to know whether the hospital is held in high esteem, as compared to others in the community? Are the people in this jurisdiction particularly dissatisfied with the quality of healthcare they receive? Is there an ongoing shortage of primary care physicians, causing patients to wait for appointments? Has the "torts crisis" argument taken hold in this community? These are all questions that can be answered with a community attitude survey. A survey of hundreds or thousands of people can be conducted for less money than a one-day focus group study with only a handful of subjects.

The judge in your case has authorized the use of a supplemental juror questionnaire, but she has made it clear that it is NOT going to be 75 pages long. She orders you to get your act together and submit no more than a dozen case-specific questions. How do you decide which questions to use? (After all, your trial consultant has written more than 30 truly outstanding questions for you) Commission a survey with your top candidate questions. This provides you an opportunity to analyze the responses of jury-eligible members of the community. You will undoubtedly discover that some questions just don't generate any useful range of answers. Others produce responses that don't seem to correlate with attitudes related to the case. The key is to find those questions that effectively distinguish between people who seem generally sympathetic to your case and those who do not. Now you can submit your short list of questions to the judge with confidence that they will produce results in jury selection.

Surveys have several advantages over other methods of pretrial research.

First, a survey typically involves many more subjects. This means more data and more confidence in the results.

Second, a survey can be conducted in the background while the trial team is busy taking care of other matters. There is no need to find a date on which the entire trial team, witnesses, parties and trial consultants can all be in the same place at the same time.

Third, since a survey is conducted without direct involvement of the trial team, subjects can be recruited for subsequent interviews or focus group study without the "contamination of advocacy." So, a survey can be a preliminary research step. leading to more targeted efforts.

Fourth, a survey can be tweaked as it is being conducted. Obviously, the analysis of any particular set of responses is limited by the number of subjects who were asked the same question. There is not harm, however, in figuring out halfway through that you really should have included a question about home foreclosures. The survey research company can include it for all subjects from that point forward. It is much harder to tweak a focus group or mock jury study in the middle.

While it is unlikely that an entire community will have decided on a verdict for your case in advance (although it could happen), a survey can help you identify pervasive attitudes among the people who will be comprising your jury. These attitudes will shape the lens through which jurors will be viewing your case. To stretch the metaphor a bit, you can't grind the right set of corrective lenses (your case presentation) until you know what prescription is needed.