The last couple of years has seen a proliferation of online mock trial tools. The creators of these products vary with respect to how hard they push their products as equal or superior to traditional techniques for conducting jury research. In this issue of the Jury Box, I will offer my own view on the utility of online mock jury tools. Along the way, I will outline what I believe to be the advantages and disadvantages of conducting jury research online. A sizable chunk of this review will be devoted to evaluating the sources of cost savings associated with online mock trials. The lowest cost option is not always the best option.
Two types of businesses have set up online mock trial programs. Several have been developed by entrepreneurs (or market research firms), hoping to market their products directly to lawyers. Such a company typically holds up its product as a low cost alternative to hiring a jury consultant to conduct a live mock trial. Perhaps motivated by perceived competition, several trial consulting firms have developed their own mock trial protocols.
An explanation of how these programs typically work reveals that the term “online mock trial” is really a misnomer. Here’s how they work:
Nuts and Bolts
The attorney reduces the case presentation to a series of files that can be uploaded to a website. Most sites allow the uploading of text, audio, images and video in fairly standard file formats. Typically, a premium is charged for loading high bandwidth material, like audio or video. Some are better than others at connecting exhibits to the corresponding presentation or testimony. While the online service might provide a basic pre-study questionnaire and post-study response form, it is usually the responsibility of the client to provide a list of case-specific questions to be answered by each respondent.
Each online site promises to recruit subjects to participate in your study. They use a variety of recruitment techniques, which is something to which you should pay close attention. I will return to this below. The more subjects you want to use, the more you will have to pay for both recruitment and participation. Many sites allow the attorney to set the participation inducement, understanding, of course, that recruitment will be trickier the less is offered in payment.
Once the materials are loaded and subjects have been recruited, the participants are given a password to “view” the case and provide feedback. Most sites, especially the low cost ones, allow respondents to log on and go through the case whenever they want, from the comfort of their own homes (or offices!). Some sites offer you the option of having all the subjects review the case at the same time and “deliberate” in the form of text chat. This lack of face-to-face interaction is one major casualty of conducting jury research in this way.
Since the respondents are all answering questions online, it is fairly straightforward to collect and collate the data. Reports are generated automatically from collected data and the client can download them at her leisure. Only the programs run by the trial consulting companies offer to have a jury expert interpret the data and write a corresponding report.
Running additional trials, with or without tweaking of the presentation materials, is straightforward since everything has already been uploaded.
So, what are you buying?
As you can tell from this brief description, there really is no way to simulate an entire trial using this technology. It lacks much of the verisimilitude that makes a true mock trial so valuable. Rather, the technology is best employed for testing case themes or evaluating juror reaction to particular arguments, testimony or evidence. As such, these companies are really offering online focus group research, not online mock trials. Perhaps this is a semantic point, but you should be aware in advance what these programs can and can’t provide in terms of jury research.
As I move into an evaluation of online mock trials, let me be clear about one thing. Most of these programs are very well crafted. The designers have been quite thoughtful and thorough and most of the programming is well done.
Any weaknesses are not due to lack of planning or effort. Rather, they all suffer from a fundamental shortcoming in the enterprise. How does one simulate a highly personal, interactive and collaborative process with an anonymous, passive and solitary methodology? The answer, of course, is that you can’t. These online programs can help lawyers learn about juror reaction to some aspects of trial, but they can’t replicate jury reaction very well at all. This fundamental distinction permeates much of what I lay out below.
Is it really such a bargain?
One of the biggest selling points for these online jury research programs is the profound cost savings. Many tell you that you can conduct your mock trial at one-tenth the cost of a traditional study. Let’s investigate the sources of the cost savings and what sacrifices accompany them.
The online jury research company charges you nothing (or very little) to load your presentation, pre-study questionnaire, exhibits and verdict forms to its website. These materials, however, do not materialize out of thin air. When a trial consultant quotes you a price for a focus group, it typically includes provisions for the consultant to assemble all these materials and help you put them in the proper form for presentation. Someone has to do this preparatory work regardless of whether the study is live or online, and it’s going to cost your client some money, paid either to you or to the consultant you hire to help you out. If you choose to prepare all the materials yourself, you lose all the expertise that a trial consultant can provide. The more you skimp on this preparation, the less reliable will be your results. As with most things, you only get out of it what you're willing to put into it.
Online companies typically charge less for participant recruitment and pay subjects less than those who recruit for live studies. The cost savings comes from two sources. First, it is probably easier to convince someone to spend a couple of hours on their laptop than to drive across town to participate in person. This is especially true for non-collaborative online studies, where each subject can log on at her leisure. Second, given the convenience, one doesn’t have to pay online participants as large an inducement.
The vast majority of what a focus group facility charges a consultant is dedicated to subject recruitment and compensation. Please be aware that not all subjects are created equal. Reputable recruiters use random dialing techniques or large proprietary subject databases to insure that the sample is representative of the venue requested. If you go on Craig’s List, MySpace, or dozens of ad posting sites, you can find hundreds of ads seeking online mock jurors. The online mock trial companies typically save time and money by recruiting in this way. Respondents to such ads hardly comprise a random sample of the community, and you run the risk of drawing a jury full of professional mock jurors. As the Econ folks say, “You get what you pay for.”
While you may be able to hire online participants more cheaply, you are definitely buying an inferior product. There is no way to monitor how attentively each respondent is reviewing your case. In some circumstances, it may be impossible to tell whether the subject reads your case, at all. You can’t examine the expression on each subject’s face as she learns about your case. You can’t hear the audible gasp when a particularly damning piece of testimony comes out. Finally, and most importantly, you cannot watch the jurors deliberate – because they don’t. For certain kinds of questions – most notably damages – a jury’s evaluation is much more than the sum of those of its jurors. Deliberation is an organic, collective experience. It is expensive to recruit for a real focus group precisely for the reasons that make it most valuable: It’s hard to get a sizable representative group of people in the same place at the same time so that you can watch them evaluate and deliberate about your case.
When your online jury study has been completed, you get a detailed report of exactly how each subject responded to each question asked. Most sites even draw fancy pie charts and allow you to view the data broken down by age, gender, verdict choice and the like. But what are you supposed to do with all this raw information? For a trial consultant, actually getting the study completed has only started the process. We really earn our money by interpreting the results. We help you figure out what the results really mean and how to implement what you’ve learned when choosing litigation strategies. I suspect that if you hire a full service trial consulting company to conduct an online focus group, using its proprietary technology, it will represent only a modest cost savings over a simple live study.
The Verdict…
So, what’s the bottom line? Well, online jury research can be an economical way to learn how more-or-less ordinary people react to particular aspects of your case. Remember, however, that the value of hiring professional trial consultants and recruiters lies in the expertise they bring to the table. All of this is lost if you choose to go it alone, either online or otherwise. These online services offer a potentially useful tool, but make sure to hire someone who really knows how to use it.
Shop before you buy. Ask explicitly about how the service recruits subjects, how they monitor attentiveness, how subjects are directed to review exhibits and what form of deliberation is available, if any. Run the tutorials on several sites to test for ease of use.
As the technology advances, however, we will be in a better position to simulate a real focus group online. Every MacIntosh already ships with a built-in camera and many PCs are similarly equipped. If teenagers can talk trash while playing World of Warcraft online, we can’t be far from being able to watch and talk to online respondents as they review a case and deliberate among themselves. The technology for such videoconferenced focus groups already exists, but requiring respondents to own compatible technology would further skew the distribution of eligible participants.
