Recently blogger Julian Sanchez coined the phrase “one-way hash argument” to describe how easy it is to throw up FUD in what passes for public debates these days and how much more difficult it is to combat that FUD with actual facts. Good points to be sure, but I think there’s a solution to this problem. It’s difficult to combat FUD using standard journalistic/bloggolistic techniques, but that doesn’t mean that the situation is hopeless. One of the goals of this blog is to showcase an alternative method of displaying expert facts and opinions without boring the audience, or requiring the audience become an expert in a field. The best example I’ve seen of such an alternative approach is in Ed Tufte’s book Visual Explanations where he examines the space shuttle Challenger disaster. With permission I’ve reproduced part of his presentation below.
On January 28, 1986, the space shuttle Challenger exploded and seven astronauts died because two rubber O-rings leaked. These rings had lost their resiliency because the shuttle was launched on a very cold day. Ambient temperatures were in the low 30s and the O-rings themselves were much colder, less than 20°F.
One day before the flight, the predicted temperature for the launch was 26° to 29°. Concerned that the rings would not seal at such a cold temperature, the engineers who designed the rocket opposed launching Challenger the next day. Their misgivings derived from several sources: a history of O-ring damage during previous cool-weather
launches of the shuttle, the physics of resiliency (which declines exponentially with cooling), and experimental data. Presented in 13 charts, this evidence was faxed to NASA, the government agency responsible for the flight. A high-level NASA official responded that he was “appalled” by the recommendation not to launch and indicated that the rocket-maker, Morton Thiokol, should reconsider, even though this was Thiokol’s only no-launch recommendation in 12 years. Other NASA officials pointed out serious weaknesses in the charts. Reassessing the situation after these skeptical responses, the Thiokol managers changed their minds and decided that they now favored launching the next day.They said the evidence presented by the engineers was inconclusive, that cool temperatures were not linked to O-ring problems.
Thus the exact cause of the accident was intensely debated during the evening before the launch. That is, for hours, the rocket engineers and managers considered the question” Will the rubber O-rings fail catastrophically tomorrow because of the cold weather?” These discussions concluded at midnight with the decision to go ahead. That morning, the Challenger blew up 73 seconds after its rockets were ignited.
Since the engineers could not convince their management to scrub the launch it seems reasonable to conclude that the science and data were so esoteric as to make it impossible to come to a conclusive no-go decision before the launch. This is rocket science after all. But is this what happened? Was the science so esoteric as to be inscrutable to managers and non-scientists? The science is esoteric, but as we will see in a moment that doesn’t mean that a successful argument couldn’t have been made to management convincing them to scrub the launch.
Before we proceed, I should point out there is a big difference between the setting that the engineers found themselves in and the sort of setting Julian Sanchez imagined in his post. Sanchez posited that the lay public is reasonably open minded but easily swayed by FUD especially when the counter arguments are detailed and complicated. The NASA engineers, on the other hand faced a very skeptical set of managers who, due to various political reasons, were strongly against further delaying the already delayed shuttle flight. With such a bias from the beginning, it is entirely possible that no argument could have convinced the managers to scrub the launch. However, given a more open minded audience and a more compelling argument, history could have been different.
As mentioned above the engineers hurriedly created a slide show to present to the NASA managers. Tufte reproduces many of the charts from that slide show in his book and they are also available online. One thing is clear from these slides, the engineers were trying to give the managers a crash course in “rocket science”. The slides discussed the theory of how the booster rocket seals should work at various temperatures and presented information on how actual seals had failed under a variety of real-world circumstances, both from actual launches and from test firings. This sort of presentation is exactly the complicated, boring, experts only, losing side of the asymmetric one-way hash argument Sanchez discusses. Below are some slides to showcase this point.
However, even here the managers used FUD to counter these points. They asked if there were any successful launches in the 50° range. There were. They also pointed out there were launches with damage as warm as 75 degrees. These 2 points are examples of what Tufte calls “dueling anecdotes”. We could all train ourselves to watch out for this sort of fallacy or, as Tufte prefers, we could force one or the other arguer to present all the data. In this case the outlier nature of the single cited anecdote becomes clear for all to see. (Whenever I think of this approach to dueling anecdotes a poker image pops into my mind. One player says “I raise you one anecdote” to which the other poker player replies “I see your one anecdote. I’m going ‘all-in’ with every bit of data I have”.)
Tufte took the data the NASA engineers presented and put it into a single chart. But before doing that he boiled the issue down to the simplest question. Specifically how are cold temperatures and O-Ring damage correlated? He showed this in a chart that plotted all the shuttle launches with temperature at launch on one axis and a “damage index” on the other axis. This chart is reproduced below.
There were a few instances of damage occurring at warm temperatures, but every launch colder than 65° had some damage. Furthermore the chart shows that damage increased as the temperature got colder. The coldest launch before Challenger was at 52° and Challenger was launched when the temperature was in the mid to high 20s!
While not strictly accurate to do so, the eye wants to connect the dots for the launches colder than 65° and furthermore extrapolate to the 26° to 29° temperature range forecast for the launch time and arrive at a damage index of more than double the previous maximum damage.
This is a simple yet powerful diagram. The compelling points about it are:
- every launch colder than 65° had some damage
- damage got worse as it got colder
- expected launch temperatures were significantly colder than any other launch
- all the data is shown
- the gap between the previous coldest launch of 52 degrees and this launch succinctly captures gap in knowledge and the severity of the cold on the launch date
Notice that we neatly sidestep scientific issues of “shore hardness” and technical identifiers like SRM-15. All that we care about is shown above. Temperature at launch and (an albeit subjective) measure of damage. Additionally note that the phrase “A temperature lower than current database” gives no idea how much lower or what the implication of that lower temperature implies. However representing this temperature graphically gives a much more useful and comprehensive view.
This technique of arguing points through graphical data is a solution to the one-way hash argument, at least for arguments that are amenable to this sort of treatment. What more could one want? Being graphical it’s simple, comprehensive and, if done right, interesting. On the negative side it is graphical and for that segment of the public who is strongly verbally aligned (or at least non-visually aligned) such a presentation may not be that compelling. But imagine for a moment 2 experts on opposite sides of an argument debating the merits of an argument framed in such a way. These experts would be forced to argue the merits and interpretation of the data and not be detoured into off-topic FUD topics such as whether the opposing expert had a financial stake in the conclusion.
To be sure, not all issues are amenable to this sort of analysis. Many issues aren’t quantifiable at all. Some issues are purely judgment calls, situations where reasonable people can disagree; there is no single right or best answer. And some issues, even if they are quantifiable don’t show as clear cut an answer as the Challenger example. But there are many more issues that can be examined in this light; issues that are not being so examined. Finding and examining these issues is one of the goals of this blog.
Ed Tufte’s treatment of the Challenger disaster in his book Visual Explanations pages 38 – 53.
Background information came from the testimony given in the “Report of the Presidential Commission on the Space Shuttle Challenger Accident”.
The slides came from this same testimony. A direct link to the slides is available here.