Effectiveness Analysis for Aviation-Safety Measures in the Absence of Actual Data

Sometimes, it seems likely that a technological advance will improve aviation safety, but it is not clear by how much. Uncertainty might be especially great when the innovation strengthens the capabilities of an existing system rather than creates entirely new capabilities. For example, newly-available “name tags” can enhance ground surveillance systems by displaying for air traffic controllers the identity of each plane on the runway. But, when two such planes are in danger of colliding, how much benefit arises because the controller knows that they are (say) United #306 and American #424? Focusing on an example about added functionality for a ground collision avoidance system, we describe a method for generating probabilistic estimates of benefits for a technology not yet deployed. The effort starts with a group activity involving technology experts and system users, in which relevant past events are studied and discussed one at a time. Then key participants are asked individually to assess the probability that the new technology would have “saved the day” in each situation, while other available measures would not. A major challenge is to avoid confusion and anxiety for participants, while preventing some individuals from exerting excess influence on group decisions (as can arise in consensus-based procedures). We present effectiveness estimates for added functionality, as well as approximate margins of error for these estimates. We also discuss the limitations of the approach used, and alternative methods of making effectiveness estimates.