2/28/2003 General Aviation Requirements Requirement ID: 860 Special Category:  NONE Sponsor Organization:  AIR Sponsor POC: Colleen Donovan Keywords:  Alerting Systems, Performance (meas/imprv), Situation awareness (SA) Title: Developing And Validating Criteria for Constraining False & Nuisance Alerts For Cockpit Display Of Traffic Information Avionics Research Statement: The objective of this project is to develop and validate criteria for constraining false and nuisance alerts for cockpit displays of traffic information (CDTI), based on what is known about other alerting algorithms and human performance issues with alerting systems, trust, situation awareness and workload. Where objective criteria are not possible, subjective means may be recommended provided they are established to be reliable and valid measures. These criteria are to be included as minimum requirements in the RTCA Minumum Operational Performance Standards document or an FAA Technical Standard Order for CDTI. Both of these documents are used by avionics manufacturers to develop their systems, and FAA aircraft certification specialists who evaluate the systems. The project should be focused on developing these objective and subjective measures as minimum certification criteria, based on research and data, for approving the Free Flight technologies known as Cockpit Displays of Traffic Information (CDTI). The CDTIs may be either stand-alone units or as part of an integrated ADS-B CDTI/Traffic Collision Avoidance System (TCAS).1148 Background: It can be argued that the efforts to modernize the NAS and enhance both capacity and safety of the nation’s air transportation system are presently technology-driven and that human factors contributions to these efforts have fallen behind the demand. The reason for this situation is apparent: The task environments in which the personnel ultimately responsible for the safe and efficient functioning of the NAS (i.e., pilots, airline dispatchers, air traffic controllers and –managers) work have increased in complexity with increase in automation applications. Consequently, scientific investigation of the impact of new technologies has become increasingly difficult due to the escalating number of variables and their interactions in the present operational environments and the shift from overt performance (i.e., manual control) to predominantly covert behavior (i.e., supervisory control) of the operators. Several constructs that attempt to describe the complex and mostly covert behaviors have been introduced. The most significant of these is situation awareness (SA), but trust and workload associated with automation are of concern as well. The measurement of these constructs is problematic, yet of critical importance. May want to insert something here talking about the numerous problems with alerting systems and false alerts- impact on human performance- pilots turn them off, ignore them (boy who cried wolf) etc. This research will span a period of three years, with three distinct phases. Each phase may be considered individually for support, but the latter phases will depend on successful completion of the previous phases. Phase 1 and the first year efforts will focus on data gathering and understand how similar issues were solved with other flight deck alerting systems, such as TCAS, enhanced ground proximity warning systems (EGPWS) and wind shear alerts. This phase will include exhaustive review of the certification standards, requirements and guidelines related to false alerts and alerting criteria published in RTCA MOPS and TSOs for the systems mentioned above. The background and basis for the currently published standards should also be examined, as well as research literature pertaining to human performance issues with alerting systems associated with situation awareness, trust, and workload. The interactions of these constructs will also be examined, with an objective of identifying common underlying structures or mechanisms. This will include a review and evaluation of the Aviation Safety Reporting (ASRS) literature associated with TCAS problems, as well as other TCAS issues in order to uncover lessons learned. Special emphasis will be paid to the three “key references” listed at the end of the paper, as a potential means to develop certification standards to enable the evaluation of traffic collision alerting systems (e.g., CDTI ADS-B, TIS, and TCAS). These key reference papers propose the use of Signal Detection Theory (SDT) methodology as a means to evaluate alerting systems and separate the impact of various decision biases. SDT can be used to study the impact of changes to the decision threshold, and also the impact of changes to the a priori base rate events in the real world. The authors of these key references establish the importance not only of high hit rates and low false alarm rates, but also of the importance of high posterior probabilities of a true alarm. Additionally, they also propose a means to access the impact of these changes, despite the fact that only a handful of airplanes are equipped with ADS-B/CDTI systems, and thus it is difficult to determine the base rate information for these events, which is required to determine the posterior probabilities. Thus, one path of pursuit towards objective criteria to evaluating the CDTI alerting system is by attempting to apply the methodologies proposed and developing recommended certification criteria for the alerting systems hit rates, false alarm rates, and posterior probabilities. This methodology may prove effective in developing objective criteria for evaluating the appropriateness of an alerting system on the “trust/use/misuse/abuse” dimension. Additional methodologies and criteria would need to be developed to evaluate the situation awareness and workload dimensions. Output: Year 1 1. Documentation review: a) empirical human factors results relevant to alerting systems, available in the public domain (journal articles, conference proceedings, and government reports); b) certification standards, requirements and guidelines related to false alerts and alerting criteria published in RTCA MOPS and TSOs for cockpit alerting systems; c) comparison of the alerting algorithms of TCAS, CDTI, CA, and URET and examination of their congruence with pilots’ and controllers’ tasks and mental models; d) previous ASRS analyses on alerting system related incidents to determine if yet another ASRS analysis is warranted; e) literature on human factors certification for guidelines for development of certification criteria for CDTIs; f) identification of other data sources (e.g., from demonstrations and simulations or from operational environments) that would allow for further examination of relevant human factors issues outside of a laboratory. 2) Examination of the roles of cockpit alerting systems. This subtask will examine the roles of a number of automatic alerting systems (GPWS, TCAS, wind shear alert) and the impact of these on the respective certification criteria of the alerting systems. 3) Development of measures and criteria for collision avoidance system evaluation. This subtask involves a comprehensive evaluation of available measures of machine, human, and human-machine system performance as they pertain to collision avoidance systems, identification of primary and secondary measures, and evaluation of empirical support for the latter. We will also examine possible sources and justification for criteria for the measures. 4) Develop designs and protocols for experiments. Based on findings from the literature review, we will develop experimental designs and protocols aimed at investigation of the most critical issues relevant to human factors certification of CDTIs and to address possible controversies in the alerting system literature. 5) Conduct Experiment 1. The goal of this component of the project is be to develop a cognitive model of the features of unaided conflict prediction, that is, pilot prediction made without the aid of intelligent automation. Regulatory Link: None