Effects of Reliable and Unreliable Automation on Subjective Measures of Mental Workload, Situation Awareness, Trust and Confidence in a Dynamic Flight Task

Eight military pilots completed a study of human-interaction with reliable and unreliable automation in a complex air-to-ground search and destroy mission. Each mission consisted of four distinct stages. Automation was available in three of the four stages to aid the pilot in completing the required tasks. Automation reliability level was manipulated and combined factorially with two levels of workload, resulting in sixteen mission scenarios. Subjective measures of mental workload, situation awareness, trust, confidence, and judgments of the automation reliability were recorded after each mission. An analysis of the subjective measure ratings by mission workload level and the reliability of the first stage automation was performed. The results indicated that pilot trust in the automation was higher when the workload level was low and that there was a significant difference in confidence ratings between high and low workload levels, but only when the automation was unreliable. Design implications are discussed.