Cleared for Landing

Using Virtual Reality to Train Air Traffic Controllers

By Brian Fisher

J. Brian Fisher is a senior research engineer in the Advanced Simulation Technologies Section of SwRI's Training Systems and Simulators Division. As project manager of the recently completed Air Traffic Control Virtual Reality (ATCVR) project for the U.S. Air Force, Fisher played a key role in the requirements analysis, design, development and delivery of the ATCVR system to the client. Since joining the Institute in 1996, Fisher has applied advanced simulation techniques to a variety of virtual reality projects.

Certification of a military air traffic controller (ATC) follows a long and intensive training process. The U.S. Air Force (USAF) is evaluating simulation technologies, including virtual reality, to accelerate and improve the training process and relieve a shortage of certified controllers. Virtual reality-based training systems provide a computer-mediated experience in which ATC trainees can interact realistically with virtual objects in a synthetic environment.

Although the USAF has used simulation extensively in other areas, it has not used it to provide site-, or in this case, base-specific training for ATCs. Instead, new controllers typically are trained to work at their assigned Air Force base with static displays, chalkboard scenarios and live traffic. Static displays provide little realism, and working with live traffic is risky and limits training opportunities. Southwest Research Institute (SwRI) developed an Air Traffic Control Virtual Reality (ATCVR) trainer to provide a realistic training environment that can accelerate and improve ATC certification.

Technical approach

In December 1998, the Air Education Training Command (AETC) contracted with a supplier for a prototype virtual reality training system for controllers at Randolph Air Force Base in San Antonio. The contractor used proprietary hardware and software that are no longer available, making the system difficult to support and impossible to expand. The AETC contracted SwRI to re-engineer the system, using only commercial off-the-shelf components. The six-month effort consisted primarily of porting the system to a new hardware configuration and developing a new software architecture.

SwRI also worked closely with the AETC to enhance the system so that it could be used at multiple sites simply by supplying a base-specific data set. This data set, consisting of three-dimensional models of aircraft and airfields, is created by SwRI engineers using a combination of digital terrain data, satellite imagery, engineering drawings and on-base photographs. Each airfield model includes detailed representations of runways and taxiways with controllable lights, an interior model of the tower and base landmarks. To reduce costs and expand performance, controllers selected only the landmarks, such as hangars, buildings, tree lines and water and radio towers, that they use as points of reference when directing aircraft to land or depart from the air base.

During a scenario, the trainee receives a simulated view from the tower that includes virtual airfield and surrounding terrain. The HMD and tracking system provide trainees with complete control over their view from the tower.

An instructor may view a scenario from anywhere in the environment, including the trainee's view, from overhead, or from any of the aircraft. Here, an instructor chooses a view from a simulated aircraft as it approaches the runway at Vance Air Force Base, Oklahoma.

System description

The ATCVR system consists of two primary components, the Instructor Operator Station and the Trainee Station. The Instructor Operator Station allows an instructor to define, control and monitor training scenarios and to communicate with the trainee. The instructor can assume the role of pilot of the simulated aircraft and also respond to the trainee's commands. The Instructor Operator Station consists of a host PC that defines and controls the scenarios, a PC-based image generator that displays the instructor's view of the scenario and an audio system.

The Trainee Station consists of a Head-Mounted Display (HMD), a PC-based image generator and an inertial-acoustic motion-tracking system used to track the orientation and movement of the trainee's head. During a scenario, the tracking system feeds data based on the orientation of the trainee's head to the image generator. As the trainee turns his or her head, the view displayed in the HMD is updated to provide a 360-degree view from the simulated tower.

System features

The ATCVR system provides controls to allow the instructor to create predefined flight plans that simulate traffic patterns specific to the base. A point-and-click interface allows an instructor to define a flight plan by placing a series of waypoints on a topographical map of the base and surrounding area. A prerecorded radio call may be assigned to any point along the flight path. During a scenario, the software will automatically play the radio call, thereby reducing the load on the instructor. Flight plans may be assigned to any of 15 aircraft types, ranging from small trainers to heavy-lift cargo aircraft. Additional controls allow the instructor to define the scenario environment including time of day, cloud ceiling and degree of visibility from the tower.

After a scenario is started, an instructor controls aircraft in the scenario as well as the environment through a Windows^TM Graphical User Interface (GUI). The GUI allows the instructor to modify the flight plan or to take manual control of an aircraft in response to trainee commands. In addition, the instructor may select from a set of views to monitor the scenario from a number of perspectives. These perspectives include an instructor-controlled view that may be placed anywhere in the environment, the trainee's view, an overhead view or even a view from any of the aircraft in the scenario.

During a training session, the trainee is provided with a simulated view from the tower that includes the virtual airfield and surrounding terrain. Through the HMD and tracking system, trainees can control the view they see from the tower, including a simulated binocular view. SwRI modified the HMD to allow the trainee to activate the binoculars in a natural way by pressing buttons on the HMD in the same way that he would hold actual binoculars. This action provides a natural interface to train controllers to verify aircraft have landing gear extended at the appropriate time during an approach.

What's next?

SwRI has delivered four ATCVR systems to selected USAF training bases. The systems are being evaluated as part of a nine-month training transfer study. Preliminary feedback from trainers has been positive, and enhancements are already being considered. One improvement is adding voice recognition and speech synthesis technologies, which could reduce the instructor's workload. A second would be to create dynamic displays, such as radar in the simulated tower reflecting the events occurring during a scenario. These dynamic displays could be accomplished through computer-generated displays or actual displays and "mixed reality," using an SwRI-developed technology that allows users to see both real and virtual objects. The ATCVR system now provides training for a single position, that of the local controller. Future systems might incorporate other tower positions into a networked simulation to allow complete training.

Simulator benefits

The ATCVR system provides a low-cost, compact solution for providing base-specific training for air traffic controllers. With off-the-shelf PCs, the cost is kept to a minimum while still meeting training requirements. The system also leaves room for future expansion. SwRI engineers designed the system to fit in a 10- by 10- by 10-foot space that needs no special modifications, allowing flexibility in installing the system.

By being immersed in a virtual environment that closely matches an actual tower, the trainee becomes familiar with local tower procedures long before working with live aircraft traffic. This simulation allows the trainee to practice and gain confidence without risk to aircraft personnel. Because the ATCVR system is programmable, an instructor can present a wide range of situations, including basic operations as well as emergency procedures that might not otherwise be covered as thoroughly if training were limited to live traffic. In addition, because the system can record a trainee's actions, the instructor can review the trainee's performance more quantitatively to track progress better.

Comments about this article? Contact Fisher at (210) 522-3762 or bfisher@swri.org.

Published in the Summer 2001 issue of Technology Today®, published by Southwest Research Institute. For more information, contact Maria Martinez.

Technics Summer 2001 Technology Today
SwRI Publications SwRI Home