We are developing a pedagogical agent called Steve -- the Soar Training Expert for Virtual Environments. Steve's role is to teach students how to perform procedural tasks, such as operating or repairing complex devices.
Our initial application for Steve is teaching students how to operate the gas turbine propulsion system aboard US Navy surface ships. These images show Steve working with a High Pressure Air Compressor (HPAC), which is part of the propulsion system. Using a head-mounted display, students enter the room containing the HPAC. The 3D model of the HPAC and its surroundings was created by Lockheed-Martin, as was the Vista Viewer software by which students view and interact with this virtual world. The image above was created by the Vista Viewer in desktop mode.
Students interact with the equipment by touching the various controls (e.g., valves and buttons) using a data glove. The behavior of the virtual objects is controlled by VIVIDS, an object-oriented simulator developed at the USC Behavioral Technology Laboratory.
Steve controls the virtual world by sending messages to VIVIDS. This image shows Steve pulling out the dipstick in order to check the oil level. Steve uses speech to provide a running commentary during demonstrations.
Steve perceives the environment by continually receiving messages from VRIDES describing changes in the virtual world (in terms of objects and attributes). Based on such perception, Steve can explain the state of the world during demonstrations.
In addition to demonstrating tasks, Steve can also monitor students performing tasks, providing help when it is needed. Above, the student has been asked to perform a functional check of the condensate drain monitor. Because Steve continuously monitors the state of the virtual world, and maintains a plan for completing the student's task, the student can always ask Steve what to do next.
Because Steve's recommendations are based on plans and goals, he can explain the rationale when asked.
Steve explains the rationale for his recommendations in terms of other relevant actions and goals. The student can ask followup questions about why these actions and goals are relevant until the rationale for Steve's initial recommendation becomes clear.
The student can also ask Steve to demonstrate the next step.
We have extended Steve to support training teams. Each Steve is assigned a role within the overall task. A Steve agent can monitor a human (or another agent) performing an assigned role.
We are experimenting with the use of nonverbal cues such as gaze to help coordinate the actions of agents within the team. This helps to convey a strong sense of team participation.