Project Topics
Get Complete Project Material File(s) Now! »
RESULTS AND DISCUSSION
Summary data obtained through the experiment, aiming to answer the study’s research questions, can be found in the graphs below. Values seen above each bar represent the group averages, and values displayed at the bottom of each bar (i.e., n=) represent the sample size for each group. This sample size does not represent the number of participants who answered the question but rather the number of repeated measures of participant responses for that question. The surprise event represented a different condition than what was experienced by participants during the other session scenarios and trials, therefore data been removed from reporting, unless otherwise stated.
“No HMI” Conditions
As described in the Methods, the preliminary results of the “with knowledge” condition indicated that levels of comfort, trust, safety, and situational awareness were much higher than originally anticipated. The research team was concerned that such elevated measures in this condition would limit the response opportunity for the treatment conditions. The concern led to a detailed review of the experimental protocol and creation of a second “no HMI” condition, which collected data under a slightly different protocol, the results of which can be seen in Figure 19.
During the “with knowledge” condition, researchers revealed more information about the vehicle limitations and testing environment safety controls. Likely because of these extra forewarnings, participants reported significantly higher, perhaps artificially elevated, levels of perceived comfort (p=0.03), trust (p=0.003), and safety (p=0.02). Situational awareness was not affected by the differences in conditions.
It is interesting to consider the implications that a few, relatively minor, changes in the information provided to participants exhibited a significant effect on the metrics. While not a primary investigation of this study, the results indicate that appropriate training on HAV may have substantive impacts on the rider’s impression, even when systems are imperfect.
A Likert scale question about whether additional information would improve levels of comfort, trust, safety, and situational awareness was also presented. From this question, participants’ desire for information when none was supplied could be measured. Summary data from the Likert scale question can be seen in Figure 20.
Researchers expected participants to want more information about the vehicle’s “intentions” during these conditions, particularly in the “without knowledge” condition, as no additional information was provided. However, when participants were asked whether more information would improve their levels of comfort, trust, safety, and situational awareness, a majority indicated neutral-slight agreement. Even though significant differences were seen between the two “no HMI” conditions in reference to comfort, safety, and trust (Figure 19), participant responses between them appear equal (Figure 20).
Considering that participants had no previous experience with HAVs, they may not have understood what types of information could have been presented during the maneuver, resulting in low desire for more information. Another likely explanation for the neutral desire for additional information was the participants’ levels of comfort, trust, safety, and situational awareness during these “no HMI” conditions. These reported metrics were already relatively high to begin with, therefore providing participants with additional vehicle information may have only caused marginal improvement. This effect is prominent in the “with knowledge” condition, which resulted in the decision to consider the “without knowledge” condition as the “true” control condition. Thus, the “with knowledge” condition has been omitted from the subsequent data reporting and analysis, unless otherwise specified. In subsequent graphs, “No HMI” indicates the “without knowledge” control condition.
HMI Condition and Perceived Comfort, Safety, Trust, and Situational Awareness
To answer one of the primary research questions of the study, HMI conditions were compared to participants’ reported levels of comfort, trust, safety, and situational awareness during experimental sessions, shown in Figure 21.
After examining both the summary data and statistical analysis outputs, researchers determined that participants who experienced the auditory HMI condition reported significantly higher levels of comfort (p=0.004), trust (p=0.002), and safety (p=0.0005) than those who experienced the visual-only condition. Unexpectedly, participants who experienced the visual-only HMI condition reported the lowest levels of the metrics, even when compared to the “no HMI” condition. Researchers initially hypothesized that the auditory HMI would produce the lowest levels of perceived comfort, trust, and safety, and would be the least effective at communicating vehicle “intentions”, since it provided users with the least detailed vehicle information. As additional, more detailed information was supplied to the user through the visual and mixed-modal HMIs, these metrics were expected to increase.
Perceived situational awareness was not significantly affected by any HMI condition, however, the reader should interpret this finding with caution. Prior to the session, situational awareness was not defined for participants and no strategies for how to quantify it were implemented into the experimental design. Therefore, participants may have had different perspectives of what situational awareness represented when asked about it during post-surveys.
The scenarios used for this experiment were variable and modeled on real-world driving situations. Since there was variability from scenario to scenario (e.g., different vehicle maneuvers, additional vehicles on the roadway, or additional research personnel present), whether these differences affected participants’ impressions was of interest. Reported levels of comfort, trust, safety, and situational awareness can be seen in Figure 22. In subsequent sections, certain scenario names have been simplified. “Ped Xing” represents the Pedestrian Crossing scenario, “Work Zone” represents the Following a Lead Vehicle/Work Zone scenario, “Turns” represents the Left Turns scenario, and “Pick Up” represents the Passenger Pick Up scenario.
Only the surprise event (e.g., undetected Pedestrian Crossing scenario) had a significant effect on these metrics (pcomfort=6.42e-09, psafety=1.96e-08, ptrust=1.15e-15). A slight decline in perceived comfort, trust, and safety was seen during the Baseline, Following a Lead Vehicle/Work Zone, and Passenger Pick Up scenarios. This slight decline may imply that factors such as the presence of obstacles or familiarity with the vehicle, could have altered participants’ feelings of comfort, safety, and trust. The Baseline scenario was always the first scenario participants experienced, so participants may have been acclimating to the vehicle and testing environment. In addition, no HMI systems were active during this scenario, so no vehicle information was being communicated to the passengers. For Following Lead Vehicle/Work Zone and Passenger Pick Up scenarios, participants may have been a bit more uneasy because there were additional vehicles and obstacles present on the road. These additional environmental factors increased the variability and complexity of the driving landscape and may have subsequently increased riders’ perceptions of risk to personnel or other vehicles.
To determine whether participants would become acclimated to the vehicle dynamics and maneuvers over the duration of the session, trial number was examined. Hypothetically, researchers believed that due to vehicle acclimation there would be a steady increase in perceived levels of comfort, trust, safety, and situational awareness across Trials #1-5 (i.e., as the session progressed), with a sharp decline in Trial #6 due to the surprise event. Summary data comparing trial to participant reported metrics can be seen in Figure 23.
As expected, there was a slight trend, though not statistically significant, of comfort, safety, and trust increasing as the session progressed. Only Trial #6 (i.e. the surprise event) created a significant decrease in perceived comfort (p=1.28e-08), trust (p=6.31e-15), and safety (p=2.59e-08). As people become more familiar with a piece of technology, they can become acclimated to it. This acclimation can either be positive, as users become more comfortable or trusting in the technology, leading to wider acceptance, or it can be a negative, as users become over-reliant on the technology and misuse it (Parasuraman & Riley, 1997; Cunningham & Regan, 2015; Sauer, Chavaillaz, & Wastell, 2016).
To determine levels of over-reliance, participants were supplied with a stop button that they were instructed to press if they felt uncomfortable or if the vehicle seemed as if it was malfunctioning. Their desire to press the stop button when experiencing different vehicle scenarios, session trials, or HMI conditions, derived from post-trial questionnaires, is shown in Figure 24. Not only was the stop button a metric of comfort and safety, as participants would presumably only feel the need to press it if they felt at risk, but it was also a metric of situational awareness, as hopefully they would notice the vehicle and HMI malfunctioning during the surprise event and press it.
Although participants reported an elevated desire to press the stop button while they experienced the surprise event (p=6.60e-07), Trial #6 (p=7.29e-07), and the visual HMI condition (p=0.05), across all sessions, only one participant out of the thirty-nine pressed the stop button during the surprise event to indicate the vehicle was not behaving appropriately. Even though the participants were only in the vehicle for a relatively short time, it was enough for them to seemingly become comfortable with the test vehicle and not recognize a system malfunction, even with sufficient time to press the stop button before impact. This lack of action could indicate an absence of situational awareness or an over-reliance of the system (Abe, Itor, & Tanaka, 2002; Noble, Dingus, Doerzaph, 2016). Another explanation for the lack of stop button depressions could be that although the balloon pedestrian in the roadway was in danger, the participants themselves were not. The participants may have thought the stop button was only supposed to be used in an actual emergency, instead of a simulated one.
With new technology, vehicle malfunctions and miscommunications will be inevitable. The roadway is a highly variable place both due to other drivers and environmental conditions, such as weather. For example, a large proportion of vehicle sensors cannot operate in adverse weather conditions such as rain or snow. Also with new technology, as seen in countless surveys about user trust and automation, these levels of user trust and comfort are delicate and can be easily reduced by both false negatives (e.g., a system does not detect a hazard) and false positives (e.g., a system detects a hazard that is not present), especially in familiar, daily driving scenarios where users are expecting vehicles to perform perfectly (Dzindolet, Peterson, Pomranky, Pierce, & Beck, 2003; Johnson, Sanchez, Fisk, & Rogers, 2004; Madhavan, Wiegmann, & Lacson, 2006). When looking at the surprise event where a false negative was introduced, participants’ levels of comfort, trust, and safety were significantly degraded and their desire to press the stop button was elevated. Both of these findings may support concerns that a single undesirable experience within a HAV can rapidly erode user impressions of the technology. In addition, the results support the claims that HAVs should not rely on any human intervention during critical system failures.
HMI and System Transparency
Previous studies have suggested that information presented in combined modalities are able to leverage multiple different sensory channels to improve user processing and understanding (Mousavi, Low, & Sweller, 1995; Leahy & Sweller, 2011) therefore reducing reaction time, if intervention is needed (Blanco, et al., 2016). Participants were asked to rate the vehicle’s effectiveness at communicating its “intentions” (i.e., where it was planning traveling, when it would start/stop) and its perceptions of the roadway (i.e., hazard detection), as seen in Figure 25. This question was important as it determined whether the HMIs aided in increasing vehicle system transparency, and if they did, which system was most effective.
The mixed-modal and audio-only HMI conditions had the most success in communicating vehicle “intentions” to the user. When communicating information about obstacles in the roadway and the vehicle travel path, the audio-only HMI was significantly more effective than the “no HMI” (pobstacles=2.30e-06, ppath=0.0001) and visual HMI (pobstacles=0.04, ppath=0.04) conditions. The audio only HMI was also significantly better at communicating the vehicle’s intention to stop compared to the “no HMI” condition (p=5.02e-05). This is an interesting finding as the audio HMI was not expected to be the most effective at communicating “intentions”, especially for vehicle path, where there was no showcasing of the future vehicle trajectory. This may indicate that occupants find little value in understanding the exact intended path of the vehicle but rather base their impressions on higher-level information such as an intent to simply start moving.
The mixed-modal HMI condition was more effective at communicating the presence of obstacles and planned vehicle path compared to the “no HMI” condition (pobstacles=2.30e-06, ppath=0.0001). It was also most effective at communicating the vehicle’s intention to stop compared to both the “no HMI” (p=6.30e-07) and visual-only (p=0.0009) HMI conditions.
Since the mixed-modal HMI had the ability to leverage multiple different sensory channels, it was theorized to be the most effective at communicating vehicle-information. These findings can be seen in the study, where the mixed-modal condition was most effective at communicating the vehicle’s intention to stop. This version of the HMI system encompassed both auditory cues, that communicated the vehicle’s intentions to stop, and visual stimuli, that communicated the actual stopping location. This combination of modalities gave participants additional details about the driving landscape and thus may have increased their understanding about the driving system.
Unexpectedly, the auditory HMI proved to be the best at communicating vehicle path, even though no specific tones were used to represent this information. The only tones that indicated vehicle movement were the acceleration and deceleration tones, which did not communicate specific information about vehicle maneuvers, such as turning. Most likely users’ overall higher preference for the auditory HMI condition artificially elevated their responses to whether the HMI clearly communicated obstacle detection, intention to stop, and planned vehicle path.
Although the “no HMI” condition received the lowest scores across all conditions, participants still rated it as “neutral” for communicating the presence of obstacles, intention to stop, and vehicle path. This HMI provided no vehicle information and participants were expected to provide ratings at the low-end of the scale. A possible explanation for the neutral ratings relate to the cues already built into a vehicle, such as the wheel turning or the engine revving. These unavoidable feedback devices could have unintentionally communicated vehicle path, via the turning of the wheel, or acceleration/deceleration, making them somewhat comparable to the other conditions. Such stimulus may be diminished in HAVs, which remove the steering wheel and/or leverage quieter drivetrains, such as electrification.
TABLE OF CONTENTS
INTRODUCTION
Automated Vehicles
HAVs and Rideshare
Trust and Acceptance of HAVs
Human-Machine Interface (HMI)
Previous Studies and Current Gaps in the Knowledge
Research Objective
Outline of Thesis
METHODS
Testing Environment
Test Vehicle
HMI Conditions
Study Participants
Experimental Scenarios
Experimental Procedure
Data Analysis Methods
RESULTS AND DISCUSSION
“No HMI” Conditions
HMI Condition and Perceived Comfort, Safety, Trust, and Situational Awareness
HMI and System Transparency
Factors Influencing Users’ Acceptance of HAV
Age and Gender vs. Comfort, Safety, Trust, and Situational Awareness
Linear Mixed-Effect Model
Participant-Specific Responses
Content Analysis
Surprise Event
CONCLUSIONS AND RECOMMENDATIONS
Limitations
Future Work
REFERENCES
GET THE COMPLETE PROJECT
Assessing Alternate Approaches for Conveying Automated Vehicle “Intentions”
