NAVAL POSTGRADUATE SCHOOL
Monterey, California
THESIS
EVALUATING DEMOGRAPHIC ITEM RELATIONSHIPS
WITH SURVEY RESPONSES ON THE MAINTENANCE
CLIMATE ASSESSMENT SURVEY (MCAS)
Bruce R. Stanley, Jr.
June 2000
Thesis Advisor:
Thesis Co-Advisor:
Second Reader:
John K. Schmidt
Robert R. Read
Lyn R. Whitaker
Approved for public release; distribution is unlimited.
20000811 033
REPORT DOCUMENTATION PAGE
Form Approved
OMB No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching
existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden
estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for
Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget,
Paperwork Reduction Project (0704-0188) Washington DC 20503.
1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
June 2000 Master’s Thesis
4. TITLE AND SUBTITLE
Evaluating Demographic Item Relationships with Survey Responses on the
Maintenance Climate Assessment Survey (MCAS)
5. FUNDING NUMBERS
6. AUTHOR(S)
Stanley, Bruce R. Jr.
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Naval Postgraduate School
Monterey, CA 93943-5000
8. PERFORMING ORGANIZATION
REPORT NUMBER
9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES)
Office of Naval Research
10. SPONSORING / MONITORING
AGENCY REPORT NUMBER
II. SUPPLEMENTARY NOTES
The views expressed in this thesis are those of the author and do not reflect the official policy or position of
the Department of Defense or the U.S. Government.
12a. DISTRIBUTION / AVAILABILITY STATEMENT
Approved for public release; distribution unlimited.
12b. DISTRIBUTION CODE
ABSTRACT (maximum 200 words)
The Maintenance Climate Assessment Survey (MCAS) was developed to proactively assess factors that contribute to a high
reliability organization and strong safety climate. The 3 rd Marine Air Wing (MAW), which was seeking to proactively improve its
safety posture requested the assistance of the School of Aviation Safety at the Naval Postgraduate School to examine its safety
climate. Previous studies of the MCAS instrument have focused on the items and their relationship to the HRO based model of
safety effectiveness components: process auditing, reward system, quality assurance, risk management, command and control, and
communication/functional relationships. The present effort is the first attempt to consider the relationship between item component
responses and demographic item responses. It evaluates 893 maintainer responses to the MCAS from 3 rd MAW and looks for
measurable effects due to demographics. This study finds that the regression models constructed using the demographics as
explanatory variables have very little utility in predicting scores for the components. This result allows planners the relief of using
the demographics as a low priority issue.
14. SUBJECT TERMS
Human Factors, Human Error, Accident Classification, High Reliability Organizations,
Corporate Safety Culture, Naval Aviation
15. NUMBER OF PAGES
116
16. PRICE CODE
17. SECURITY
CLASSIFICATION OF REPORT
Unclassified
18. SECURITY CLASSIFICATION OF
THIS PAGE
Unclassified
19. SECURITY CLASSIFI-CATION OF
ABSTRACT
Unclassified
20. LIMITATION OF
ABSTRACT
UL
NSN 7540-01-280-5500
Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. 239-18
1
THIS PAGE INTENTIONALLY LEFT BLANK
Approved for public release; distribution is unlimited
EVALUATING DEMOGRAPHIC ITEM RELATIONSHIPS WITH SURVEY
RESPONSES ON THE MAINTENANCE CLIMATE ASSESSMENT SURVEY
(MCAS)
Bruce R. Stanley, Jr.
Lieutenant, United States Navy
B.S., United States Naval Academy, 1993
Submitted in partial fulfillment of the
requirements for the degree of
MASTER OF SCIENCE IN OPERATIONS RESEARCH
from the
NAVAL POSTGRADUATE SCHOOL
June 2000
Robert R. Read, Co-Advisor
r<7Mf)0
Lyn R. Whitaker, Second Reader
^JZcUj_ €_
Richard E. Rosenthal, Chairman
Department of Operations Research
in
THIS PAGE INTENTIONALLY LEFT BLANK
IV
ABSTRACT
The Maintenance Climate Assessment Survey (MCAS) was developed to
proactively assess factors that contribute to a high reliability organization and strong
safety climate. The 3 rd Marine Air Wing (MAW), which was seeking to proactively
improve its safety posture requested the assistance of the School of Aviation Safety at the
Naval Postgraduate School to examine its safety climate. Previous studies of the MCAS
instrument have focused on the items and their relationship to the HRO based model of
safety effectiveness components: process auditing, reward system, quality assurance, risk
management, command and control, and communication/functional relationships. The
present effort is the first attempt to consider the relationship between item component
responses and demographic item responses. It evaluates 893 maintainer responses to the
MCAS from 3 rd MAW and looks for measurable effects due to demographics. This study
finds that the regression models constructed using the demographics as explanatory
variables have very little utility in predicting scores for the components. This result
allows planners the relief of using the demographics as a low priority issue.
vi
TABLE OF CONTENTS
I. INTRODUCTION.1
A. BACKGROUND. 1
B. PURPOSE.4
C. PROBLEM STATEMENT.5
D. SCOPE AND LIMITATIONS.6
E. DEFINITIONS.6
H. LITERATURE REVIEW.9
A. HUMAN ERROR.9
B. ORGANIZATIONAL SAFETY CULTURE.12
1. Definition. 12
2. Composition.12
C. HIGH RELIABILITY ORGANIZATIONS.13
1. Definition.13
2. Characteristics of HROs.14
D. ASSESSING SAFETY CLIMATE.16
1. Safety Space.16
2. MOSE and MCAS. 17
3. Instrument Design and Demographics.19
E. SUMMARY.20
IB. METHODOLOGY.23
vii
A. RESEARCH APPROACH
23
B. DATA COLLECTION
23
1. Subjects.23
2. Instrument.24
3. Procedure.24
C. DATA ANALYSIS.25
1. Data Tabul ati on.25
2. Statistical Analysis.26
IV. RESULTS.29
A. SIMPLE MODELS WITHOUT INTERACTION.29
B. MODELS WITH TWO FACTOR INTERACTION.32
C. REDUCED MODELS WITHOUT INTERATION.33
D. REDUCED TWO FACTOR INTERACTION MODELS.34
E. COMPARING MODELS.35
F. INTERVIEWS. 36
V. CONCLUSIONS.37
A. FINDINGS.37
B. RECOMMENDATIONS.38
APPENDIX A. 43-ITEM MAINTENANCE CLIMATE ASSESSMENT SURVEY.... 41
APPENDIX B. MODEL OF SAFETY EFFECTIVENESS COMPONENTS.45
APPENDIX C. INFLUENCE PLOTS FOR AUGMENTED MOSE COMPONENTS.. 47
APPENDIX D. SIMPLE MODELS: SCATTERPLOTS WITH SUPERIMPOSED
REGRESSION LINE.49
APPENDIX E. SIMPLE MODELS: HISTOGRAMS OF RESIDUALS.51
viii
APPENDIX F. SIMPLE MODELS: RESIDUALS VERSUS FITTED VALUES WITH
LOWESS SMOOTHING.53
APPENDIX G. SIMPLE MODELS: QQ-PLOTS.55
APPENDIX H. SIMPLE MODELS: ANALYSIS OF VARIANCE.57
APPENDIX I. TWO FACTOR INTERACTION MODELS: SCATTERPLOTS WITH
SUPERIMPOSED REGRESSION LINE.59
APPENDIX J. TWO FACTOR INTERACTION MODELS: HISTOGRAMS OF
RESIDUALS.61
APPENDIX K. TWO FACTOR INTERACTION MODELS: RESIDUALS VERSUS
FITTED VALUES WITH LOWESS SMOOTHING.63
APPENDIX L. TWO FACTOR INTERACTION MODELS: QQ-PLOTS.65
APPENDIX M. TWO FACTOR INTERACTION MODELS: ANALYSIS OF
VARIANCE.67
APPENDIX N. REDUCED SIMPLE MODELS.69
APPENDIX O. REDUCED SIMPLE MODELS: SCATTERPLOTS WITH
SUPERIMPOSED REGRESSION LINE. 71
APPENDIX P. REDUCED SIMPLE MODELS: HISTOGRAMS OF RESIDUALS.... 73
APPENDIX Q. REDUCED SIMPLE MODELS: RESIDUALS VERSUS FITTED
VALUES .75
APPENDIX R. REDUCED SIMPLE MODELS: QQ-PLOTS.77
APPENDIX S. REDUCED TWO FACTOR INTERACTION MODELS.79
APPENDIX T. REDUCED TWO FACTOR INTERACTION MODELS:
SCATTERPLOTS WITH SUPERIMPOSED REGRESSION LINE.81
APPENDIX U. REDUCED TWO FACTOR INTERACTION MODELS:
HISTOGRAMS OF RESIDUALS.83
APPENDIX V. REDUCED TWO FACTOR INTERACTION MODELS: RESIDUALS
VERSUS FITTED VALUES.85
IX
APPENDIX W. REDUCED TWO FACTOR INTERACTION MODELS:
QQ-PLOTS. 87
APPENDIX X. COMPARISONS ON MODELS USING ANOVA.89
APPENDIX Y. SUBJECT MATTER EXPERT RECOMMENDED CHANGES TO
MCAS DEMOGRAPHIC FACTORS.91
LIST OF REFERENCES.93
INITIAL DISTRIBUTION LIST.97
EXECUTIVE SUMMARY
Naval Aviation is a hazardous undertaking, but in spite of its inherent risk, its
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 to 1990.
Over the last decade, however, the proportion of aircraft losses in which human error has
been cited as a contributor has remained relatively constant. To address human factors
issues in flight mishaps, the Human Factors Quality Management Board (HFQMB) was
established in 1996. By using Mishap Data Analysis (MDA), Organizational
Benchmarking (OB), and Command Safety Assessment (CSA), the efforts of the
HFQMB resulted in a significant reduction in FM incidence from the perspective of
aircrew operations.
Although human error in maintenance is a smaller contributor, it has been shown
to be a factor in nearly one in five Class A FMs. To address human error in maintenance,
the Human Factors Analysis and Classification System - Maintenance Extension
(HFACS-ME) was developed to classify error types in maintenance. Since Naval
Aviation is shown to be a high reliability organization (HRO) defined as an organization
that operates in hazardous environment with less than its fair share of accidents, it shares
common characteristics with other HROs. These common characteristics are outlined in
the Model of Organizational Safety Effectiveness (MOSE) and are process auditing,
quality, reward system, risk management and command and control. Military aviation has
communication/functional relationships as a sixth component. The augmented MOSE is
the basis of the Maintenance Climate Assessment Survey (MCAS), which is used to
evaluate the organizational safety climate from the perspective of the maintenance
XI
personnel. The MCAS consists of six demographic items and 43 perception items. Each
of the 43 perception items maps into a single component of the augmented MOSE. These
questions are collapsed into six component scores for each respondent.
Although MCAS has been shown to be an effective tool for evaluating the safety
environment in a maintenance organization, demographic factors and their potential
relationship with maintainer responses have not been investigated. This study evaluates
MCAS responses from 894 maintenance personnel of the 3 rd MAW, and looks at how the
demographic factors of maintenance personnel might be biasing the component scores of
the MCAS. The results of this thesis are intended to further refine MCAS demographic
factors and provide Squadron Commanders with insight into the construct of their
maintenance personnel.
The component scores are fitted using the demographics as explanatory factors.
Univariate analysis is performed for each component using simple models without
interaction and also with models using two-factor interactions. These models are then
simplified in order to reduce the number of terms to a more manageable level.
The results of this thesis show that up to two-factor interaction, the demographic
factors of the MCAS poorly account for the variance in the responses. The reliance on
subjective perception in the scoring is the cause of the large amount of variance. Since
variance cannot be explained by the demographic factors, the MCAS appears to be
demographically unbiased. Input from subject matter experts is used to refine the
demographic factors. These revised factors are more usable for Squadron Commanders in
that they provide more insight into the make up of the maintenance organization.
xii
LIST OF ACRONYMS
CSA Command Safety Assessment
FM Flight Mishap
HFACS Human Factors Analysis and Classification System
HFACS-ME Human Factors Analysis and Classification System Maintenance
Extension
HFQMB Human Factors Quality Management Board
HRO High Reliability Organization
MAG Marine Air Group
MAGTF Marine Air Ground Task Force
MAW Marine Air Wing
MCAS Maintenance Climate Assessment Survey
MDA Mishap Data Analysis
MOSE Model of Organizational Safety Effectiveness
OB Organizational Benchmarking
School of Aviation Safety
SAS
I. INTRODUCTION
A. OVERVIEW
Naval Aviation is a hazardous undertaking, but in spite of its inherent risk, its
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 and
1990 (Naval Safety Center, 1997). Class A Mishaps are defined as Naval aircraft
incidents resulting in death, permanent disability, or property loss or damage in excess of
one million dollars (OPNAV 3750.6Q, 1989). A flight mishap (FM) is defined as those
mishaps in which there is $10,000 or greater DoD aircraft damage or loss of a DoD
aircraft, and intent for flight for DoD aircraft existed at the time of the mishap. Other
property damage, injury, or death may or may not have occurred. Naval Aviation
consistently maintains high levels of operability coupled with less than its fair share of
accidents (Goodrum, 1999). Naval Aviation also possesses the requisite characteristics of
a high reliability organization (HRO): process auditing, reward system, quality, risk
management, and command and control. For these reasons, Roberts (1988) labeled Naval
Aviation an HRO.
Although Naval Aviation is successful in reducing its Class A FM rate, over the
last decade the proportion of aircraft losses in which human error has been cited as a
contributor has remained relatively constant at four of five FMs (Naval Safety Center,
2000). In 1996, a Human Factors Quality Management Board (HFQMB) is established
after 17 Class A FMs occurred in only 75 days, climaxing when a Navy F-14 crashes into
a Nashville, TN neighborhood, to address human factors issues related to mishaps
(Nutwell & Sherman, 1997). The goal of the HFQMB is to cut the current Class A FM
rate due to human error in half by year 2000 (HFQMB Charter, 1996). The HFQMB
1
adopts three approaches to identify and target factors contributing to human error: 1)
Mishap Data Analysis (MDA), 2) Organizational Benchmarking (OB), and 3) Command
Safety Assessment (CSA).
MDA establishes the development of HFACS, which is used to identify and
prioritize human factors contributors to FMs. Among others, it determines inadequate
supervision and aircrew violations are significant contributors (Shappel & Wiegman,
1997). Using OB which explores programs which influence aircrew performance, the
HFQMB determines use of feedback mechanisms in commercial airlines improve crew
resource management training benefits (Nutwell & Sherman, 1997). Finally, a CSA
survey, based on a model of HROs, is developed to determine a command’s safety
posture from an aircrew perspective. This survey finds that 55% of the Navy respondents
and 65% of the Marine Corps respondents feel that their commands are committed
beyond what available resources can provide (Ciavarelli & Figlock, 1997). These
combined efforts make significant progress toward the HFQMB’s goal as evidenced by
fiscal year 1999 being the safest year in Naval Aviation history in terms of Class A FM
rate.
Maintenance is shown to be a contributing factor in nearly one in five Class A
FMs (Naval Safety Center, 2000). Additionally, during FY90-97, Class C FMs account
for 75% of all maintenance related mishaps (MRMs). Maintenance is one area where
hazards can be controlled and risk can be managed while an aircraft is on the ground.
Much work is done in the field of human factors in maintenance safety for commercial
airlines (“Human Factors in,” 2000). In 1988, the Aviation Safety Research Act (ASRA)
mandates close study of aging aircraft structures and human factors affecting safety
2
(“History,” 2000). In the spirit of this mandated study, Boeing finds that incomplete
installation (34%), damaged on installation (15%), improper installation (11%) and
equipment not installed/missing (11%) were the top contributors in maintenance error
(Komamiski, 2000). This investigation and classification of types of human error in
maintenance leads to the development of Boeing’s Maintenance Error Decision Aid
(MEDA), a system that aids operators and maintainers in the investigation and mitigation
of maintenance related errors (Allen, Rankin, & Sargent, 1998).
The ASRA is one of the precursors for the FAA’s current goal to reduce the fatal
accident rate 80% by 2007 as compared to 1994-1996 baseline data (FAA, 1998). Several
key initiatives are the stepping stones for this ultimate goal: 1) the development of a
maintenance resource management system; 2) establishment of new training
requirements; 3) implementation of technical advances in aircraft maintenance at repair
stations; 4) enforcement of safety recommendations from the National Transportation
Safety Board (NTSB); and 5) recommendations for aging systems maintenance. Little
work until recently is done for military aviation, and recent efforts involve studying
forms of maintenance error (Schmorrow, 1998) and the perceived maintenance safety
climate (Baker, 1998). From a proactive perspective, efforts must be made to continue
developing assessment tools to identify potential areas for risk management and control
of conditions before a mishap occurs.
Using the Maintenance Climate Assessment Survey (MCAS), Goodrum (1999)
and Oneto (1999) are able to show the prototype survey effectively evaluates a
maintainer’s perception of safety in maintenance operations. They also further refine the
MCAS into a present 43 question format. One aspect of the MCAS yet to be addressed is
3
the demographic categories to determine their potential relationship with maintainer
responses. Given the structure of maintenance organizations within aircraft communities
vary, it is unlikely that any pair of samples will have equal proportions of all
demographic variables. It is anticipated that individual demographic characteristics may
influence MCAS responses and are therefore potentially biasing the results. By
understanding the effects of the demographics, one can understand if they impact their
organization’s safety climate.
B. BACKGROUND
The 3 rd Marine Air Wing (MAW) is a combat-ready expeditionary aviation force
capable of short-notice worldwide employment to Marine Air Ground Task Force
(MAGTF) fleet and unified commanders. It is composed of 28 squadrons divided into
four Marine Air Groups (MAGs) based in Southern California and Arizona. Each MAG
has its own combat mission: MAG 11 provides air support to MAGTF commanders;
MAG-13 provides close-air support, conducts armed reconnaissance, and assumes
limited air-defense roles; MAG-16 transports and resupplies Marine air and ground units;
and MAG-39 provides utility helicopter support, close-in fire support, fire support
coordination, aerial reconnaissance, observation and forward air control in aerial and
ground escort operations during ship-to-shore movement and subsequent operations
ashore. The aircraft used in these missions are AH-lWs, UH-lNs, CH-53s, CH-46Es,
F/A-18DS, F/A-18s, AV-8s and C-130Ts.
From 1990 to 1996, maintenance is a causal factor in 17 percent of all Naval
Aviation class A FMs (Naval Safety Center, 1997). From April 1997 to July 1999,
maintenance, maintenance personnel or maintenance depot is cited as a causal factor in
4
14 FMs (eight class C FMs, four class B FMs, and two class A FMs) experienced by 3 rd
MAW. The Commander of 3 rd MAW requests the assistance of the School of Aviation
Safety (SAS) at the Naval Postgraduate School, which in turn provides safety and risk
management training to personnel, mishap data analysis, and administration of safety
climate surveys to help locate problems in the organization.
C. PROBLEM STATEMENT
Human error in aviation is an issue that needs to be addressed, and it is recognized
that the organization has an impact on factors that lead to it. Organizations that possess
the attributes of a HRO tend to generate environments conducive to the reduction or
control of human error and consequently experience fewer mishaps. Organizations
aspiring towards the reduction of mishaps need to assess their safety posture as it relates
to the attributes of HROs. The School of Aviation Safety at the Naval Postgraduate
School has developed surveys to assess HRO characteristics in the operational
environment for aircrew and maintenance personnel.
The 3 rd MAW in an attempt to improve its safety posture enlists to have the
School of Aviation Safety employ the MCAS survey to assess maintainer perception of
HRO characteristics in its recent operations. These results are revealing, however in order
to provide for better interpretation of the results, an exploration of the demographic
variables is in order. This will help commanders to target more effectively specific areas
of the organization that require attention.
The current version of the MCAS is administered to the 3 rd MAW during the last
half of 1999. Using statistical methods, the collected data is analyzed to assess
5
differences in responses that are correlated to differences in demographics. This thesis
explores the following questions:
1. Are there measurable demographic effects to the responses on the MCAS?
2. Is there enough information in the demographics to be used in an adjustment
process of the overall scores?
3. Can the MCAS be refined further to either collapse or expand demographic
factors?
D. SCOPE AND LIMITATIONS
Active duty U.S. Marine Corps Squadrons maintenance personnel of the 3 rd
MAW are surveyed during the fall of 1999. Only those squadrons with a representative
number of respondents are used in the survey. Chapter II provides a basis for
understanding human error, organizational safety culture, high reliability organizations
and the assessment of a safety climate. Chapter III presents a discussion of the
methodology used in this study. Results of data analysis are presented in Chapter IV.
Chapter V summarizes previous chapters and provides conclusions and recommendations
as they relate to the material.
E. DEFINITIONS
This thesis uses the following definitions (DON, 1989):
Naval Aircraft . Refers to U.S. Navy, Naval Reserve, U.S. Marine Corps, and U.S.
Marine Corps Reserve aircraft.
Mishap . A Naval Aviation mishap is an unforeseen or unplanned event that directly
involves naval aircraft, which result in $10,000 or greater cumulative damage to naval
aircraft or personnel. The mishap is further divided into three classes based on the
6
amount of damage to the aircraft, property and personnel injury. The following are
the definitions of the three classes:
a. Class A . A mishap in which the total cost of property damage (including all
aircraft damage) is $1,000,000 or greater; or a naval aircraft is destroyed or
missing; or any fatality or permanent total disability of a person occurs with direct
involvement of naval aircraft.
b. Class B . A mishap in which the total cost of property damage (including all
aircraft damage) is $200,000 or more but less than $1,000,000 and/or a permanent
partial disability, and or the hospitalization of five or more personnel.
c. Class C . A mishap in which the total cost of property damage (including all
aircraft damage) is $10,000 or more but less than $200,000 and/or injury results in
one or more lost workdays.
Mishap rate . The total number of Class A,B and C mishaps per 100,000 flight hours.
MCAS . A 43-question survey used to gain insight into the maintenance community’s
perception concerning aviation mishaps within the Navy and Marine Corps.
HFACS-ME . A taxonomic system used to classify causal factors that contribute to
maintenance related mishaps.
HRO . High-Reliability Organization, is an organization that operates in a hazardous
environment, yet produces very low rate of accidents and incidents, operating effectively
and safely and having the characteristics of leadership, sound management policies,
procedure standardization, adequacy of resources and staffing, a defined system for risk
management, and other factors.
7
THIS PAGE INTENTIONALLY LEFT BLANK
8
II. LITERATURE REVIEW
A. HUMAN ERROR
Reason (1990) defines error as a planned sequence that fails to achieve its
intended outcome in the absence of external influence. He (1997) later describes error
types as active or latent. Where the effects of active errors are often immediate and
confined, latent conditions lie dormant until set off by a chain of local events and can be
contributing factors in a variety of failures. This model of latent conditions and active
failures is adopted by the Naval Safety Center to investigate Class A FMs with respect to
aircrew error, and was the basis for the development of Human Factors Analysis and
Classification System (HFACS) (Shappel & Wiegman, 1997).
I Activity
‘Hands On’
■KifllifgW
Normal control
Low
Moderate
HHE39H
Emergency control
Moderate
Low
| Maintenance-related
■■
HES3HI
MEM3HI
Table 1. Likelihood of Performance Problems in Given Activities.
Reason (1997) also models human error in the scope of universal human activities
(see Table 1) and the likelihood of performance problems within each of these types of
activities. With HFACS, the Naval Safety Center is able to address human error in
normal control and emergency control conditions. But Reason asserts that maintenance is
the area with the highest likelihood of human error because maintenance related activities
are almost exclusively reliant on human performance in the three areas of hands on,
criticality and frequency. Even with progress in technology, human fallibility remains
9
constant (Reason, 1997) and with the frequency of planned maintenance compounded
with the many pairs of fallible human hands working on exceptionally complicated
systems, Reason’s model is a chilling prediction of 3 rd MAW’s situation.
To address the maintenance related activity as a contributor to the total of human
errors in FMs, Schmidt, Schmorrow and Hardee (1998) extend HFACS to specifically
address the maintenance component of accident causation with the HFACS-Maintenance
Extension (ME). The HFACS-ME expands upon Reason’s model of latent states and
psychological precursors to unsafe acts. Reason (1990; 1997) differentiates these latent
states in that they create the potential for human error. HFACS-ME classifies latent states
in the maintenance environment, with three levels of error causation under four
categories of conditions. The first order conditions are broad precursor categories
(supervisory conditions, maintainer conditions, working conditions and maintainer acts)
that are further divided into more specific precursors to human error of the second and
third order (see Table 2).
The causes of human error are many. Wickens, Gordon and Lui (1997) state that
human error can be induced by “inattentiveness, poor work habits, lack of training, poor
decision making, personality traits, social pressures, and so forth” (p. 427-428). Redmill
& Rajan (1997) note that a common component in accidents is a worker’s loss of
concentration which can be caused by “boredom, disinterest, distraction, or attempt to do
two or more things at once” (p.12). Wickens, Gordon and Lui (1997) assert that the most
common type of maintenance related error is that of omission. Considering that the nature
of planned maintenance is to frequently disassemble, inspect then reassemble
components, it is clear to see the high probability of human error in maintenance (Reason
10
1997). But as Perrow and Reason point out, the errors themselves are seldom isolated, but
often a single event in a causal chain (Perrow, 1984), or encouraged, or at least not
prohibited from occurring by latent conditions in the organization (Reason, 1997).
First Order
Second Order
Third Order
Supervisory Conditions
Unforeseen
Hazardous Operations
Inadequate Documentation
Inadequate Design
Squadron
Inadequate Supervision
Inappropriate Operations
Failed to Correct Problem
Supervisory Violation
Maintainer Conditions
Medical
Mental State
Physical State
Physical/Mental Limitation
Crew Coordination
Communication
Assertiveness
Adaptability/Flexibility
Readiness
Preparation/Training
Qualification/Certification
Violation
Working Conditions
Environment
Lighting/Light
Exposure/Weather
Environmental Hazards
Equipment
Damaged
Unavailable
Dated/Uncertified
Workspace
Confining
Obstructed
Inaccessible
Maintainer Acts
Error
Attention
Memory
Rule/Knowledge
Skill
Violation
Routine
Infraction
Exceptional
Table 2. HFACS-ME Levels of Error Causation.
li
B. ORGANIZATIONAL SAFETY CULTURE
1. Definition
Organizational culture is defined as shared values and beliefs that interact with an
organization’s structures and control systems to produce behavioral norms (Uttal, 1983).
All organizations have their own engineered culture whether good or bad. A safety
culture is ideal for complex organizations and is defined as the product of individual and
group values, attitude, competencies, and patterns of behavior that determine the
commitment to, and the style and proficiency of, an organization’s health and safety
programmes (Booth, 1993).
2. Composition
Redmill and Raj an (1997) state that there are three general aspects of safety
culture: awareness, commitment and competence. Awareness must be present in all
aspects of design, management and decision making. When a safety mishap does occur,
it is commitment that drives the leadership of the organization to investigate and locate
contributing factors and take immediate action to prevent another occurrence.
Competence is a combination of education, training, professionalism and personality
traits that are appropriate for a given task or job (Redmill & Rajan, 1997).
Reason (1997) prefers the term informed culture, and he divides informed culture
into four subcultures: reporting culture, just culture, flexible culture and learning culture.
Reporting culture is “an organizational climate in which people are prepared to report
their errors and near-misses.” Just culture is an “atmosphere of trust in which people are
encouraged, even awarded, for provided essential safety-related information - but in
which they are also clear about where the line must be drawn between acceptable and
12
unacceptable behavior.” Flexible culture involves “shifting from the conventional
hierarchical mode to a flatter professional structure, where control passes to task experts
on the spot, and then reverts back to the traditional bureaucratic mode once the
emergency has passed. Such adaptability is an essential feature of the crisis-prepared
organization.” Learning culture is “the willingness and the competence to draw the right
conclusions from its safety information system, and the will to implement major reforms
when their need is indicated.”
Safety culture has powerful effects. First, it is self perpetuating where workers
learn from each other and encourage each other to work in a manner consistent with the
organization’s safety culture. People are quick to follow the example of coworkers, even
if this means a lack of vigilance in safety (Redmill & Rajan, 1997). Wogalter, Allison, &
McKenna (1989) assert that “people are extremely susceptible to social norms; they are
likely to engage in safe or unsafe behaviors to the extent that others around them do so.”
Safety culture is also self preserving as new workers learn to identify acceptable methods
of accomplishing work and are able to pass those standards along to new employees,
good or bad (Redmill & Rajan, 1997).
C. HIGH RELIABILITY ORGANIZATIONS
1. Definition
Roberts (1990) and Libuser (1994) explain that High-Reliability Organizations
(HROs), organizations that operate in a hazardous environment, yet produce very low
rates of accidents and incidents, which operate effectively and safely have certain key
characteristics in common. Reason (1997) calls them organizations with less than their
fair share of accidents, and “highly complex, technology-intensive organizations that
13
must operate, as far as humanly possible, to a failure-free standard.” He also explains
that HROs manage complex technologies that have very low tolerance for error, yet
maintain the flexibility to successfully function in environments of extreme intensity.
2. Characteristics of HROs
Examples of HROs are the nuclear power industry, petrochemical industry, and
airline industry. Additionally, Figlock (1998) identifies Naval aviation is an HRO.
Although diverse in purpose, Roberts and Libuser believe these organizations share
several common characteristics: leadership style, management policies, procedures
standardization, superior training, a reward system that recognizes safety achievement,
adequacy of resources and staffing, effective management of risks associated with
hazardous operations, and other factors.
HROs have a requisite variety. As Weick (1987) states, having diverse people
from diverse backgrounds and experiences builds requisite variety that is required for
relatively simple humans to operate complex systems. Additionally, this diversity is
essential in problem solving, as individuals will approach the same problem uniquely, so
that the collective contribution is greater than any one individual’s input.
HROs typically exhibit a high degree of training. Weick (1987) notes that
“training for the operation of high reliability systems is often tough and demanding so
that the faint of heart and the incompetent are weeded out.” This is because HROs are not
afforded the luxury of trial and error. Training is often in the form of simulation and
stories. Stories have a big affect on the reliability of an HRO by lending experience to the
inexperienced:
14
The basic idea is that a system which values stories, story tellers, and
storytelling will be more reliable than a system that derogates these
substitutes for trial and error. A system that values stories and storytelling
is potentially more reliable because people know more about their system,
know more of the potential errors that might occur, and they are more
confident that they can handle those errors that do occur because they
know that other people have already handled similar errors (Weick, 1987,
P- 113).
By sharing the experiences of skilled personnel, novices do not have to learn from their
own mistakes and are also granted the insight of the skilled.
HROs appear bureaucratic and uneventful on the surface. There is a strict chain of
command in place that dictates policy, procedure and environment. This strong
centralization is apparent during periods of relatively low intensity, but as intensity of
operating increases, the true nature of the HRO is revealed, where flexibility, delegation,
improvisation and technical expertise dominate (Reason, 1997; Weick, 1987). This is
how HROs can be simultaneously centralized and decentralized. Responsibility and
judgment remain centralized while creativity, improvisation and unsupervised problem
solving become decentralized in environments of high intensity (Weick, 1987).
Reliability in HROs is another deceptive aspect. Weick (1987) calls reliability a
“dynamic non-event,” meaning a great deal of effort goes into ensuring nothing happens.
This dynamicism is based on the belief that reliability is fleeting and systems tend to
move to states of unreliability unless constantly maintained:
Part of the mindset for reliability requires chronic suspicion that small
deviations may enlarge, a sensitivity that may be encouraged by a more
dynamic view of reliability (Weick, 1987, p. 119).
It is the organizational culture for personnel to look for problems before they happen.
Weick continues that because of the invisibility of the dynamics behind reliability, there
15
is a perception that reliability is easily achievable and is only noticed in the presence of a
breakdown.
D. ASSESSING SAFETY CLIMATE
1. Safety Space
Reason (1997) states that organizations can be mapped into a safety space which
is a continuum of degrees of susceptibility to accidents (see Figure 1). Organizations with
higher resistance will generally have fewer mishaps while organizations with higher
vulnerability will generally experience more mishaps. No organization is immune.
Chance, unforeseen circumstances, failures in defenses and human error can cause even
the most resistant organizations to experience accidents. Within the safety space, currents
tend to push organizations away from the extremes of resistance or vulnerability and
toward the center, a compromise between the two. If an organization has the desire to
become more resistant, it must swim upstream.
Figure 1. Organizational Safety Space.
16
The effort required to move in the direction of increased resistance must be put
into reactive and proactive measures. Reason (1997) contends that investigating mishaps
to find causal factors to be addressed is not even half the battle. To effectively move the
organization, mishap investigation must be coupled with the identification of conditions
needing correction, and regular checks.
2. MOSE and MCAS
Libuser’s (1994) current Model of Organizational Safety Effectiveness (MOSE) is
based on work by Roberts and is a categorization of the common characteristics of
HROs. These characteristics are mapped into five components: 1) Process Auditing (PA)
- checks by members to identify hazards; 2) Reward System (RS)- expected rewards or
disciplinary action used to shape behavior; 3) Quality Assurance (QA)- promotion of
quality performance; 4) Risk Management (RM)- system to identify hazards and control
operational risks; and 5) Command and Control (CC)- safety climate, leadership
effectiveness /policies, and procedures for mitigating risks. These components are very
similar to the aspects of Reason’s (1997) informed culture (Table 3).
Libuser’s MOSE Components
Reason’s Informed Culture
Process Auditing (PA).
Leaning Culture
Reward System (RS)
Just Culture
Quality Control (QA)
Reporting Culture
Risk Management (RM)
Flexible Culture
Command and Control (CC)
Flexible Culture
Table 3. Comparison of Libuser’s MOSE and Reason’s Informed Culture.
17
Ciavarelli and Figlock (1997) adapt the MOSE for use in Naval Aviation using
practices and terminology of that environment and develop the Command Safety
Assessment, a survey that addresses each of the MOSE categories from the viewpoint of
the aircrewman. This survey is administered to 1,254 aviators revealing that
organizational and supervisory issues are seen by aircrewmen as impacting flight safety.
The Maintenance Climate Assessment Survey (MCAS) is the product of the
implementation of the MOSE and CSA in a maintenance context. Baker (1998) starts by
reducing 155 candidate questions to 67 items that specifically addressed aviation
maintenance. Augmenting Libuser’s (1994) five category MOSE model with a sixth
category, Communication/Functional Relationships, Baker (1998) modifies the CSA to
look at aviation safety from the point of view of the maintenance person. Using
regression techniques, Baker is able to further reduce the survey into a compact 35 item
form, with almost all questions mapped to a single category of the augmented MOSE.
Goodrum (1999) and Oneto (1999) show that the MCAS is a valid tool to
accurately assess an aviation maintenance environment, but note that some items in the
survey need restructuring. Oneto notes that these items address more than one category of
the MOSE. Their inputs help change the MCAS to its current 43 item format (see
Appendix A). While Goodrum and Oneto are able to show content validity in the MCAS,
there is much left to examine. Since there is no known or accepted measure for MCAS
results and providing feedback to the concerned squadrons, an effort is underway to
explore concurrent validity in the survey on a per question basis (Schmidt, personal
communication). Questions that have a low response mean are noted as areas that need
18
attention, corresponding to a particular category of the MOSE, and which part of the
HRO needs closer examination.
Additionally, by looking at squadron mean scores to the survey and available
mishap data, Harris (personal communication) is looking at the MCAS predictive validity
in the incidence of mishaps within a squadron based on adjusted mean scores. While the
work by Harris seeks to broaden the scope and applicability of the MCAS, this study
explores internal aspects of the survey. By separating responses by demographics, this
thesis will further explore the attitude of the Naval aviation maintenance person with
respect to safety.
3. Instrument Design and Demographics
The design of the MCAS is a cross-sectional one time look at a maintenance
organization (see Appendix A). It is a self-administered questionnaire that polls
respondent perception about the safety of their working environment at all levels.
Demographic items preserve anonymity by excluding personal questions which could be
linked to individuals allowing personnel the freedom to express their true opinion to the
perception items (Oneto, 1999).
Though there are many methods for data collection from people, surveys, when
designed properly, are very effective for recording respondent scores based on a
particular model (Fink & Kosecoff, 1985). MCAS responses are forced using a five point
Likert scale (see Appendix A). Using a forced scale instead of subjective comments
allows for rapid compilation of data and analyzation of responses as numerical values,
which is convenient for numerical scoring and comparisons of scores between
respondents, or groups of respondents (Fink & Kosecoff, 1985). The grouping is
19
facilitated by the six demographic items of the survey (Baker, 1998). By being able to
group types of maintainers independent of their scores, it is possible to investigate for
unequal perceptions across demographic groups.
The demographic items in the MCAS capture aspects of the maintainer within the
organization such as experience, and rank, and ignore personal information such as age,
race, sex and ethnicity (Baker, 1998). Although it would be difficult to capture every
possible combination of demographic factors, the MCAS is able to capture most job types
from most aircraft communities. Each of the demographic line items in the MCAS
represents a simple factor that might influence the scoring. It is unknown which of these
demographics constitutes a valid or invalid factor.
E. SUMMARY
HROs are complex and dynamic by nature but are not impervious to unsafe trends
in human performance. Perrow (1984) points out that regardless of an organization’s
structure and nature, “normal accidents” will continue to occur. With HROs, these
accidents tend to occur less frequently, but the consequences of the accidents tend to be
large in magnitude. These organizations must put effort into swimming upstream through
Reason’s safety space towards increased resistance to accidents. This happens through
reactive measures like mishap analysis and proactive measures to identify “pathogenic
conditions” (Reason, 1997). Though reactive measures are in place, proactive measures
are coming up to speed.
Two critical parallel developments in organization safety theory in regards to
aviation safety are the development of effective taxonomies (e.g., HFACS-ME) and the
identification and accurate modeling of HROs (e.g., augmented MOSE). The
20
development of CSA and MCAS are steps to link the taxonomies and the organization
models. Since MOSE parallels Reason’s informed culture, it is possible with MCAS to
identify those conditions that are not conducive to safety and take proactive measures to
move the organization towards increased resistance to accidents in the safety space. Since
3 rd MAW is a HRO, CSA and MCAS allow for the identification of the MOSE
components that require attention.
The MCAS has been revised into a more usable form, and has been validated,
showing that the individual items do address specific MOSE components. This thesis is
another step into revising the MCAS further by looking at the validity of the demographic
factors. Mapping the demographic factors into the MOSE component scores will show if
the factors are relevant or can be removed from the survey. Additionally, if these factors
do not account for the variance in scores, the survey is not asking the correct
demographic questions and will require further revision.
21
THIS PAGE INTENTIONALLY LEFT BLANK
22
III. METHODOLOGY
A. RESEARCH APPROACH
The intent of this study is to assess the maintained s perception of safety in his or
her work environment. This research involves the use of analysis techniques to partition
the collected data into smaller groups based on demographics then investigate differences
in responses among the groups. If statistical differences are found, a comparison between
the group mean and a particular squadron mean shows how a particular demographic
differs from the rest of the squadron. Conversely, it shows which demographics have
response means that are more reflective of the squadron means.
B. DATA COLLECTION
1. Subjects
Surveys are administered to 977 officers and enlisted personnel responsible for
Naval Aviation maintenance. These subjects come from squadrons and maintenance units
of the 3 rd Marine Air Wing located at MCAS Miramar, CA, Camp Pendleton, CA, and
MCAS Yuma, AZ. The aircraft represented are the AH-1 “Super Cobra,” UH-1 “Huey,”
CH-53 “Super Stallion,” CH-46E “Sea Knight ” F/A-18D “Night Attack Hornet,” F/A-18
“Hornet,” AV-8B “Harrier,” and the C130T “Hercules.”
Additionally, subject matter experts are interviewed about what they consider to
be important demographic information about the personnel in their maintenance
organizations. Subject matter experts are military aviators with at least eight years active
duty service. The results of these interviews are given in the next section.
23
2. Instrument
The MCAS is a self-administered, group survey consisting of two parts: 1)
demographics; 2) perception. Part I captures demographic factors of each subject:
community, squadron, rank, years of aviation maintenance experience, work center, and
shift. There are eight choices available for community, with an additional option of
“other.” The squadron factor records the three-digit squadron designator. Embedded in
these two factors is aircraft type (seventh factor). Rank is divided into four levels of
enlisted personnel and three levels of officer personnel. Years of aviation maintenance
experience is partitioned into seven levels. Work center or shop, is divided into eight
shops with the option of “other.” Shift divides subjects into dayshift or nightshift.
Part II captures subject perception of his or her work environment. There are 42
items, each of which is mapped into a single augmented MOSE component: process
auditing (six questions), reward system (eight questions), quality assurance (six
questions), risk management (nine questions), command and control (eight questions),
and communication/functional relations (six questions). Each perception item asks
subjects to rank a specific safety related activity or aspect of their organization using a
five point Likert rating scale with verbal anchors as follows: Strongly Disagree, Disagree,
Neutral, Agree, Strongly Agree. When completed, the items for each MOSE component
are averaged to attain six composite scores, each one corresponding to the subject’s rating
of that particular augmented MOSE component for his or her organization.
3. Procedure
The survey is administered on site and in a group setting at the various
participating Squadrons of the 3 rd MAW. Additionally, the survey is given in
24
conjunction with a scheduled maintenance safety presentation on human factors issues in
aviation. The Squadrons are in various stages of training and operational tasking at the
time of the survey being administered. The variety of operational tasking with which the
squadrons are simultaneously involved during the administration of the MCAS accounts
for much of the variance in the number of surveys collected from each squadron.
Potential MCAS respondents are briefed on maintenance issues, the survey and its
purpose and questions that arise pertaining to the survey are answered by the survey
administer. Respondents fill out the surveys using scannable computer forms. The
surveys are then immediately collected upon completion to allow for maximum
accountability.
For personal interviews, subject matter experts report what they think are the
important factors when considering the demographics of personnel in their maintenance
organizations. Notes are taken during these interviews and a list is compiled from the
responses of the personnel interviewed. This list represents potential MCAS demographic
items.
C. DATA ANALYSIS
1. Data Tabulation
Survey results were compiled into a database using a scanning machine, then
imported into Microsoft Excel. The spreadsheet consists of rows of respondents and
columns of survey items (both demographic and survey items). Demographic items
record mainly bivariate and multivariate responses, such as squadron (aircraft type
embedded), rank, and years experience. Survey item responses were assigned a numerical
value of 1 through 5 corresponding to the Likert scale, with higher values being assigned
25
to more positive responses (strongly agree) and lower values assigned to more negative
responses (strongly disagree). Each of the questions in the survey addresses one of the
six MOSE components. Items addressing similar MOSE components were collapsed into
an average score for that particular component. Any items that were missing values were
excluded and not averaged into the component score. No weighting is assigned to items
in the event of a missing item score. The three demographic response items
corresponding to the respondent’s squadron number were collapsed into a single coded
value.
2. Statistical Analysis
Microsoft Excel is used to provide summary statistics and initial familiarization
with the data. The data were cleansed by removing subject responses with omitted
demographic items leaving 894 responses. Items corresponding to augmented MOSE
components were averaged for each respondent, leaving six scores along with
demographic response items. If subjects omitted a perspective item response, the
component score is averaged for the completed items of that component. No weighting
for omitted perspective items is administered. Histograms of the demographic make up of
the data are constructed. The data are then exported to ARG for its powerful graphing
capabilities. Initial scatterplot matrices of each component over all demographic factors
revealed no linear, exponential, or power trend, although scatterplots matrices of the
components over themselves revealed a linear trend.
The data are then transported to MathSoft S-plus for analysis. Categorical
demographics are coded as factors and the aov function is implemented to fit the six
component scores based on demographic factors. Six models (one for each component)
26
are constructed for a regression without interaction among factors. Six additional models
are fit for a regression looking at two factor interaction. Each model is then simplified by
using S-plus to remove unimportant terms. Models are compared and similar models for
some of the components are found. Three term interaction models are not explored in this
thesis.
27
THIS PAGE INTENTIONALLY LEFT BLANK
28
IV. RESULTS
A. SIMPLE MODELS WITHOUT INTERACTION
A scatterplot matrix is constructed to see if there are any trends in the data. This is
done without designating the independent variables as factors. The scatterplot shows that
there is some relationship between the component scores, but little information about the
demographics is revealed. Figure 3 is a scatterplot matrix for PA versus each of the
demographic factors. Visual inspection reveals a possible relationship between PA and
Rank and Total years of Aviation experience.
Figure 1. Scatterplot of PA versus Demographic Factors.
Three dimensional bar plots for the components versus each factor are constructed
to see if there are any visual clues as to some type of relationship between the factors and
the response. No linear, log linear or exponential relationship is visible. However, the
29
plots seem compatible with those of normally distributed histograms for each level of
factor, all centered in the same approximate region of the component score (e.g. Figure
4).
Figure 2. Three Dimensional Bar Plot of PA Scores. PA scores
are plotted against different levels of Total Years of Aviation
Maintenance Experience.
Since there is no indication that a transformation of the data is required due to
visible trends, linear models for the six individual components are fit against
untransformed factors without interaction using the aov function in S-plus. Model
checking plots are constructed and case 219 is shown to have very high influence in all
models. Case 219 is an E-6/7 with 15-20 years of aviation maintenance experience,
works the day shift in “other” work center, in a VMH squadron. Although there is no
significance test associated with Cook’s Distance, case 219 is deleted from the data set
due to its unusually high influence and new models were fit (see Appendix C).
Model checking plots are constructed to check the fit of the models with case 219
removed. The scatter plot of the data with the regression superimposed reveals what
30
might be a slight upward trend in the response for all components (see Appendix D). The
histograms of the residuals show that the distribution of the residuals appear to be normal
(see Appendix E), and the scatterplot of the residuals versus the fitted values shows no
discemable pattern in the residuals (see Appendix F). The QQ-plot shows that the
residuals are thin at both tails for all models (see Appendix G), but the normal shape is
tenable.
Component
R :2
<7
PA
0.1095
0.5529
QA
0.1219
0.6327
RS
0.1427
0.5896
RM
0.1869
0.5690
CC
0.1280
0.6349
CR
0.1233
0.6846
Table 4. R-squared and o for Models without Interaction.
Values for the coefficient of determination, R 2 , show that these models account
but poorly for the variance in the data. The best model is RM, accounting for less than
19% of the total variance (see Table 4). The model with the lowest R 2 is PA with only
11% of the total variance explained. Values for standard error, a, indicate that there is a
large spread in the response values. For example, with a perfect R 2 of 1, the model for PA
tells us that 68% of the respondents score between 3.31 and 4.42, and that 96% score
31
between 2.76 and 4.97. Since the only possible scores are between 1 and 5, relatively
high values for & are not much help with understanding the data.
The results from the analysis of variance from each model causes rejection of the
null hypothesis that all of the coefficients in the model are zero, accepting the alternate
hypothesis that at least one coefficient is not equal to zero (see Appendix H).
Additionally, the models for PA, CC and CR fail the lack of fit test indicating that the
shape of the fit is not correct. These models do not do well in describing the data.
B. MODELS WITH TWO FACTOR INTERACTION
Two-factor interaction models are constructed for all of the components using S-
plus to see if more of the variance in the data can be modeled and model checking plots
are constructed. The scatterplot of the data with the superimposed regression shows a
linear trend in the response against the factors (see Appendix I), and the distribution of
the residuals appears to be normal (see Appendix J). The scatterplot of the residuals
versus the fitted values shows no discemable pattern in the residuals (see Appendix K).
The QQ-plots show strange behavior at values close to zero, but that they are close to
being normal for all components (see Appendix L). R 2 and & are given in Table 5.
These two term interaction models are better at explaining more of the variance in
the data as indicated by the values for the coefficients of determination, however there is
very little reduction in the values for the standard error. This improvement in the values
for R 2 comes at the cost of increased complexity in the models. While the models without
interaction have 43 terms, the two term interaction models have 343 terms. By adding
300 terms to the model, 300 degrees of freedom are lost resulting in no significant
improvement in the standard error.
32
Component
R 2
<7
PA
0.4813
0.5246
QA
0.4832
0.6034
RS
0.5046
0.5572
RM
0.5207
0.5431
CC
0.4968
0.5996
CR
0.4850
0.6523
Table 5. R 2 and o for Two Factor Interaction Models.
Analysis of variance on the two factor interaction models causes rejection of the
null hypothesis that all of the coefficients are equal to zero. The alternative hypothesis is
accepted meaning that at least one coefficient is not equal to zero. Present in the anova
tables is evidence that some of the terms are not necessary in the model and that
simplification is possible.
C. REDUCED MODELS WITHOUT INTERACTION
The step function in S-plus is used to subtract terms from the simple models in an
effort to simplify the models without losing too much of the information they provide. S-
plus accomplishes this by using Akaike’s information criterion which is of the form:
AIC =-2logL(x m+u ...,x n \xi, ...,x m ) + 2r
where r is the total number of estimated parameters. The AIC is a value that penalizes a
model for having high complexity when compared to simpler models with fewer terms.
33
The step function is applied to the simple models without interaction and the results are
given in Appendix N. The reduced models have between 15 and 33 terms compared to 43
terms for the original models. Model checking plots are constructed with no significant
graphical differences between the reduced simple models and the simple models (see
Appendices O-R). Values for the coefficient of determination and standard error are
given in Table 6. As expected, less of the total variance is explained by the reduced
models, and the value for a increases.
Component
R 2
a
PA
0.0915
0.5532
QA
0.1002
0.6345
RS
0.1240
0.5925
RM
0.1703
0.5711
CC
0.1071
0.6365
CR
0.0784
0.6906
Table 6. R 2 and a for Reduced Models without Interaction.
D. REDUCED TWO FACTOR INTERACTION MODELS
The step function in S-plus is applied to the two factor interaction models and the
results are given in Appendix S. The data were fit to the new models and model checking
plots are constructed and given in Appendices T-W. The scatterplot of the data with the
superimposed regression line indicates that there might be a linear relationship between
the factors and the score, but most of the data looks like a point cloud. The histograms of
34
the residuals have the appearance of a normal distribution. The predicted values versus
the residuals have no pattern and the QQ-plots look normal.
The reduced models have significantly decreased values for R 2 with little change
in values for a. These models use between 33 and 59 terms which is a significant
simplification over the 343 term models, but a lot of the explanation of the total variance
is lost in the transition. Table 6 summarizes R 2 and a for the reduced models.
Component
R 2
a
PA
0.1098
0.5496
QA
0.1228
0.6287
RS
0.1854
0.5802
RM
0.2453
0.5534
CC
0.1747
0.6225
CR
0.1342
0.6779
Table 7. R 2 and a for Reduced Two Factor Models with Interaction.
E. COMPARING MODELS
The S-plus anova function is applied to pairs of models to see if they are
statistically different. The results are given in Appendix U. At a = 0.05, eight pairs of
models were found to be statistically similar, with the highest similarity between simple
and reduced two term interaction models for PA and QA (p-Value > 0.99). Most of the
model pairs are statistically different.
35
Choosing the most appropriate model for a complex data set is compromise
between a model that explains enough of the data while being simple enough to use. The
two factor interaction models are too complicated to be practical with 343 terms. The rest
of the models are not different enough to distinguish in practice, so the models with the
fewest terms are the preferred models, which in this case are the reduced simple models.
Realistically, none of the models are useful. Not enough of the total variance in the data
is explained nor is the range of expected scores reduced to a useful level.
F. INTERVIEWS
Subject matter experts interviewed agree that while all of the factors in the survey
are important, additional factors could be added to more effectively group personnel and
provide more information about the maintenance organization. They also think that
dividing total years of aviation maintenance experience into two items, years worked in
MOS and years worked outside of MOS is necessary to clarify that Total years of
aviation maintenance experience is not simply time on active duty. The experts also
indicate factors that look at levels of education and training and levels of morale and
motivation are important and should be included in the demographic items of the survey.
Responses from subject matter experts are given in Appendix Y.
36
V. CONCLUSIONS
A. FINDINGS
The results of this thesis show that at the first level of interaction, the
demographic factors of the MCAS poorly account for the variance in the responses. The
models constructed using linear regression and analysis of variance do not capture the
responses of the surveyed population, showing that the demographic factors have low
utility in data analysis. While analysis of variance shows that the models are preferred to
no model at all, in use the models are too complex and do not provide enough insight into
the surveyed group.
Since the component scores are subjective perceptions, there is no correct score to
any of the perception items. The reliance on the human component in the scoring is the
cause of the large amount of variance, and since variance cannot be explained by the
demographic factors, the MCAS appears to be demographically unbiased. The three
dimensional bar plots of the component scores versus the levels of factors seem to
support this (see Figure 4). Either the MCAS has insignificant biasing across factors or
the present factors do not correctly group respondents to allow the biasing to be
conclusively measured.
Although the demographics do not effectively group respondents, they do provide
information about the demographic composition of the surveyed group. This information
alone can be useful to commanders in understanding the substance of their squadrons. To
make these items more useful, changes to the MCAS demographic items based on the
responses from subject matter experts are recommended below.
37
B. RECOMMENDATIONS
With the MCAS moving to the internet, it will be possible to use more
demographic factors than what is currently constrained by the layout of scannable
computer response sheets. The MCAS demographic items should be changed to include
the following items:
1) Check the box corresponding to your community:
2) Type in your unit number.
3) Type in the number of months have you been with your current squadron.
4) Type in the total number of maintenance activities to which you have been
assigned.
5) Type in the number of deployments you have made.
6) Check the box corresponding to your rank.
7) Type in the number years have you worked in your MOS.
8) Type in the number of years have you worked outside of your MOS.
9) Type in any supervisory designations that you hold.
. 10) Check the box corresponding to your work center.
11) Check the box corresponding to your shift.
12) Have you attended A School?
With these new questions, further analysis can be conducted to investigate for
valid factors that properly describe the data in the responses, in addition to investigating
if personal performance makes a difference in scoring. Additionally, MCAS could be
modified to gauge safety climates in other military activities such as military ordnance
38
handling facilities and flight deck operations by adjusting the demographic items to suit
those specific activities.
39
THIS PAGE INTENTIONALLY LEFT BLANK
APPENDIX A. 43-ITEM MAINTENANCE CLIMATE ASSESSMENT SURVEY
MAINTENANCE CLIMATE ASSESSMENT SURVEY (MCAS)
Purpose : The MCAS was designed to capture maintainer perceptions of maintenance operations as they
relate to safety. Your responses help guide Naval Aviation’s on-going efforts to reduce aviation related
mishaps. Thank you in advance for your participation !
Directions : Do not write on this form. Fill in all of your responses using the computer sheet provided.
Fill in each box that corresponds to your response completely using a pencil. This is not a timed event, so
answer each question carefully and honestly. Individual responses will not be reported, only compiled
results will be provided to each squadron.
Part I - Demographics has six items requesting unit and biographical data. This information will aid in the
response analysis. NO attempts will be made to identify individuals.
Part II - Perceptions has 43 questions pertaining to the maintenance operations. Please choose the response
to each item that most correctly reflects your honest opinion. Responses are:
A- Strongly Agree B- Agree C- Neutral D- Disagree E- Strongly Disagree
Part I- Demographics
Line 1 Fill in the numbered circle corresponding to your community?
(l)VMGR (2) VMA (3) VMFA (4) HMT (5) HMM
(6) VMAQ (7) HMH (8) VMH (9) Other
Line 2-4 Fill in the circles corresponding to your squadron number
Line 5 Fill in the numbered circle corresponding with you rank
(1) El-3 (2) E4-5 (3) E6-7 (4) E8-9 (5)W01-4 (6)01-03 (7)04-5
Line 6 Fill in the numbered circle corresponding to your total years
of Aviation Maintenance experience
(1) <1 (2) 1-2 (3)3-5 (4)6-10 (5) 11-15 (6) 15-20 (7) 204-
Line 7 Fill in the numbered circle corresponding to your work center
(1) Power Plants (2) Airframes (3) Survival (4) Quality Assurance
(5) Ordnance (6) Avionics (7) MAINT Control (8) Line (9) Other
Line 8 Fill in the numbered circle corresponding to your shift
(1) Day (2) Night
Part II Perceptions
Fill in the lettered circle that corresponds with your response to each item.
SA
A
N
D
SD
1 . The command adequately reviews and updates safety.
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
2. The command monitors maintainer qualifications and has
(A)
(B)
(C)
(D)
(E)
a program that targets training deficiencies.
SA
A
N
D
SD
3. The command uses safety and medical staff to identify/
(A)
(B)
(C)
(D)
(E)
41
manage personnel at risk.
4. CDIs/QARs routinely monitor maintenance evolutions.
5. Tool control is taken seriously in the command and
support equipment licensing is closely monitored.
6. Signing off personnel qualifications are taken seriously.
7. Our command climate promotes safe maintenance.
8. Supervisors discourage SOP, NAMP, or other procedural
violations and encourage reporting safety concerns.
9. Peer influence discourages SOP, NAMP, or other
violations and individuals feel free to report them.
10. Violations of SOP, NAMP, or other procedures are not
common in this command.
11. The command recognizes individual safety achievement
through rewards and incentives.
12. Personnel are comfortable approaching supervisors
about personal problems/illness
13. Safety NCO, QAR, and CDI, are sought after billets.
14. Unprofessional behavior is not tolerated in the command
15. The command has a reputation for quality maintenance
and sets standards to maintain quality control.
16. QA and Safety are well respected, and are seen as
essential to mission accomplishment.
17. QARs/CDIs sign-off after required actions are complete
and are not pressured by supervisors to sign-off.
18. Maintenance on detachments is of the same quality
as that at home station.
19. Required publications/tools/equipment are available,
current/serviceable, and used.
20. QARs are helpful, and QA is not “feared” in my unit.
21. Multiple job assignments and collateral duties adversely
affect maintenance.
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
42
22. Safety is part of maintenance planning, and additional
training/support is provided as needed.
23. Supervisors recognize unsafe conditions and manage
hazards associated with maintenance and the flight-line.
24. I am provided adequate resources, time, personnel to
accomplish my job.
25. Personnel turnover does not negatively impact the
command’s ability to operate safely.
26. Supervisors are more concerned with safe maintenance
than the flight schedule, and do no permit cutting corners.
27. Day/Night Check have equal workloads, and staffing is
sufficient on each shift.
28. Supervisors shield personnel from outside pressures and
are aware of individual workload.
29. Based upon my command’s current assets/manning it
is not over-committed.
30. My command temporarily restricts maintainers who are
having a problems.
31. Safety decisions are made at the proper levels and
work center supervisor decisions are respected.
32. Supervisors communicate command safety goals and
are actively engaged in the safety program.
33. Supervisors set the example for following to
maintenance standards and ensure compliance.
34. In my command safety is a key part of all maintenance
operations, and all are responsible/accountable for safety.
35. Safety education and training are comprehensive and
effective.
36. All maintenance evolutions are properly briefed,
supervised, and staffed by qualified personnel.
37. Maintenance Control is effective in managing all
maintenance activities.
38. Good communication exists up/down the chain of
command.
39. I get all the information I need to do my job safely.
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
43
40. Work center supervisors coordinate their actions.
41. My command has effective pass-down between shifts.
42. Maintenance Control troubleshoots/resolves gripes before
flight.
43. Maintainers are briefed on potential hazards associated
with maintenance activities.
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
SA
A
N
D
SD
(A)
(B)
(C)
(D)
(E)
44
APPENDIX B. MODEL OF SAFETY EFFECTIVENESS COMPONENTS.
COMPONENT 1: PROCESS AUDITING
1. The command adequately reviews and updates safety practices.
2. The command monitors maintainer qualifications and has a program that targets
training deficiencies.
3. The command uses safety and medical staff to identify/manage personnel at risk.
4. CDIs/QARs routinely monitor maintenance evolutions.
5. Tool Control is taken seriously in the command and support equipment licensing is
closely monitored.
6. Signing personal qualifications are taken seriously.
COMPONENT 2: Reward System and Safety Climate
1. Our command climate promotes safe maintenance and flight operations.
2. Supervisors discourage SOP, NAMP or other procedure violations and encourage
reporting safety concern.
3. Peer influence discourages SOP, NAMP or other violations and individuals feel free
to report them.
4. Violations of SOP, NAMP or other procedures are not common in this command.
5. The command recognizes individual safety achievement through rewards and
incentives.
6. Personnel are comfortable approaching supervisors about personal problems/illness.
7. Safety NCO, QAR, and CDI, are sought after billets.
8. Unprofessional behavior is not tolerated in the command.
COMPONENT 3: QUALITY ASSURANCE
1. The command has a reputation for quality maintenance and has set standards to
maintain quality control.
2. QA and Safety are well respected, and are seen as essential to mission
accomplishment.
3. QARs/CDIs sign-off after required actions are complete and are not pressured by
supervisors to sign-off.
4. Maintenance on detachments is the same quality as that at home station.
5. Required publications/tools/equipment are available, current/serviceable, and used.
6. QARs are helpful, and QA is not “feared” in my unit.
45
COMPONENT 4: RISK MANAGEMENT
1. Multiple job assignments and collateral duties adversely affect maintenance.
2. Safety is part of maintenance planning, and additional training/support is provided as
needed.
3. Supervisors recognize unsafe conditions and manage hazards associated with
maintenance and the flight line.
4. I am provided adequate resources, time, personnel to accomplish my job.
5. Personnel turnover does not negatively impact the command’s ability to operate
safely.
6. Supervisors are more concerned with safe maintenance than the flight schedule, and
do not permit cutting comers.
7. Day/Night Check have equal workloads, and staffing is sufficient on each shift.
8. Supervisors shield personnel from outside pressures and are aware of individual
workload.
9. Based upon my command’s current assets/manning it is not over-committed.
COMPONENT 5: COMMAND AND CONTROL
1. My command temporarily restricts maintainers who are having problems.
2. Safety decisions are made at the proper levels, work center supervisors decisions are
respected.
3. Supervisors communicate command safety goals and are actively engaged in the
safety program.
4. Supervisors set the example for following to maintenance standards and ensure
compliance.
5. In my command safety is a key part of all maintenance operations and all are
responsible/accountable for safety.
6. Safety education and training are comprehensive and effective.
7. All maintenance evolutions are properly briefed, supervised, and staffed by qualified
personnel.
8. Maintenance control is effective in managing all maintenance activities.
COMPONENT 6: COMMUNICATION / FUNCTIONAL RELATIONSHIPS
1. Good communication exists up/down the chain of command.
2. I get all the information I need to do my job safely.
3. Work center supervisors coordinate their actions.
4. My command has effective pass-down between shifts.
5. Maintenance Control troubleshoots/resolves gripes before flight.
6. Maintainers are briefed on potential hazards associated with maintenance activities.
46
THIS PAGE INTENTIONALLY LEFT BLANK
48
APPENDIX D. SIMPLE MODELS: SCATTERPLOTS WITH SUPERIMPOSED
REGRESSION LINE
PA
RS
Fitted : {lactor(Community) + lactor(Squadron) + factor(Rank) + }actor{Tot.Years) + factor(Shop) + factorfSh Fitted : (factor(Community) + (aclor(Squadron) + factor(Rank) + )actor(Tot.Years) ♦ tactor(Shop) + Jador(Sh
QA RM
Fitted : (faetor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Sh Fitted : (factor(Community) + tactor(Squadron) + factor(Rank) + tactor{Tol.Years) + factor(Shop) + factor(Sh
CC CR
Fitted: (factor(Community) + tactor(Squadron) + factor (Rank) + factor(Tot. Years) + factor(Shop) + tactor(Sh Fitted : (factor(Community) + (actor(Squadron) + factor(Rank) + lactor(Tot.Years) + factor(Shop) + factor(Sh'
49
THIS PAGE INTENTIONALLY LEFT BLANK
50
100 200 300 ^ 0 100 200 300
APPENDIX E. SIMPLE MODELS: HISTOGRAMS OF RESIDUALS
51
THIS PAGE INTENTIONALLY LEFT BLANK
52
THIS PAGE INTENTIONALLY LEFT BLANK
54
THIS PAGE INTENTIONALLY LEFT BLANK
56
APPENDIX H. SIMPLE MODELS: ANALYSIS OF VARIANCE
PA
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 8.3925 1.198929 3.922101 0.0003198
factor(Squadron) 13 8.0763 0.621257 2.032343 0.0160499
factor(Rank) 6 6.0359 1.005978 3.290893 0.0033082
factor(Tot.Years) 6 3.4403 0.573388 1.875746 0.0821892
factor(Shop) 9 3.2828 0.364753 1.193231 0.2957039
factor(Shift) 1 2.7153 2.715335 8.882780 0.0029608
Residuals 850 259.8325 0.305685
QA
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 8.3910 1.198710 2.994574 0.0041284
factor(Squadron) 13 14.3600 1.104618 2.759519 0.0007443
factor(Rank) 6 12.2469 2.041156 5.099143 0.0000372
factor(Tot.Years) 6 7.6351 1.272514 3.178950 0.0043234
factor(Shop) 9 3.9649 0.440543 1.100548 0.3596172
factor(Shift) 1 0.6428 0.642779 1.605767 0.2054349
Residuals 850 340.2498 0.400294
RS
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 3.7699 0.538562 1.549360 0.1471909
factor(Squadron) 13 12.7604 0.981569 2.823824 0.0005582
factor(Rank) 6 17.8319 2.971978 8.549925 0.0000000
factor(Tot.Years) 6 7.6649 1.277490 3.675142 0.0013048
factor(Shop) 9 5.4436 0.604846 1.740050 0.0761483
factor(Shift) 1 1.7291 1.729082 4.974302 0.0259869
Residuals 850 295.4624 0.347603
RM
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 15.9912 2.284458 7.056521 0.0000000
factor(Squadron) 13 11.8830 0.914080 2.823525 0.0005590
factor(Rank) 6 19.2158 3.202630 9.892685 0.0000000
factor(Tot.Years) 6 10.5421 1.757022 5.427311 0.0000161
factor(Shop) 9 4.8880 0.543110 1.677627 0.0901813
factor(Shift) 1 0.7475 0.747503 2.308980 0.1290003
Residuals 850 275.1766 0.323737
cc
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 9.8503 1.407189 3.490669 0.0010676
factor(Squadron) 13 12.2826 0.944816 2.343707 0.0045142
factor(Rank) 6 15.1026 2.517097 6.243904 0.0000020
factor(Tot.Years) 6 6.7176 1.119607 2.777295 0.0111256
factor(Shop) 9 3.5477 0.394188 0.977822 0.4567736
factor(Shift) 1 2.8159 2.815870 6.985039 0.0083707
Residuals 850 342.6594 0.403129
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 12.4423 1.777478 3.792880 0.0004599
factor(Squadron) 13 11.0908 0.853140 1.820477 0.0361248
factor(Rank) 6 18.6478 3.107964 6.631942 0.0000007
factor(Tot.Years) 6 9.4231 1.570522 3.351265 0.0028616
factor(Shop) 9 3.7036 0.411516 0.878115 0.5443707
factor(Shift) 1 0.7389 0.738887 1.576677 0.2095850
Residuals 850 398.3402 0.468636
57
THIS PAGE INTENTIONALLY LEFT BLANK
58
APPENDIX I.
TWO FACTOR INTERACTION MODELS: SCATTERPLOTS
WITH SUPERIMPOSED REGRESSION LINE
Fitted : (factor(Commumty) + tactor(Squadron) + factor(Rank) + factor(Tot.Years) + tactor(Shop) + factor{Shil Fitted : (factor(Community} + tactor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shrt
o oo> o o o o
o o
O O CD O O O
OO O CD O OOOO <
ODD 00 O 005 O O O
OO O COdXDCOO OODO .OO
o o ocomi
® o ooaa
O 00 OOOOJ
O O OODMiOU
o oo oo g^a—
o o o o &um
$ O ODD OCO
OO <SD OOOOO .O'
GD O OO O OOOO O
OOO® O CBCD QO O
00000)00 <WP OSOEMDO
o ,ooo«m)ooo®o o
o owijp OD QDO o to o o
00 O O 00000 OOODO o
00 35 ® 4 B
o ooo oooot
0^0 o <o <
Fitted : (factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) faetor($hh Fitted : (faetor(Community) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shop) + tactor(Shil
8 00 00 o o
O ODOC 3 X> O .0
o ottao o ao oca .o'
o o ® aw^nnaiBOaD 10
° oocdo°<^ 555 ^°^ o
O 0 OO II 111 —l|W I m o
O O tmaamn O 0 ,0'
OO O OOOo ®D
OOOO COCO OO O O O
O MO O © OO 0,0 '
od a>8 8 do©o acoooo
ODD 0.0 anODOOD ODD CD
o oo o as o o
oonomo oaoo o
OOOO OOO OCCD o
CO O CD O
O COCOO OODCO OO ooo o o
5 COB (DO rrri CD CO O O
0 % OO &BDOOB> OOO
Ooo 0 CD O O 0 O
Fitted : (tactor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shi) Fitted: (factor(Community) + factor(Squadron) + factor(Rank) + 1actor{Tot.Years) + factor(Shop) *■ Jactor(Shil
THIS PAGE INTENTIONALLY LEFT BLANK
60
100 300 0 100 300 0 50 100 200
APPENDIX J. TWO FACTOR INTERACTION MODELS: HISTOGRAMS OF
RESIDUALS
PA QA
-3 -2 -10 12 - 2-1012
resid(cc.2.aov) resid(cr.2.aov)
61
THIS PAGE INTENTIONALLY LEFT BLANK
62
APPENDIX K. TWO FACTOR INTERACTION MODELS: RESIDUALS
VERSUS FITTED VALUES WITH LOESS SMOOTHING
QA
THIS PAGE INTENTIONALLY LEFT BLANK
64
THIS PAGE INTENTIONALLY LEFT BLANK
66
APPENDIX M. TWO FACTOR INTERACTION MODELS: ANALYSIS OF
VARIANCE
PA
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 8.3925 1.198929 4.356653 0.0001027
factor(Squadron) 13 8.0763 0.621257 2.257518 0.0068172
factor(Rank) 6 6.0359 1.005978 3.655510 0.0014394
factor(Tot.Years) 6 3.4403 0.573388 2.083571 0.0534953
factor(Shop) 9 3.2828 0.364753 1.325436 0.2203764
factor(Shift) 1 2.7153 2.715335 9.866954 0.0017732
factor(Community):factor(Rank) 27 8.5470 0.316557 1.150301 0.2755360
factor(Community):factor(Tot.Years) 32 5.5049 0.172029 0.625117 0.9480617
factor(Community):factor(Shop) 46 16.8120 0.365478 1.328069 0.0778513
factor(Community):factor(Shift) 5 4.8802 0.976038 3.546715 0.0036412
factor(Squadron):factor(Rank) 27 7.1875 0.266205 0.967334 0.5134968
factor(Squadron);factor(Tot.Years) 37 16.5409 0.447050 1.624486 0.0126771
factor(Squadron):factor(Shop) 56 23.4663 0.419042 1.522709 0.0108588
factor(Squadron):factor(Shift) 6 3.7496 0.624928 2.270855 0.0356434
factor(Rank):factor(Tot.Years) 8 3.4152 0.426899 1.551259 0.1367135
factor(Rank):factor(Shop) 15 5.1111 0.340743 1.238189 0.2382294
factor(Rank):factor(Shift) 2 0.4713 0.235625 0.856213 0.4253331
factor(Tot.Years):factor(Shop) 29 8.8454 0.305014 1.108357 0,3199687
factor(Tot.Years):factor(Shift) 3 0.7165 0.238846 0.867915 0.4574968
factor(Shop):factor(Shift) 7 3.2273 0.461047 1.675347 0.1124277
Residuals 550 151.3572 0.275195
QA
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 8.3910 1.198710 3.292186 0.0019394
factor(Squadron) 13 14.3600 1.104618 3.033770 0.0002431
factor(Rank) 6 12.2469 2.041156 5.605916 0.0000116
factor(Tot.Years) 6 7.6351 1.272514 3.494886 0.0021165
factor(Shop) 9 3.9649 0.440543 1.209925 0.2860380
factor(Shift) 1 0.6428 0.642779 1.765354 0.1845090
factor(Community):factor(Rank) 27 10.0763 0.373196 1.024962 0.4313503
factor(Community):factor(Tot.Years) 32 10.2932 0.321663 0.883428 0.6538426
factor(Community):factor(Shop) 46 23.9700 0.521087 1.431136 0.0362521
factor(Community):factor(Shift) 5 11.2388 2.247751 6.173318 0.0000140
factor(Squadron):factor(Rank) 27 17.7225 0.656389 1.802734 0.0083195
factor(Squadron):factor(Tot.Years) 37 15.4891 0,418624 1.149727 0.2540719
factor(Squadron):factor(Shop) 56 22.7429 0.406123 1.115393 0.2703154
factor(Squadron):factor(Shift) 6 3.5604 0.593400 1.629737 0.1365990
factor(Rank):factor(Tot.Years) 8 5.2739 0.659237 1.810557 0.0725066
factor(Rank):factor(Shop) 15 7.4882 0.499213 1.371060 0.1561216
factor(Rank):factor(Shift) 2 1.7580 0.879015 2.414162 0.0903895
factor(Tot.Years):factor(Shop) 29 7.0553 0.243286 0.668170 0.9075797
factor(Tot.Years):factor(Shift) 3 0.8364 0.278789 0.765679 0.5136070
factor(Shop):factor(Shift) 7 2.4857 0.355105 0.975276 0.4483208
Residuals 550 200.2591 0.364107
RS
Df Sum of Sq Mean Sq F Value Pr(F)
factor(Community) 7 3.7699 0.538562 1.734800 0.0984123
factor(Squadron) 13 12.7604 0.981569 3.161802 0.0001365
factor(Rank) 6 17.8319 2.971978 9.573249 0.0000000
factor(Tot.Years) 6 7.6649 1.277490 4.115013 0.0004719
factor(Shop) 9 5.4436 0.604846 1.948314 0.0432207
factor(Shift) 1 1.7291 1.729082 5.569667 0.0186225
factor(Community):factor(Rank) 27 8.6010 0.318554 1.026117 0.4297560
factor(Community):factor(Tot.Years) 32 5.9900 0.187188 0.602964 0.9597519
factor(Community):factor(Shop) 46 19.4596 0.423035 1.362667 0.0607806
factor(Community):factor(Shift) 5 6.8711 1.374220 4.426596 0.0005820
factor(Squadron):factor(Rank) 27 12.1684 0.450680 1.451719 0.0672413
factor(Squadron):factor(Tot.Years) 37 14.7707 0.399208 1.285918 0.1242051
factor(Squadron):factor(Shop) 56 26.1193 0.466416 1.502405 0.0132179
factor(Squadron):factor(Shift) 6 4.7507 0.791781 2.550461 0.0191331
factor(Rank):factor(Tot.Years) 8 3.5565 0.444564 1.432017 0.1800429
factor(Rank):factor(Shop) 15 4.1334 0.275561 0.887628 0.5783215
factor(Rank):factor(Shift) 2 1.6918 0.845879 2.724720 0.0664497
factor(Tot.Years):factor(Shop) 29 10.6388 0.366855 1.181702 0.2373524
factor(Tot.Years):factor(Shift) 3 2.3787 0.792884 2.554013 0.0546501
factor(Shop):factor(Shift) 7 3.5872 0.512457 1.650712 0.1187383
Residuals 550 170.7454 0.310446
67
RM
Df
factor(Community) 7
factor(Squadron) 13
factor(Rank) 6
factor(Tot.Years) 6
factor(Shop) 9
factor(Shift) 1
factor(Community):factor(Rank) 27
factor(Community):factor(Tot.Years) 32
factor(Community):factor(Shop) 46
factor(Community):factor(Shift) 5
factor(Squadron):factor(Rank) 27
factor(Squadron):factor(Tot.Years) 37
factor(Squadron):factor(Shop) 56
factor(Squadron):factor(Shift) 6
factor(Rank):factor(Tot.Years) 8
factor(Rank):factor(Shop) 15
factor(Rank):factor(Shift) 2
factor(Tot.Years):factor(Shop) 29
factor(Tot.Years):factor(Shift) 3
factor(Shot):factor(Shift) 7
Residuals 550
CR
Df
factor(Community) 7
factor(Squadron) 13
factor(Rank) 6
factor(Tot.Years) 6
factor(Shop) 9
factor(Shift) 1
factor(Community):factor(Rank) 27
factor(Community):factor(Tot.Years) 32
factor(Community):factor(Shop) 46
factor(Community):factor(Shift) 5
factor(Squadron):factor(Rank) 27
factor(Squadron):factor(Tot.Years) 37
factor(Squadron):factor(Shop) 56
factor(Squadron):factor(Shift) 6
factor(Rank):factor(Tot.Years) 8
factor(Rank):factor(Shop) 15
factor(Rank):factor(Shift) 2
factor(Tot.Years):factor(Shop) 29
factor(Tot.Years):factor(Shift) 3
factor(Shop):factor(Shift) 7
Residuals 550
cc
Df
factor(Community) 7
factor(Squadron) 13
factor(Rank) 6
factor(Tot.Years) 6
factor(Shop) 9
factor(Shift) 1
factor(Community):factor(Rank) 27
factor(Community):factor(Tot.Years) 32
factor(Community):factor(Shop) 46
factor(Community):factor(Shift) 5
factor(Squadron):factor(Rank) 27
factor(Squadron):factor(Tot.Years) 37
factor(Squadron):factor(Shop) 56
factor(Squadron):factor(Shift) 6
factor(Rank):factor(Tot.Years) 8
factor(Rank):factor(Shop) 15
factor(Rank):factor(Shift) 2
factor(Tot.Years):factor(Shop) 29
factor(Tot.Years):factor(Shift) 3
factor(Shop):factor(Shift) 7
Residuals 550
Sum of Sq Mean Sq F Value Pr(F)
15.9912 2.284458 7.74590 0.0000000
11.8830 0.914080 3.09937 0.0001810
19.2158 3.202630 10.85914 0.0000000
10.5421 1.757022 5.95752 0.0000048
4.8880 0.543110 1.84152 0.0583837
0.7475 0.747503 2.53455 0.1119530
6.1172 0.226562 0.76820 0.7946819
7.0593 0.220602 0.74799 0.8421223
15.9480 0.346696 1.17554 0.2054702
12.6939 2.538785 8.60824 0.0000001
10.8087 0.400322 1.35737 0.1093488
12.4608 0.336778 1.14191 0.2636165
20.8355 0.372062 1.26155 0.1036550
5.6544 0.942395 3.19537 0.0043116
2.4103 0.301291 1.02158 0.4183172
4.4618 0.297452 1.00857 0.4442584
2.6308 1.315404 4.46013 0.0119820
8.4000 0.289655 0.98213 0.4942530
1.2055 0.401827 1.36247 0.2533738
2.2818 0.325976 1.10528 0.3581240
162.2087 0.294925
Sum of Sq Mean Sq F Value Pr(F)
12.4423 1.777478 4.177394 0.0001697
11.0908 0.853140 2.005033 0.0185347
18.6478 3.107964 7.304274 0.0000002
9.4231 1.570522 3.691009 0.0013214
3.7036 0.411516 0.967136 0.4662855
0.7389 0.738887 1.736518 0.1881293
10.0529 0.372329 0.875041 0.6494108
10.0165 0.313016 0.735645 0.8559696
30.3247 0.659233 1.549317 0.0137593
8.1375 1.627495 3.824905 0.0020492
14.7948 0.547957 1.287797 0.1526769
19.5299 0.527835 1.240507 0.1600404
36.1636 0.645778 1.517696 0.0114023
3.4718 0.578626 1.359876 0.2288249
4.5586 0.569829 1.339200 0.2212122
6.5391 0.435941 1.024540 0.4275257
1.2651 0.632559 1.486627 0.2270415
14.0630 0.484930 1.139672 0.2827731
2.2102 0.736717 1.731418 0.1594515
3.1879 0.455411 1.070298 0.3811256
234.0246 0.425499
Sum of Sq Mean Sq F Value Pr(F)
9.8503 1.407189 3.913888 0.0003535
12.2826 0.944816 2.627865 0.0014467
15.1026 2.517097 7.000932 0.0000003
6.7176 1.119607 3.114022 0.0052210
3.5477 0.394188 1.096376 0.3634308
2.8159 2.815870 7.831926 0.0053132
8.8346 0.327206 0.910074 0.5979349
8.7848 0.274526 0.763554 0.8237228
20.7928 0.452016 1.257217 0.1253496
10.1673 2.033454 5.655750 0.0000426
11.6354 0.430941 1.198599 0.2264270
17.7331 0.479274 1.333030 0.0940917
27.8577 0.497460 1.383610 0.0390184
5.1754 0.862573 2.399120 0.0268520
5.4297 0.678707 1.887723 0.0595526
6.9073 0.460485 1.280770 0.2089887
1.7807 0.890356 2.476393 0.0849826
14.6705 0.505881 1.407033 0.0788824
2.0737 0.691223 1.922534 0.1248178
3.0709 0.438695 1.220166 0.2894603
197.7456 0.359537
68
APPENDIX N. REDUCED SIMPLE MODELS
Model:
PA - factor(Squadron) + factor(Tot.Years) + factor(Shift)
scale: 0.3056853
Sum of Sq RSS Cp
265.0655 281.5725
19.09854 284.1640 289.0550
6.98011 272.0456 284.8844
2.98713 268.0526 283.9482
Df
<none>
factor(Squadron) 19
factor(Tot.Years) 6
factor(Shift) 1
Terms:
Sum of Squares
Deg. of Freedom
factor(Squadron)
16.4369
19
factor(Tot.Years)
7.2861
6
factor(Shift) Residuals
2.9871 265.0655
1 866
Residual standard error: 0.5532452
Estimated effects may be unbalanced
QA ~ factor(Squadron) + factor(Tot.Years) + factor(Shift)
scale: 0.4002939
Df
<none>
factor(Squadron) 19
factor(Tot.Years) 6
factor(Shift) 1
Sum of Sq RSS Cp
348.6818 370.2977
23.06513 371.7469 378.1516
15.45358 364.1354 380.9477
0.90043 349.5822 370.3975
Terms:
factor(Squadron)
Sum of Squares 22.4130
Deg. of Freedom 19
factor(Tot.Years)
15.4952
6
factor(Shift) Residuals
0.9004 348.6818
1 866
Residual standard error: 0.6345352
Estimated effects may be unbalanced
Model:
RS ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shift)
scale: 0.3476029
Df
<none>
factor(Squadron) 19
factor(Rank) 6
factor(Tot.Years) 6
factor(Shift) 1
Sum of Sq RSS Cp
301.9366 324.8784
16.56125 318.4979 328.2308
5.50006 307.4367 326.2072
7.70716 309.6438 328.4143
1.04953 302.9861 325.2327
Terms:
factor(Squadron)
Sum of Squares 16.0094
Deg. of Freedom 19
factor(Rank)
18.0278
6
factor(Tot.Years)
7.6388
6
factor(Shift) Residuals
1.0495 301.9366
1 860
Residual standard error: 0.5925277
Estimated effects may be unbalanced
Model:
RM - factor(Squadron) + factor(Rank) + factor(Tot.Years)
scale: 0.3237372
<none>
Df Sum of Sq RSS Cp
280.8129 301.5321
69
factor(Squadron) 19 23.32159 304.1345 312.5517
factor(Rank) 6 5.07801 285.8909 302.7253
factor(Tot.Years) 6 10.54164 291.3546 308.1889
Terms:
Sum of Squares
Deg. of Freedom
factor{Squadron)
27.8628
19
factor(Rank)
19.2269
6
factor(Tot.Years)
10.5416
6
Residuals
280.8129
861
Residual standard error: 0.5710932
Estimated effects may be unbalanced
Model:
CC - factor(Squadron) + factor(Tot.Years) + factor(Shift)
scale: 0.4031287
Df Sum of Sq RSS Cp
<none> 350.8909 372.6598
factor(Squadron) 19 22.16263 373.0535 379.5036
factor(Tot.Years) 6 17.43463 368.3255 385.2569
factor(Shift) 1 2.80445 353.6953 374.6580
Terms:
Sum of Squares
Deg. of Freedom
factor(Squadron)
22.0953
19
factor(Tot.Years)
17.1855
6
factor(Shift)
2.8045
1
Residuals
350.8909
866
Residual standard error: 0.636542
Estimated effects may be unbalanced
Model:
CR ~ factor(Community) + factor(Tot.Years) + factor(Shift)
scale: 0.4686355
Df Sum of Sq RSS Cp
<none> 418.7697 432.8288
factor(Community) 7 10.24208 429.0118 436.5100
factor(Tot.Years) 6 21.33850 440.1082 448.5437
factor(Shift) 1 1.89642 420.6661 433.7879
Terms:
Sum of Squares
Deg. of Freedom
factor(Community)
12.4423
7
factor(Tot.Years)
21.2783
6
factor(Shift)
1.8964
1
Residuals
418.7697
878
Residual standard error: 0.690622
Estimated effects may be unbalanced
70
APPENDIX O. REDUCED SIMPLE MODELS: SCATTERPLOTS WITH
SUPERIMPOSED REGRESSION LINE
Fitted : factor(Squadron) factor{Tot.Years) + tactor(Shitt)
Fitted : fador(Community) + factor(Tot.Years) + factor{Shift)
THIS PAGE INTENTIONALLY LEFT BLANK
72
100 200 300 0 100 200 300 0 100 200 300
APPENDIX P. REDUCED SIMPLE MODELS: HISTOGRAMS OF RESIDUALS
73
THIS PAGE INTENTIONALLY LEFT BLANK
74
APPENDIX O. REDUCED SIMPLE MODELS: RESIDUALS VERSUS FITTED
VALUES
3.4 3.6 3.8 4.0 4.2 4.4
Fitted: lactor(Squadron) + factorfTot. Years) + factor(Shift)
3.0 3.2 3.4 3.6 3.8 4,0 4.2
Fitted: factor(Squadron) + faetor(Rank) + factor(Tot.Years) + factor(Shift)
cc
3.2 3.4 3.6 3.8 4.0 4.2
Fitted : faetor(Squadron) + factor(Tol.Years) + factor(Shift)
3.2 3.4 3.6 3.8 4.0
Fitted : factor(Community) + factorfTot.Years) + factor(Shift)
75
THIS PAGE INTENTIONALLY LEFT BLANK
76
APPENDIX R. REDUCED SIMPLE MODELS: QQ-PLOTS
of Standard Normal
Quantiles of Standard Normal
THIS PAGE INTENTIONALLY LEFT BLANK
78
APPENDIX S. REDUCED TWO FACTOR INTERACTION MODELS
Model:
PA - factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) +
factor(Community):factor(Shift)
scale: 0.2751949
Df
<none>
factor(Squadron) 12
factor(Tot.Years) 6
factor(Community):factor(Shift) 5
Sum of Sq RSS
259.7296
8.235337 267.9649
7.737269 267.4669
5.319236 265.0488
Cp
277.8925
279.5231
282.3274
280.4597
Model:
QA - factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) +
factor(Community):factor(Shift)
scale: 0.3641075
<none>
factor(Squadron)
factor(Tot.Years)
factor(Community):factor(Shift)
Df Sum of Sq RSS Cp
339.8904 363.9215
12 12.36975 352.2601 367.5526
6 16.34776 356.2381 375.8999
5 8.42356 348.3139 368.7039
Model:
RS - factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) +
factor(Squadron):factor(Shift) + factor(Tot.Years):factor(Shift)
scale: 0.3104461
<none>
factor(Rank)
factor(Shop)
factor(Squadron):factor(Shift)
factor(Tot.Years):factor(Shift)
Df Sum of Sq RSS
280.7776
6 5.38955 286.1672
9 6.45093 287.2286
12 11.55724 292.3349
5 4.31919 285.0968
Cp
317.4103
319.0745
318.2732
321.5168
318.6250
Model:
RM - factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) +
factor(Squadron):factor(Shift) + factor(Tot.Years):factor(Shift)
scale: 0.2949249
<none>
factor(Rank)
factor(Shop)
factor(Squadron):factor(Shift)
factor(Tot.Years):factor(Shift)
Df Sum of Sq RSS
255.4226
6 3.99087 259.4134
9 5.55127 260.9738
12 17.43743 272.8600
5 3.57653 258.9991
Cp
290.2237
290.6755
290.4663
300.5829
290.8510
Model:
CC - factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) +
factor(Shift) + factor(Community):factor(Shift) + factor(Squadron):factor(Shift) +
factor(Rank):factor(Shift) + factor(Tot.Years):factor(Shift)
scale: 0.3595374
<none>
factor(Community):factor(Shift)
factor(Squadron):factor(Shift)
factor(Rank):factor(Shift)
factor(Tot.Years):factor(Shift)
Df Sum of Sq RSS
324.3285
0 0.000000 324.3285
7 5.543296 329.8718
5 4.227122 328.5556
5 5.275932 329.6045
Cp
364.5967
364.5967
365.1065
365.2285
366.2773
79
Model:
CR ~ factor(Community) + factor(Rank) + factor(Tot.Years) + factor(Shift) +
factor(Community):factor(Shift) + factor(Rank):factor(Shift) + factor(Tot.Years)
factor(Shift)
scale: 0.4254993
<none>
factor(Community):factor(Shift)
factor(Rank):factor(Shift)
factor(Tot.Years):factor(Shift)
Df Sum of Sq
6 9.058145
5 7.047982
5 7.483000
RSS Cp
393.4122 424.8992
402.4703 428.8513
400.4602 427.6921
400.8952 428.1272
80
APPENDIX T. REDUCED TWO FACTOR INTERACTION MODELS:
SCATTERPLOTS WITH SUPERIMPOSED REGRESSION
LINE
PA
QA
Fitted : factor(Community) + factor(Squadron) + 1actor(Tot. Years) + lactor(Shift) + factor(Community):1actor(S Fitted : (actor(Community) + factor(Squadron) + factor{Tot.Years) + factor(Shift) + 1actor(Community):<aetor(S
RS RM
iadron) + tactor(Rank) + factor(Tot.Years) + factor(Shop) + faetor(Shitt) + faclor{Squadron):lac1or(Shitt) + faclonadron) + factor(Rank) + faetor(Tot. Years) + factor(Shop) + factor(Shift) + iaclor(Squadron):factor(Shift) + factor
cc
CR
lity) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shitt) + factor(Community):factor(Shift) + faoity) + factor(Rank) + factor(Tot.Years) + factor(Shift) + factor{Community):factor(Shift) + factor(Rank):factor(Sh
81
THIS PAGE INTENTIONALLY LEFT BLANK
82
APPENDIX U. REDUCED TWO FACTOR INTERACTION MODELS
HISTOGRAMS OF RESIDUALS
THIS PAGE INTENTIONALLY LEFT BLANK
84
APPENDIX V. REDUCED TWO FACTOR INTERACTION MODELS:
RESIDUALS VERSUS FITTED VALUES
PA QA
2.0 2.5 3.0 3.5 4.0 4.S 1 2 3 4
Fitted : factor(Community) + faetor(Squadron) + factor(Tot.Years) + lactor(Shift) + (aclor(Conimunity):»actor{S Fitted : factor(Comrminity) + factor(Squadron) + factor(Tot.Years) + factor(Shrft) + factor(Community):faetor(S
RS RM
2 3 4 5 1 2 3 4
ladron) + factor(Rank) + tactor(Tot.Years) + tactor(Shop) + factor(Shift) + factor(Squadron):factor(Shrft) + factonadron) + factor(Rank) + (actor{Tot.Years) + factor(Shop) + factor(Shift) + factor(Squadron):factor(Shift) + factor
CC CR
itty) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shitl) + factor{Community):factor(Shift) +faoity) + faetor(Rank) + factor(Tot.Years) + factor(Shift) + factor(Community):tactor(Shift) * factor(Rank):factor(Sh
85
THIS PAGE INTENTIONALLY LEFT BLANK
86
THIS PAGE INTENTIONALLY LEFT BLANK
88
APPENDIX X. COMPARISONS ON MODELS USING ANOVA
The values listed in the following tables are the probabilities that the models are
equivalent. Values printed in bold type indicate similar models.
Component
Simple vs
Two Factor
Interaction
Simple vs.
Reduced
Simple
Simple vs.
Reduced Two
Factor
Interaction
PA
0.00316
0.37994
0.99999
QA
0.00657
0.17934
0.99990
RS
0.00174
0.04694
0.00029
RM
0.00730
0.09862
1.86442e-7
CC
0.00156
0.20517
0.00001
CR
0.00580
0.03360
0.10600
Component
Two Factor
Interaction
vs. Reduced
Simple
Two Factor
Interaction
vs. Reducd
Two Factor
Interaction
PA
0.00324
0.00788
QA
0.00488
0.01596
RS
0.00087
0.01474
RM
0.00470
0.14662
CC
0.00121
0.02202
CR
0.00200
0.02111
Component
Reduced
Simple vs.
Reduced Two
Factor
Interaction
PA
0.00752
QA
0.00120
RS
0.00011
RM
3:50999e-7
CC
0.00008
CR
0.00016
89
THIS PAGE INTENTIONALLY LEFT BLANK
90
APPENDIX Y. SUBJECT MATTER EXPERT RECOMMMENDED CHANGES
TO MCAS DEMOGRAPHIC FACTORS
Factors currently included in MCAS (no change):
1. Community
2. Squadron
3. Rank
4. Work Center
5. Shift
Items to modify in MCAS: *
Total years of aviation maintenance experience (single item)
modified to:
6. Years worked in MOS
7. Years worked outside MOS
Factors to add to MCAS:
education/training level
8. attended an A school
9. highest level of education attained
10. number of maintenance activities assigned to during career
morale/motivation indicators
11. command advanced
12. number of personal awards
13. level of job satisfaction
14. time to attain qualifications
15. past performance on personal evaluations
16. assigned to B tour
other factors
17. Age
18. number of months in current squadron
19. number of deployments
20. supervisory designations earned
21. level of confidence in Maintenance Control Officer
91
THIS PAGE INTENTIONALLY LEFT BLANK
92
LIST OF REFERENCES
Allen, J., Rankin, B. & Sargent, B. (1998). Human Factors Process for Reducing
Maintenance Errors [On-line]. Available:
http://www.boeing.coni/commerciaI/aeromagazine/aero 03/textonlv/m01 txt.html
Baker, R.H. (1998). Climate Survey Analysis for Aviation Maintenance Safety . Master’s
Thesis, Naval Postgraduate School, Monterey, California.
Brown, F.G. (1976). Principles of Educational and Psychological Testing . Holt, Rinehart
and Winston.
Booth, R. (1993). Safety Culture: Concept, Measurement and Training Implications.
Proceedings of British Health and Safety Society Spring Conference: Safety Culture and
the Management of Risk .
Ciavarelli, A.P. (Nov 1997). Organizational Safety Effectiveness Model [On-line].
Available: http://web.nps.navv.mil/~avsafety/research/qmbmodel.htm
Ciavarelli, A.P. & Figlock, R.C. (Nov 1997). Organizational Factors in Naval Aviation
Accidents: Command Safety Assessment [On-line], Available:
http://web.nps.navv.mil/~avsafetv/research/Qmbrept.htm
Federal Aviation Administration Strategic Plan, 1998.
Figlock, R. C. (1998). Qualitative Evaluation of a Notional Model of Organizational
Safety Effectiveness and a Prototype Assessment Questionnaire . Unpublished Doctoral
dissertation.
Goodrum, B.W. (1999). Assessment of Maintenance Safety Climate in U.S. Navy Fleet
Logistics Support Wing Squadrons . Masters Thesis, Naval Postgraduate School,
Monterey, California.
History [On-line], (April 2000). Available: http://www.faa.gov/historv.htm
Human Factors in Aviation Maintenance and Inspection [On-line], (April 2000)
Available: http://hfskyway.faa,gov/
Interview between C. Harris, Lieutenant, USN, Naval Postgraduate School, Monterey,
California, and the author, 14 March 2000.
Interview between J. Schmidt, Commander, MSC, USN, School of Aviation Safety,
Naval Postgraduate School, Monterey, California, and the author, 17 December 1999.
93
Komamiski, A. (April 2000). Human Factors in Aircraft Maintenance. Microsoft
Powerpoint presentation presented at the Advances in Aviation Safety Conference and
Exposition, Daytona Beach, Florida.
Libuser, C. B. (1994). Organizational structure and risk mitigation. Unpublished doctoral
dissertation. University of California, Los Angeles.
Maintenance and Inspection Human Factors Program [On-line]. (April 2000). Available:
http://hfskyway.faa.gov/researchv3.htm
Naval Safety Center Brief, Fiscal Year 1997.
Naval Safety Center Brief, Fiscal Year 2000.
Nutwell, R. & Sherman, K. (1997, March-April). Safety: Changing the Way We Operate.
Naval Aviation News , 12-15.
Oneto, T. J. (1999). Safety Climate Assessment in Naval Reserve Aviation Maintenance
Operations. Master’s Thesis, Naval Postgraduate School, Monterey, California.
OPNAV INSTRUCTION 3750.60, The Naval Aviation Safety Program. Department of
the Navy, 1989.
Perrow, C. (1984). Normal Accidents. New York: Basic Books.
Reason, J. (1990). Human Error . Cambridge: University Press.
Reason, J. (1997). Managing the Risks of Organizational Accidents . Brookfield: Ashgate
Publishing Company.
Redmill, F. & Rajan, J. (1997). Human Factors in Safety-critical Systems . Oxford:
Butterworth-Heinemann.
Roberts, K.H. (1990). Some characteristics of one type of high reliability organization.
Organizational Science . 1(2), 160-176.
Roberts, K.H. (1990). Managing high-reliability organizations. California Management
Review . 32(4), 101-113.
Safety, Security and System Efficiency [On-line]. (April 2000). Available:
http://www.faa. gov/safetyinfo2. htm
94
Schmidt, J., Schmorrow, D., & Hardee, M. (1998). A Preliminary Human Factors
Analysis of Naval Aviation Maintenance Related Mishaps. Proceedings of the Society of
Automotive Engineers, Airframes. Engines, Maintenance, and Repair Conference . (Paper
Number 983 111.)
Schmorrow, D. D. (1998). Human Error Analysis and Model of Aviation Maintenance
Related Mishaps , Master’s Thesis, Naval Postgraduate School, Monterey, California.
United States General Accounting Office (1993). Program Evaluation and Methodology
Division. Developing and Using Questionnaires.
Uttal, B. The corporate culture vultures. Fortune , 17 October 1983.
Weick, Karl E. (1987). Organizational culture as a source of high reliability. California
Management Review . 29(2), 112-127.
Wogalter, M., Allison, S. & McKenna, N. (1989). Effects of cost and social influence on
warning compliance. Human Factors . 31(2), 133-140.
Wickens, C. D., Gordon, S. E. & Lui, Y. (1997). An Introduction to Human Factors
Engineering . New York: Longman, Inc.
95
THIS PAGE INTENTIONALLY LEFT BLANK
96
INITIAL DISTRIBUTION LIST
1. Defense Technical Information Center.2
8725 John J. Kingman Rd„ STE 0944
Ft. Bel voir, Virginia 22060-6218
2. Dudley Knox Library.2
Naval Postgraduate School
411 Dyer Rd.
Monterey, California 93943-5101
3. Defense Logistic Studies Information Exchange..1
U.S. Army Logistics Management College
Fort Lee, Virginia 23801-6043
4. Professor Robert R. Read, Code OR/Re.2
Department of Operations Research
Naval Postgraduate School
Monterey, California 93943
5. CDR John K. Schmidt.3
School of Aviation Safety
Naval Postgraduate School
Monterey, California 93943
6. Professor Lyn R. Whitaker, Code OR/Wh.1
Department of Operations Research
Naval Postgraduate School
Monterey, California 93943
7. LT Bruce R. Stanley, Jr.2
3753 Stanford Drive
Oceanside, California 92056
97
Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Books to Borrow
Open Library
TV News
Understanding 9/11