Skip to main content

Full text of "DTIC ADA380379: Evaluating Demographic Item Relationships With Survey Responses on the Maintenance Climate Assessment Survey (MCAS)"

See other formats


NAVAL POSTGRADUATE SCHOOL 
Monterey, California 



THESIS 


EVALUATING DEMOGRAPHIC ITEM RELATIONSHIPS 
WITH SURVEY RESPONSES ON THE MAINTENANCE 
CLIMATE ASSESSMENT SURVEY (MCAS) 


Bruce R. Stanley, Jr. 


June 2000 


Thesis Advisor: 
Thesis Co-Advisor: 
Second Reader: 


John K. Schmidt 
Robert R. Read 
Lyn R. Whitaker 



Approved for public release; distribution is unlimited. 


20000811 033 



REPORT DOCUMENTATION PAGE 

Form Approved 

OMB No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching 
existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden 
estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for 
Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, 

Paperwork Reduction Project (0704-0188) Washington DC 20503. 

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 

June 2000 Master’s Thesis 

4. TITLE AND SUBTITLE 

Evaluating Demographic Item Relationships with Survey Responses on the 
Maintenance Climate Assessment Survey (MCAS) 

5. FUNDING NUMBERS 

6. AUTHOR(S) 

Stanley, Bruce R. Jr. 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Naval Postgraduate School 

Monterey, CA 93943-5000 

8. PERFORMING ORGANIZATION 
REPORT NUMBER 

9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

Office of Naval Research 

10. SPONSORING / MONITORING 

AGENCY REPORT NUMBER 

II. SUPPLEMENTARY NOTES 

The views expressed in this thesis are those of the author and do not reflect the official policy or position of 
the Department of Defense or the U.S. Government. 

12a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited. 

12b. DISTRIBUTION CODE 

ABSTRACT (maximum 200 words) 

The Maintenance Climate Assessment Survey (MCAS) was developed to proactively assess factors that contribute to a high 
reliability organization and strong safety climate. The 3 rd Marine Air Wing (MAW), which was seeking to proactively improve its 
safety posture requested the assistance of the School of Aviation Safety at the Naval Postgraduate School to examine its safety 
climate. Previous studies of the MCAS instrument have focused on the items and their relationship to the HRO based model of 
safety effectiveness components: process auditing, reward system, quality assurance, risk management, command and control, and 
communication/functional relationships. The present effort is the first attempt to consider the relationship between item component 
responses and demographic item responses. It evaluates 893 maintainer responses to the MCAS from 3 rd MAW and looks for 
measurable effects due to demographics. This study finds that the regression models constructed using the demographics as 
explanatory variables have very little utility in predicting scores for the components. This result allows planners the relief of using 
the demographics as a low priority issue. 

14. SUBJECT TERMS 

Human Factors, Human Error, Accident Classification, High Reliability Organizations, 
Corporate Safety Culture, Naval Aviation 

15. NUMBER OF PAGES 

116 

16. PRICE CODE 

17. SECURITY 

CLASSIFICATION OF REPORT 

Unclassified 

18. SECURITY CLASSIFICATION OF 
THIS PAGE 

Unclassified 

19. SECURITY CLASSIFI-CATION OF 
ABSTRACT 

Unclassified 

20. LIMITATION OF 
ABSTRACT 

UL 


NSN 7540-01-280-5500 


Standard Form 298 (Rev. 2-89) 
Prescribed by ANSI Std. 239-18 


1 
























THIS PAGE INTENTIONALLY LEFT BLANK 




Approved for public release; distribution is unlimited 


EVALUATING DEMOGRAPHIC ITEM RELATIONSHIPS WITH SURVEY 
RESPONSES ON THE MAINTENANCE CLIMATE ASSESSMENT SURVEY 

(MCAS) 


Bruce R. Stanley, Jr. 
Lieutenant, United States Navy 
B.S., United States Naval Academy, 1993 

Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN OPERATIONS RESEARCH 

from the 

NAVAL POSTGRADUATE SCHOOL 
June 2000 



Robert R. Read, Co-Advisor 

r<7Mf)0 

Lyn R. Whitaker, Second Reader 

^JZcUj_ €_ 

Richard E. Rosenthal, Chairman 
Department of Operations Research 


in 



THIS PAGE INTENTIONALLY LEFT BLANK 


IV 




ABSTRACT 


The Maintenance Climate Assessment Survey (MCAS) was developed to 
proactively assess factors that contribute to a high reliability organization and strong 
safety climate. The 3 rd Marine Air Wing (MAW), which was seeking to proactively 
improve its safety posture requested the assistance of the School of Aviation Safety at the 
Naval Postgraduate School to examine its safety climate. Previous studies of the MCAS 
instrument have focused on the items and their relationship to the HRO based model of 
safety effectiveness components: process auditing, reward system, quality assurance, risk 
management, command and control, and communication/functional relationships. The 
present effort is the first attempt to consider the relationship between item component 
responses and demographic item responses. It evaluates 893 maintainer responses to the 
MCAS from 3 rd MAW and looks for measurable effects due to demographics. This study 
finds that the regression models constructed using the demographics as explanatory 
variables have very little utility in predicting scores for the components. This result 
allows planners the relief of using the demographics as a low priority issue. 



vi 



TABLE OF CONTENTS 


I. INTRODUCTION.1 

A. BACKGROUND. 1 

B. PURPOSE.4 

C. PROBLEM STATEMENT.5 

D. SCOPE AND LIMITATIONS.6 

E. DEFINITIONS.6 

H. LITERATURE REVIEW.9 

A. HUMAN ERROR.9 

B. ORGANIZATIONAL SAFETY CULTURE.12 

1. Definition. 12 

2. Composition.12 

C. HIGH RELIABILITY ORGANIZATIONS.13 

1. Definition.13 

2. Characteristics of HROs.14 

D. ASSESSING SAFETY CLIMATE.16 

1. Safety Space.16 

2. MOSE and MCAS. 17 

3. Instrument Design and Demographics.19 

E. SUMMARY.20 

IB. METHODOLOGY.23 


vii 


A. RESEARCH APPROACH 


23 























B. DATA COLLECTION 


23 


1. Subjects.23 

2. Instrument.24 

3. Procedure.24 

C. DATA ANALYSIS.25 

1. Data Tabul ati on.25 

2. Statistical Analysis.26 

IV. RESULTS.29 

A. SIMPLE MODELS WITHOUT INTERACTION.29 

B. MODELS WITH TWO FACTOR INTERACTION.32 

C. REDUCED MODELS WITHOUT INTERATION.33 

D. REDUCED TWO FACTOR INTERACTION MODELS.34 

E. COMPARING MODELS.35 

F. INTERVIEWS. 36 

V. CONCLUSIONS.37 

A. FINDINGS.37 

B. RECOMMENDATIONS.38 

APPENDIX A. 43-ITEM MAINTENANCE CLIMATE ASSESSMENT SURVEY.... 41 

APPENDIX B. MODEL OF SAFETY EFFECTIVENESS COMPONENTS.45 

APPENDIX C. INFLUENCE PLOTS FOR AUGMENTED MOSE COMPONENTS.. 47 

APPENDIX D. SIMPLE MODELS: SCATTERPLOTS WITH SUPERIMPOSED 
REGRESSION LINE.49 

APPENDIX E. SIMPLE MODELS: HISTOGRAMS OF RESIDUALS.51 

viii 























APPENDIX F. SIMPLE MODELS: RESIDUALS VERSUS FITTED VALUES WITH 
LOWESS SMOOTHING.53 

APPENDIX G. SIMPLE MODELS: QQ-PLOTS.55 

APPENDIX H. SIMPLE MODELS: ANALYSIS OF VARIANCE.57 

APPENDIX I. TWO FACTOR INTERACTION MODELS: SCATTERPLOTS WITH 
SUPERIMPOSED REGRESSION LINE.59 

APPENDIX J. TWO FACTOR INTERACTION MODELS: HISTOGRAMS OF 
RESIDUALS.61 

APPENDIX K. TWO FACTOR INTERACTION MODELS: RESIDUALS VERSUS 
FITTED VALUES WITH LOWESS SMOOTHING.63 

APPENDIX L. TWO FACTOR INTERACTION MODELS: QQ-PLOTS.65 

APPENDIX M. TWO FACTOR INTERACTION MODELS: ANALYSIS OF 
VARIANCE.67 

APPENDIX N. REDUCED SIMPLE MODELS.69 

APPENDIX O. REDUCED SIMPLE MODELS: SCATTERPLOTS WITH 
SUPERIMPOSED REGRESSION LINE. 71 

APPENDIX P. REDUCED SIMPLE MODELS: HISTOGRAMS OF RESIDUALS.... 73 

APPENDIX Q. REDUCED SIMPLE MODELS: RESIDUALS VERSUS FITTED 
VALUES .75 

APPENDIX R. REDUCED SIMPLE MODELS: QQ-PLOTS.77 

APPENDIX S. REDUCED TWO FACTOR INTERACTION MODELS.79 

APPENDIX T. REDUCED TWO FACTOR INTERACTION MODELS: 
SCATTERPLOTS WITH SUPERIMPOSED REGRESSION LINE.81 

APPENDIX U. REDUCED TWO FACTOR INTERACTION MODELS: 
HISTOGRAMS OF RESIDUALS.83 

APPENDIX V. REDUCED TWO FACTOR INTERACTION MODELS: RESIDUALS 
VERSUS FITTED VALUES.85 


IX 


















APPENDIX W. REDUCED TWO FACTOR INTERACTION MODELS: 

QQ-PLOTS. 87 

APPENDIX X. COMPARISONS ON MODELS USING ANOVA.89 

APPENDIX Y. SUBJECT MATTER EXPERT RECOMMENDED CHANGES TO 
MCAS DEMOGRAPHIC FACTORS.91 

LIST OF REFERENCES.93 

INITIAL DISTRIBUTION LIST.97 








EXECUTIVE SUMMARY 


Naval Aviation is a hazardous undertaking, but in spite of its inherent risk, its 
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 to 1990. 
Over the last decade, however, the proportion of aircraft losses in which human error has 
been cited as a contributor has remained relatively constant. To address human factors 
issues in flight mishaps, the Human Factors Quality Management Board (HFQMB) was 
established in 1996. By using Mishap Data Analysis (MDA), Organizational 
Benchmarking (OB), and Command Safety Assessment (CSA), the efforts of the 
HFQMB resulted in a significant reduction in FM incidence from the perspective of 
aircrew operations. 

Although human error in maintenance is a smaller contributor, it has been shown 
to be a factor in nearly one in five Class A FMs. To address human error in maintenance, 
the Human Factors Analysis and Classification System - Maintenance Extension 
(HFACS-ME) was developed to classify error types in maintenance. Since Naval 
Aviation is shown to be a high reliability organization (HRO) defined as an organization 
that operates in hazardous environment with less than its fair share of accidents, it shares 
common characteristics with other HROs. These common characteristics are outlined in 
the Model of Organizational Safety Effectiveness (MOSE) and are process auditing, 
quality, reward system, risk management and command and control. Military aviation has 
communication/functional relationships as a sixth component. The augmented MOSE is 
the basis of the Maintenance Climate Assessment Survey (MCAS), which is used to 
evaluate the organizational safety climate from the perspective of the maintenance 


XI 



personnel. The MCAS consists of six demographic items and 43 perception items. Each 
of the 43 perception items maps into a single component of the augmented MOSE. These 
questions are collapsed into six component scores for each respondent. 

Although MCAS has been shown to be an effective tool for evaluating the safety 
environment in a maintenance organization, demographic factors and their potential 
relationship with maintainer responses have not been investigated. This study evaluates 
MCAS responses from 894 maintenance personnel of the 3 rd MAW, and looks at how the 
demographic factors of maintenance personnel might be biasing the component scores of 
the MCAS. The results of this thesis are intended to further refine MCAS demographic 
factors and provide Squadron Commanders with insight into the construct of their 
maintenance personnel. 

The component scores are fitted using the demographics as explanatory factors. 
Univariate analysis is performed for each component using simple models without 
interaction and also with models using two-factor interactions. These models are then 
simplified in order to reduce the number of terms to a more manageable level. 

The results of this thesis show that up to two-factor interaction, the demographic 
factors of the MCAS poorly account for the variance in the responses. The reliance on 
subjective perception in the scoring is the cause of the large amount of variance. Since 
variance cannot be explained by the demographic factors, the MCAS appears to be 
demographically unbiased. Input from subject matter experts is used to refine the 
demographic factors. These revised factors are more usable for Squadron Commanders in 
that they provide more insight into the make up of the maintenance organization. 


xii 




LIST OF ACRONYMS 


CSA Command Safety Assessment 

FM Flight Mishap 

HFACS Human Factors Analysis and Classification System 
HFACS-ME Human Factors Analysis and Classification System Maintenance 
Extension 

HFQMB Human Factors Quality Management Board 

HRO High Reliability Organization 

MAG Marine Air Group 

MAGTF Marine Air Ground Task Force 

MAW Marine Air Wing 

MCAS Maintenance Climate Assessment Survey 

MDA Mishap Data Analysis 

MOSE Model of Organizational Safety Effectiveness 

OB Organizational Benchmarking 

School of Aviation Safety 


SAS 



I. INTRODUCTION 


A. OVERVIEW 

Naval Aviation is a hazardous undertaking, but in spite of its inherent risk, its 
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 and 
1990 (Naval Safety Center, 1997). Class A Mishaps are defined as Naval aircraft 
incidents resulting in death, permanent disability, or property loss or damage in excess of 
one million dollars (OPNAV 3750.6Q, 1989). A flight mishap (FM) is defined as those 
mishaps in which there is $10,000 or greater DoD aircraft damage or loss of a DoD 
aircraft, and intent for flight for DoD aircraft existed at the time of the mishap. Other 
property damage, injury, or death may or may not have occurred. Naval Aviation 
consistently maintains high levels of operability coupled with less than its fair share of 
accidents (Goodrum, 1999). Naval Aviation also possesses the requisite characteristics of 
a high reliability organization (HRO): process auditing, reward system, quality, risk 
management, and command and control. For these reasons, Roberts (1988) labeled Naval 
Aviation an HRO. 

Although Naval Aviation is successful in reducing its Class A FM rate, over the 
last decade the proportion of aircraft losses in which human error has been cited as a 
contributor has remained relatively constant at four of five FMs (Naval Safety Center, 
2000). In 1996, a Human Factors Quality Management Board (HFQMB) is established 
after 17 Class A FMs occurred in only 75 days, climaxing when a Navy F-14 crashes into 
a Nashville, TN neighborhood, to address human factors issues related to mishaps 
(Nutwell & Sherman, 1997). The goal of the HFQMB is to cut the current Class A FM 
rate due to human error in half by year 2000 (HFQMB Charter, 1996). The HFQMB 


1 



adopts three approaches to identify and target factors contributing to human error: 1) 
Mishap Data Analysis (MDA), 2) Organizational Benchmarking (OB), and 3) Command 
Safety Assessment (CSA). 

MDA establishes the development of HFACS, which is used to identify and 
prioritize human factors contributors to FMs. Among others, it determines inadequate 
supervision and aircrew violations are significant contributors (Shappel & Wiegman, 
1997). Using OB which explores programs which influence aircrew performance, the 
HFQMB determines use of feedback mechanisms in commercial airlines improve crew 
resource management training benefits (Nutwell & Sherman, 1997). Finally, a CSA 
survey, based on a model of HROs, is developed to determine a command’s safety 
posture from an aircrew perspective. This survey finds that 55% of the Navy respondents 
and 65% of the Marine Corps respondents feel that their commands are committed 
beyond what available resources can provide (Ciavarelli & Figlock, 1997). These 
combined efforts make significant progress toward the HFQMB’s goal as evidenced by 
fiscal year 1999 being the safest year in Naval Aviation history in terms of Class A FM 
rate. 

Maintenance is shown to be a contributing factor in nearly one in five Class A 
FMs (Naval Safety Center, 2000). Additionally, during FY90-97, Class C FMs account 
for 75% of all maintenance related mishaps (MRMs). Maintenance is one area where 
hazards can be controlled and risk can be managed while an aircraft is on the ground. 
Much work is done in the field of human factors in maintenance safety for commercial 
airlines (“Human Factors in,” 2000). In 1988, the Aviation Safety Research Act (ASRA) 
mandates close study of aging aircraft structures and human factors affecting safety 


2 



(“History,” 2000). In the spirit of this mandated study, Boeing finds that incomplete 
installation (34%), damaged on installation (15%), improper installation (11%) and 
equipment not installed/missing (11%) were the top contributors in maintenance error 
(Komamiski, 2000). This investigation and classification of types of human error in 
maintenance leads to the development of Boeing’s Maintenance Error Decision Aid 
(MEDA), a system that aids operators and maintainers in the investigation and mitigation 
of maintenance related errors (Allen, Rankin, & Sargent, 1998). 

The ASRA is one of the precursors for the FAA’s current goal to reduce the fatal 
accident rate 80% by 2007 as compared to 1994-1996 baseline data (FAA, 1998). Several 
key initiatives are the stepping stones for this ultimate goal: 1) the development of a 
maintenance resource management system; 2) establishment of new training 
requirements; 3) implementation of technical advances in aircraft maintenance at repair 
stations; 4) enforcement of safety recommendations from the National Transportation 
Safety Board (NTSB); and 5) recommendations for aging systems maintenance. Little 
work until recently is done for military aviation, and recent efforts involve studying 
forms of maintenance error (Schmorrow, 1998) and the perceived maintenance safety 
climate (Baker, 1998). From a proactive perspective, efforts must be made to continue 
developing assessment tools to identify potential areas for risk management and control 
of conditions before a mishap occurs. 

Using the Maintenance Climate Assessment Survey (MCAS), Goodrum (1999) 
and Oneto (1999) are able to show the prototype survey effectively evaluates a 
maintainer’s perception of safety in maintenance operations. They also further refine the 
MCAS into a present 43 question format. One aspect of the MCAS yet to be addressed is 


3 


the demographic categories to determine their potential relationship with maintainer 
responses. Given the structure of maintenance organizations within aircraft communities 
vary, it is unlikely that any pair of samples will have equal proportions of all 
demographic variables. It is anticipated that individual demographic characteristics may 
influence MCAS responses and are therefore potentially biasing the results. By 
understanding the effects of the demographics, one can understand if they impact their 
organization’s safety climate. 

B. BACKGROUND 

The 3 rd Marine Air Wing (MAW) is a combat-ready expeditionary aviation force 
capable of short-notice worldwide employment to Marine Air Ground Task Force 
(MAGTF) fleet and unified commanders. It is composed of 28 squadrons divided into 
four Marine Air Groups (MAGs) based in Southern California and Arizona. Each MAG 
has its own combat mission: MAG 11 provides air support to MAGTF commanders; 
MAG-13 provides close-air support, conducts armed reconnaissance, and assumes 
limited air-defense roles; MAG-16 transports and resupplies Marine air and ground units; 
and MAG-39 provides utility helicopter support, close-in fire support, fire support 
coordination, aerial reconnaissance, observation and forward air control in aerial and 
ground escort operations during ship-to-shore movement and subsequent operations 
ashore. The aircraft used in these missions are AH-lWs, UH-lNs, CH-53s, CH-46Es, 
F/A-18DS, F/A-18s, AV-8s and C-130Ts. 

From 1990 to 1996, maintenance is a causal factor in 17 percent of all Naval 
Aviation class A FMs (Naval Safety Center, 1997). From April 1997 to July 1999, 
maintenance, maintenance personnel or maintenance depot is cited as a causal factor in 


4 




14 FMs (eight class C FMs, four class B FMs, and two class A FMs) experienced by 3 rd 
MAW. The Commander of 3 rd MAW requests the assistance of the School of Aviation 
Safety (SAS) at the Naval Postgraduate School, which in turn provides safety and risk 
management training to personnel, mishap data analysis, and administration of safety 
climate surveys to help locate problems in the organization. 

C. PROBLEM STATEMENT 

Human error in aviation is an issue that needs to be addressed, and it is recognized 
that the organization has an impact on factors that lead to it. Organizations that possess 
the attributes of a HRO tend to generate environments conducive to the reduction or 
control of human error and consequently experience fewer mishaps. Organizations 
aspiring towards the reduction of mishaps need to assess their safety posture as it relates 
to the attributes of HROs. The School of Aviation Safety at the Naval Postgraduate 
School has developed surveys to assess HRO characteristics in the operational 
environment for aircrew and maintenance personnel. 

The 3 rd MAW in an attempt to improve its safety posture enlists to have the 
School of Aviation Safety employ the MCAS survey to assess maintainer perception of 
HRO characteristics in its recent operations. These results are revealing, however in order 
to provide for better interpretation of the results, an exploration of the demographic 
variables is in order. This will help commanders to target more effectively specific areas 
of the organization that require attention. 

The current version of the MCAS is administered to the 3 rd MAW during the last 
half of 1999. Using statistical methods, the collected data is analyzed to assess 


5 



differences in responses that are correlated to differences in demographics. This thesis 
explores the following questions: 

1. Are there measurable demographic effects to the responses on the MCAS? 

2. Is there enough information in the demographics to be used in an adjustment 
process of the overall scores? 

3. Can the MCAS be refined further to either collapse or expand demographic 
factors? 

D. SCOPE AND LIMITATIONS 

Active duty U.S. Marine Corps Squadrons maintenance personnel of the 3 rd 
MAW are surveyed during the fall of 1999. Only those squadrons with a representative 
number of respondents are used in the survey. Chapter II provides a basis for 
understanding human error, organizational safety culture, high reliability organizations 
and the assessment of a safety climate. Chapter III presents a discussion of the 
methodology used in this study. Results of data analysis are presented in Chapter IV. 
Chapter V summarizes previous chapters and provides conclusions and recommendations 
as they relate to the material. 

E. DEFINITIONS 

This thesis uses the following definitions (DON, 1989): 

Naval Aircraft . Refers to U.S. Navy, Naval Reserve, U.S. Marine Corps, and U.S. 
Marine Corps Reserve aircraft. 

Mishap . A Naval Aviation mishap is an unforeseen or unplanned event that directly 
involves naval aircraft, which result in $10,000 or greater cumulative damage to naval 
aircraft or personnel. The mishap is further divided into three classes based on the 


6 



amount of damage to the aircraft, property and personnel injury. The following are 
the definitions of the three classes: 

a. Class A . A mishap in which the total cost of property damage (including all 
aircraft damage) is $1,000,000 or greater; or a naval aircraft is destroyed or 
missing; or any fatality or permanent total disability of a person occurs with direct 
involvement of naval aircraft. 

b. Class B . A mishap in which the total cost of property damage (including all 
aircraft damage) is $200,000 or more but less than $1,000,000 and/or a permanent 
partial disability, and or the hospitalization of five or more personnel. 

c. Class C . A mishap in which the total cost of property damage (including all 
aircraft damage) is $10,000 or more but less than $200,000 and/or injury results in 
one or more lost workdays. 

Mishap rate . The total number of Class A,B and C mishaps per 100,000 flight hours. 
MCAS . A 43-question survey used to gain insight into the maintenance community’s 
perception concerning aviation mishaps within the Navy and Marine Corps. 
HFACS-ME . A taxonomic system used to classify causal factors that contribute to 
maintenance related mishaps. 

HRO . High-Reliability Organization, is an organization that operates in a hazardous 
environment, yet produces very low rate of accidents and incidents, operating effectively 
and safely and having the characteristics of leadership, sound management policies, 
procedure standardization, adequacy of resources and staffing, a defined system for risk 
management, and other factors. 


7 


THIS PAGE INTENTIONALLY LEFT BLANK 


8 




II. LITERATURE REVIEW 


A. HUMAN ERROR 

Reason (1990) defines error as a planned sequence that fails to achieve its 
intended outcome in the absence of external influence. He (1997) later describes error 
types as active or latent. Where the effects of active errors are often immediate and 
confined, latent conditions lie dormant until set off by a chain of local events and can be 
contributing factors in a variety of failures. This model of latent conditions and active 
failures is adopted by the Naval Safety Center to investigate Class A FMs with respect to 
aircrew error, and was the basis for the development of Human Factors Analysis and 
Classification System (HFACS) (Shappel & Wiegman, 1997). 


I Activity 

‘Hands On’ 

■KifllifgW 


Normal control 

Low 

Moderate 

HHE39H 

Emergency control 

Moderate 


Low 

| Maintenance-related 

■■ 

HES3HI 

MEM3HI 


Table 1. Likelihood of Performance Problems in Given Activities. 


Reason (1997) also models human error in the scope of universal human activities 
(see Table 1) and the likelihood of performance problems within each of these types of 
activities. With HFACS, the Naval Safety Center is able to address human error in 
normal control and emergency control conditions. But Reason asserts that maintenance is 
the area with the highest likelihood of human error because maintenance related activities 
are almost exclusively reliant on human performance in the three areas of hands on, 
criticality and frequency. Even with progress in technology, human fallibility remains 


9 
















constant (Reason, 1997) and with the frequency of planned maintenance compounded 
with the many pairs of fallible human hands working on exceptionally complicated 
systems, Reason’s model is a chilling prediction of 3 rd MAW’s situation. 

To address the maintenance related activity as a contributor to the total of human 
errors in FMs, Schmidt, Schmorrow and Hardee (1998) extend HFACS to specifically 
address the maintenance component of accident causation with the HFACS-Maintenance 
Extension (ME). The HFACS-ME expands upon Reason’s model of latent states and 
psychological precursors to unsafe acts. Reason (1990; 1997) differentiates these latent 
states in that they create the potential for human error. HFACS-ME classifies latent states 
in the maintenance environment, with three levels of error causation under four 
categories of conditions. The first order conditions are broad precursor categories 
(supervisory conditions, maintainer conditions, working conditions and maintainer acts) 
that are further divided into more specific precursors to human error of the second and 
third order (see Table 2). 

The causes of human error are many. Wickens, Gordon and Lui (1997) state that 
human error can be induced by “inattentiveness, poor work habits, lack of training, poor 
decision making, personality traits, social pressures, and so forth” (p. 427-428). Redmill 
& Rajan (1997) note that a common component in accidents is a worker’s loss of 
concentration which can be caused by “boredom, disinterest, distraction, or attempt to do 
two or more things at once” (p.12). Wickens, Gordon and Lui (1997) assert that the most 
common type of maintenance related error is that of omission. Considering that the nature 
of planned maintenance is to frequently disassemble, inspect then reassemble 
components, it is clear to see the high probability of human error in maintenance (Reason 


10 




1997). But as Perrow and Reason point out, the errors themselves are seldom isolated, but 
often a single event in a causal chain (Perrow, 1984), or encouraged, or at least not 
prohibited from occurring by latent conditions in the organization (Reason, 1997). 


First Order 

Second Order 

Third Order 

Supervisory Conditions 

Unforeseen 

Hazardous Operations 
Inadequate Documentation 
Inadequate Design 


Squadron 

Inadequate Supervision 
Inappropriate Operations 

Failed to Correct Problem 
Supervisory Violation 

Maintainer Conditions 

Medical 

Mental State 

Physical State 

Physical/Mental Limitation 


Crew Coordination 

Communication 

Assertiveness 

Adaptability/Flexibility 


Readiness 

Preparation/Training 

Qualification/Certification 

Violation 

Working Conditions 

Environment 

Lighting/Light 
Exposure/Weather 
Environmental Hazards 


Equipment 

Damaged 

Unavailable 

Dated/Uncertified 


Workspace 

Confining 

Obstructed 

Inaccessible 

Maintainer Acts 

Error 

Attention 

Memory 

Rule/Knowledge 

Skill 


Violation 

Routine 

Infraction 

Exceptional 


Table 2. HFACS-ME Levels of Error Causation. 


li 








B. ORGANIZATIONAL SAFETY CULTURE 


1. Definition 

Organizational culture is defined as shared values and beliefs that interact with an 
organization’s structures and control systems to produce behavioral norms (Uttal, 1983). 
All organizations have their own engineered culture whether good or bad. A safety 
culture is ideal for complex organizations and is defined as the product of individual and 
group values, attitude, competencies, and patterns of behavior that determine the 
commitment to, and the style and proficiency of, an organization’s health and safety 
programmes (Booth, 1993). 

2. Composition 

Redmill and Raj an (1997) state that there are three general aspects of safety 
culture: awareness, commitment and competence. Awareness must be present in all 
aspects of design, management and decision making. When a safety mishap does occur, 
it is commitment that drives the leadership of the organization to investigate and locate 
contributing factors and take immediate action to prevent another occurrence. 
Competence is a combination of education, training, professionalism and personality 
traits that are appropriate for a given task or job (Redmill & Rajan, 1997). 

Reason (1997) prefers the term informed culture, and he divides informed culture 
into four subcultures: reporting culture, just culture, flexible culture and learning culture. 
Reporting culture is “an organizational climate in which people are prepared to report 
their errors and near-misses.” Just culture is an “atmosphere of trust in which people are 
encouraged, even awarded, for provided essential safety-related information - but in 
which they are also clear about where the line must be drawn between acceptable and 


12 



unacceptable behavior.” Flexible culture involves “shifting from the conventional 
hierarchical mode to a flatter professional structure, where control passes to task experts 
on the spot, and then reverts back to the traditional bureaucratic mode once the 
emergency has passed. Such adaptability is an essential feature of the crisis-prepared 
organization.” Learning culture is “the willingness and the competence to draw the right 
conclusions from its safety information system, and the will to implement major reforms 
when their need is indicated.” 

Safety culture has powerful effects. First, it is self perpetuating where workers 
learn from each other and encourage each other to work in a manner consistent with the 
organization’s safety culture. People are quick to follow the example of coworkers, even 
if this means a lack of vigilance in safety (Redmill & Rajan, 1997). Wogalter, Allison, & 
McKenna (1989) assert that “people are extremely susceptible to social norms; they are 
likely to engage in safe or unsafe behaviors to the extent that others around them do so.” 
Safety culture is also self preserving as new workers learn to identify acceptable methods 
of accomplishing work and are able to pass those standards along to new employees, 
good or bad (Redmill & Rajan, 1997). 

C. HIGH RELIABILITY ORGANIZATIONS 

1. Definition 

Roberts (1990) and Libuser (1994) explain that High-Reliability Organizations 
(HROs), organizations that operate in a hazardous environment, yet produce very low 
rates of accidents and incidents, which operate effectively and safely have certain key 
characteristics in common. Reason (1997) calls them organizations with less than their 
fair share of accidents, and “highly complex, technology-intensive organizations that 


13 




must operate, as far as humanly possible, to a failure-free standard.” He also explains 
that HROs manage complex technologies that have very low tolerance for error, yet 
maintain the flexibility to successfully function in environments of extreme intensity. 

2. Characteristics of HROs 

Examples of HROs are the nuclear power industry, petrochemical industry, and 
airline industry. Additionally, Figlock (1998) identifies Naval aviation is an HRO. 
Although diverse in purpose, Roberts and Libuser believe these organizations share 
several common characteristics: leadership style, management policies, procedures 
standardization, superior training, a reward system that recognizes safety achievement, 
adequacy of resources and staffing, effective management of risks associated with 
hazardous operations, and other factors. 

HROs have a requisite variety. As Weick (1987) states, having diverse people 
from diverse backgrounds and experiences builds requisite variety that is required for 
relatively simple humans to operate complex systems. Additionally, this diversity is 
essential in problem solving, as individuals will approach the same problem uniquely, so 
that the collective contribution is greater than any one individual’s input. 

HROs typically exhibit a high degree of training. Weick (1987) notes that 
“training for the operation of high reliability systems is often tough and demanding so 
that the faint of heart and the incompetent are weeded out.” This is because HROs are not 
afforded the luxury of trial and error. Training is often in the form of simulation and 
stories. Stories have a big affect on the reliability of an HRO by lending experience to the 
inexperienced: 


14 




The basic idea is that a system which values stories, story tellers, and 
storytelling will be more reliable than a system that derogates these 
substitutes for trial and error. A system that values stories and storytelling 
is potentially more reliable because people know more about their system, 
know more of the potential errors that might occur, and they are more 
confident that they can handle those errors that do occur because they 
know that other people have already handled similar errors (Weick, 1987, 

P- 113). 

By sharing the experiences of skilled personnel, novices do not have to learn from their 

own mistakes and are also granted the insight of the skilled. 

HROs appear bureaucratic and uneventful on the surface. There is a strict chain of 

command in place that dictates policy, procedure and environment. This strong 

centralization is apparent during periods of relatively low intensity, but as intensity of 

operating increases, the true nature of the HRO is revealed, where flexibility, delegation, 

improvisation and technical expertise dominate (Reason, 1997; Weick, 1987). This is 

how HROs can be simultaneously centralized and decentralized. Responsibility and 

judgment remain centralized while creativity, improvisation and unsupervised problem 

solving become decentralized in environments of high intensity (Weick, 1987). 

Reliability in HROs is another deceptive aspect. Weick (1987) calls reliability a 

“dynamic non-event,” meaning a great deal of effort goes into ensuring nothing happens. 

This dynamicism is based on the belief that reliability is fleeting and systems tend to 

move to states of unreliability unless constantly maintained: 

Part of the mindset for reliability requires chronic suspicion that small 
deviations may enlarge, a sensitivity that may be encouraged by a more 
dynamic view of reliability (Weick, 1987, p. 119). 

It is the organizational culture for personnel to look for problems before they happen. 

Weick continues that because of the invisibility of the dynamics behind reliability, there 


15 



is a perception that reliability is easily achievable and is only noticed in the presence of a 
breakdown. 

D. ASSESSING SAFETY CLIMATE 

1. Safety Space 

Reason (1997) states that organizations can be mapped into a safety space which 
is a continuum of degrees of susceptibility to accidents (see Figure 1). Organizations with 
higher resistance will generally have fewer mishaps while organizations with higher 
vulnerability will generally experience more mishaps. No organization is immune. 
Chance, unforeseen circumstances, failures in defenses and human error can cause even 
the most resistant organizations to experience accidents. Within the safety space, currents 
tend to push organizations away from the extremes of resistance or vulnerability and 
toward the center, a compromise between the two. If an organization has the desire to 
become more resistant, it must swim upstream. 



Figure 1. Organizational Safety Space. 


16 





























The effort required to move in the direction of increased resistance must be put 
into reactive and proactive measures. Reason (1997) contends that investigating mishaps 
to find causal factors to be addressed is not even half the battle. To effectively move the 
organization, mishap investigation must be coupled with the identification of conditions 
needing correction, and regular checks. 

2. MOSE and MCAS 

Libuser’s (1994) current Model of Organizational Safety Effectiveness (MOSE) is 
based on work by Roberts and is a categorization of the common characteristics of 
HROs. These characteristics are mapped into five components: 1) Process Auditing (PA) 
- checks by members to identify hazards; 2) Reward System (RS)- expected rewards or 
disciplinary action used to shape behavior; 3) Quality Assurance (QA)- promotion of 
quality performance; 4) Risk Management (RM)- system to identify hazards and control 
operational risks; and 5) Command and Control (CC)- safety climate, leadership 
effectiveness /policies, and procedures for mitigating risks. These components are very 
similar to the aspects of Reason’s (1997) informed culture (Table 3). 


Libuser’s MOSE Components 

Reason’s Informed Culture 

Process Auditing (PA). 

Leaning Culture 

Reward System (RS) 

Just Culture 

Quality Control (QA) 

Reporting Culture 

Risk Management (RM) 

Flexible Culture 

Command and Control (CC) 

Flexible Culture 


Table 3. Comparison of Libuser’s MOSE and Reason’s Informed Culture. 


17 














Ciavarelli and Figlock (1997) adapt the MOSE for use in Naval Aviation using 
practices and terminology of that environment and develop the Command Safety 
Assessment, a survey that addresses each of the MOSE categories from the viewpoint of 
the aircrewman. This survey is administered to 1,254 aviators revealing that 
organizational and supervisory issues are seen by aircrewmen as impacting flight safety. 

The Maintenance Climate Assessment Survey (MCAS) is the product of the 
implementation of the MOSE and CSA in a maintenance context. Baker (1998) starts by 
reducing 155 candidate questions to 67 items that specifically addressed aviation 
maintenance. Augmenting Libuser’s (1994) five category MOSE model with a sixth 
category, Communication/Functional Relationships, Baker (1998) modifies the CSA to 
look at aviation safety from the point of view of the maintenance person. Using 
regression techniques, Baker is able to further reduce the survey into a compact 35 item 
form, with almost all questions mapped to a single category of the augmented MOSE. 

Goodrum (1999) and Oneto (1999) show that the MCAS is a valid tool to 
accurately assess an aviation maintenance environment, but note that some items in the 
survey need restructuring. Oneto notes that these items address more than one category of 
the MOSE. Their inputs help change the MCAS to its current 43 item format (see 
Appendix A). While Goodrum and Oneto are able to show content validity in the MCAS, 
there is much left to examine. Since there is no known or accepted measure for MCAS 
results and providing feedback to the concerned squadrons, an effort is underway to 
explore concurrent validity in the survey on a per question basis (Schmidt, personal 
communication). Questions that have a low response mean are noted as areas that need 


18 




attention, corresponding to a particular category of the MOSE, and which part of the 
HRO needs closer examination. 

Additionally, by looking at squadron mean scores to the survey and available 
mishap data, Harris (personal communication) is looking at the MCAS predictive validity 
in the incidence of mishaps within a squadron based on adjusted mean scores. While the 
work by Harris seeks to broaden the scope and applicability of the MCAS, this study 
explores internal aspects of the survey. By separating responses by demographics, this 
thesis will further explore the attitude of the Naval aviation maintenance person with 
respect to safety. 

3. Instrument Design and Demographics 

The design of the MCAS is a cross-sectional one time look at a maintenance 
organization (see Appendix A). It is a self-administered questionnaire that polls 
respondent perception about the safety of their working environment at all levels. 
Demographic items preserve anonymity by excluding personal questions which could be 
linked to individuals allowing personnel the freedom to express their true opinion to the 
perception items (Oneto, 1999). 

Though there are many methods for data collection from people, surveys, when 
designed properly, are very effective for recording respondent scores based on a 
particular model (Fink & Kosecoff, 1985). MCAS responses are forced using a five point 
Likert scale (see Appendix A). Using a forced scale instead of subjective comments 
allows for rapid compilation of data and analyzation of responses as numerical values, 
which is convenient for numerical scoring and comparisons of scores between 
respondents, or groups of respondents (Fink & Kosecoff, 1985). The grouping is 


19 



facilitated by the six demographic items of the survey (Baker, 1998). By being able to 
group types of maintainers independent of their scores, it is possible to investigate for 
unequal perceptions across demographic groups. 

The demographic items in the MCAS capture aspects of the maintainer within the 
organization such as experience, and rank, and ignore personal information such as age, 
race, sex and ethnicity (Baker, 1998). Although it would be difficult to capture every 
possible combination of demographic factors, the MCAS is able to capture most job types 
from most aircraft communities. Each of the demographic line items in the MCAS 
represents a simple factor that might influence the scoring. It is unknown which of these 
demographics constitutes a valid or invalid factor. 

E. SUMMARY 

HROs are complex and dynamic by nature but are not impervious to unsafe trends 
in human performance. Perrow (1984) points out that regardless of an organization’s 
structure and nature, “normal accidents” will continue to occur. With HROs, these 
accidents tend to occur less frequently, but the consequences of the accidents tend to be 
large in magnitude. These organizations must put effort into swimming upstream through 
Reason’s safety space towards increased resistance to accidents. This happens through 
reactive measures like mishap analysis and proactive measures to identify “pathogenic 
conditions” (Reason, 1997). Though reactive measures are in place, proactive measures 
are coming up to speed. 

Two critical parallel developments in organization safety theory in regards to 
aviation safety are the development of effective taxonomies (e.g., HFACS-ME) and the 
identification and accurate modeling of HROs (e.g., augmented MOSE). The 


20 



development of CSA and MCAS are steps to link the taxonomies and the organization 
models. Since MOSE parallels Reason’s informed culture, it is possible with MCAS to 
identify those conditions that are not conducive to safety and take proactive measures to 
move the organization towards increased resistance to accidents in the safety space. Since 
3 rd MAW is a HRO, CSA and MCAS allow for the identification of the MOSE 
components that require attention. 

The MCAS has been revised into a more usable form, and has been validated, 
showing that the individual items do address specific MOSE components. This thesis is 
another step into revising the MCAS further by looking at the validity of the demographic 
factors. Mapping the demographic factors into the MOSE component scores will show if 
the factors are relevant or can be removed from the survey. Additionally, if these factors 
do not account for the variance in scores, the survey is not asking the correct 
demographic questions and will require further revision. 


21 



THIS PAGE INTENTIONALLY LEFT BLANK 


22 




III. METHODOLOGY 


A. RESEARCH APPROACH 

The intent of this study is to assess the maintained s perception of safety in his or 
her work environment. This research involves the use of analysis techniques to partition 
the collected data into smaller groups based on demographics then investigate differences 
in responses among the groups. If statistical differences are found, a comparison between 
the group mean and a particular squadron mean shows how a particular demographic 
differs from the rest of the squadron. Conversely, it shows which demographics have 
response means that are more reflective of the squadron means. 

B. DATA COLLECTION 

1. Subjects 

Surveys are administered to 977 officers and enlisted personnel responsible for 
Naval Aviation maintenance. These subjects come from squadrons and maintenance units 
of the 3 rd Marine Air Wing located at MCAS Miramar, CA, Camp Pendleton, CA, and 
MCAS Yuma, AZ. The aircraft represented are the AH-1 “Super Cobra,” UH-1 “Huey,” 
CH-53 “Super Stallion,” CH-46E “Sea Knight ” F/A-18D “Night Attack Hornet,” F/A-18 
“Hornet,” AV-8B “Harrier,” and the C130T “Hercules.” 

Additionally, subject matter experts are interviewed about what they consider to 
be important demographic information about the personnel in their maintenance 
organizations. Subject matter experts are military aviators with at least eight years active 
duty service. The results of these interviews are given in the next section. 


23 


2. Instrument 


The MCAS is a self-administered, group survey consisting of two parts: 1) 
demographics; 2) perception. Part I captures demographic factors of each subject: 
community, squadron, rank, years of aviation maintenance experience, work center, and 
shift. There are eight choices available for community, with an additional option of 
“other.” The squadron factor records the three-digit squadron designator. Embedded in 
these two factors is aircraft type (seventh factor). Rank is divided into four levels of 
enlisted personnel and three levels of officer personnel. Years of aviation maintenance 
experience is partitioned into seven levels. Work center or shop, is divided into eight 
shops with the option of “other.” Shift divides subjects into dayshift or nightshift. 

Part II captures subject perception of his or her work environment. There are 42 
items, each of which is mapped into a single augmented MOSE component: process 
auditing (six questions), reward system (eight questions), quality assurance (six 
questions), risk management (nine questions), command and control (eight questions), 
and communication/functional relations (six questions). Each perception item asks 
subjects to rank a specific safety related activity or aspect of their organization using a 
five point Likert rating scale with verbal anchors as follows: Strongly Disagree, Disagree, 
Neutral, Agree, Strongly Agree. When completed, the items for each MOSE component 
are averaged to attain six composite scores, each one corresponding to the subject’s rating 
of that particular augmented MOSE component for his or her organization. 

3. Procedure 

The survey is administered on site and in a group setting at the various 
participating Squadrons of the 3 rd MAW. Additionally, the survey is given in 


24 




conjunction with a scheduled maintenance safety presentation on human factors issues in 
aviation. The Squadrons are in various stages of training and operational tasking at the 
time of the survey being administered. The variety of operational tasking with which the 
squadrons are simultaneously involved during the administration of the MCAS accounts 
for much of the variance in the number of surveys collected from each squadron. 
Potential MCAS respondents are briefed on maintenance issues, the survey and its 
purpose and questions that arise pertaining to the survey are answered by the survey 
administer. Respondents fill out the surveys using scannable computer forms. The 
surveys are then immediately collected upon completion to allow for maximum 
accountability. 

For personal interviews, subject matter experts report what they think are the 
important factors when considering the demographics of personnel in their maintenance 
organizations. Notes are taken during these interviews and a list is compiled from the 
responses of the personnel interviewed. This list represents potential MCAS demographic 
items. 

C. DATA ANALYSIS 

1. Data Tabulation 

Survey results were compiled into a database using a scanning machine, then 
imported into Microsoft Excel. The spreadsheet consists of rows of respondents and 
columns of survey items (both demographic and survey items). Demographic items 
record mainly bivariate and multivariate responses, such as squadron (aircraft type 
embedded), rank, and years experience. Survey item responses were assigned a numerical 
value of 1 through 5 corresponding to the Likert scale, with higher values being assigned 


25 


to more positive responses (strongly agree) and lower values assigned to more negative 
responses (strongly disagree). Each of the questions in the survey addresses one of the 
six MOSE components. Items addressing similar MOSE components were collapsed into 
an average score for that particular component. Any items that were missing values were 
excluded and not averaged into the component score. No weighting is assigned to items 
in the event of a missing item score. The three demographic response items 
corresponding to the respondent’s squadron number were collapsed into a single coded 
value. 

2. Statistical Analysis 

Microsoft Excel is used to provide summary statistics and initial familiarization 
with the data. The data were cleansed by removing subject responses with omitted 
demographic items leaving 894 responses. Items corresponding to augmented MOSE 
components were averaged for each respondent, leaving six scores along with 
demographic response items. If subjects omitted a perspective item response, the 
component score is averaged for the completed items of that component. No weighting 
for omitted perspective items is administered. Histograms of the demographic make up of 
the data are constructed. The data are then exported to ARG for its powerful graphing 
capabilities. Initial scatterplot matrices of each component over all demographic factors 
revealed no linear, exponential, or power trend, although scatterplots matrices of the 
components over themselves revealed a linear trend. 

The data are then transported to MathSoft S-plus for analysis. Categorical 
demographics are coded as factors and the aov function is implemented to fit the six 
component scores based on demographic factors. Six models (one for each component) 


26 




are constructed for a regression without interaction among factors. Six additional models 
are fit for a regression looking at two factor interaction. Each model is then simplified by 
using S-plus to remove unimportant terms. Models are compared and similar models for 
some of the components are found. Three term interaction models are not explored in this 
thesis. 


27 


THIS PAGE INTENTIONALLY LEFT BLANK 


28 



IV. RESULTS 


A. SIMPLE MODELS WITHOUT INTERACTION 

A scatterplot matrix is constructed to see if there are any trends in the data. This is 
done without designating the independent variables as factors. The scatterplot shows that 
there is some relationship between the component scores, but little information about the 
demographics is revealed. Figure 3 is a scatterplot matrix for PA versus each of the 
demographic factors. Visual inspection reveals a possible relationship between PA and 
Rank and Total years of Aviation experience. 



Figure 1. Scatterplot of PA versus Demographic Factors. 


Three dimensional bar plots for the components versus each factor are constructed 
to see if there are any visual clues as to some type of relationship between the factors and 
the response. No linear, log linear or exponential relationship is visible. However, the 


29 





































plots seem compatible with those of normally distributed histograms for each level of 
factor, all centered in the same approximate region of the component score (e.g. Figure 
4). 



Figure 2. Three Dimensional Bar Plot of PA Scores. PA scores 
are plotted against different levels of Total Years of Aviation 
Maintenance Experience. 

Since there is no indication that a transformation of the data is required due to 
visible trends, linear models for the six individual components are fit against 
untransformed factors without interaction using the aov function in S-plus. Model 
checking plots are constructed and case 219 is shown to have very high influence in all 
models. Case 219 is an E-6/7 with 15-20 years of aviation maintenance experience, 
works the day shift in “other” work center, in a VMH squadron. Although there is no 
significance test associated with Cook’s Distance, case 219 is deleted from the data set 
due to its unusually high influence and new models were fit (see Appendix C). 

Model checking plots are constructed to check the fit of the models with case 219 
removed. The scatter plot of the data with the regression superimposed reveals what 


30 








might be a slight upward trend in the response for all components (see Appendix D). The 
histograms of the residuals show that the distribution of the residuals appear to be normal 
(see Appendix E), and the scatterplot of the residuals versus the fitted values shows no 
discemable pattern in the residuals (see Appendix F). The QQ-plot shows that the 
residuals are thin at both tails for all models (see Appendix G), but the normal shape is 
tenable. 


Component 

R :2 

<7 

PA 

0.1095 

0.5529 

QA 

0.1219 

0.6327 

RS 

0.1427 

0.5896 

RM 

0.1869 

0.5690 

CC 

0.1280 

0.6349 

CR 

0.1233 

0.6846 


Table 4. R-squared and o for Models without Interaction. 


Values for the coefficient of determination, R 2 , show that these models account 
but poorly for the variance in the data. The best model is RM, accounting for less than 
19% of the total variance (see Table 4). The model with the lowest R 2 is PA with only 
11% of the total variance explained. Values for standard error, a, indicate that there is a 
large spread in the response values. For example, with a perfect R 2 of 1, the model for PA 
tells us that 68% of the respondents score between 3.31 and 4.42, and that 96% score 


31 




between 2.76 and 4.97. Since the only possible scores are between 1 and 5, relatively 
high values for & are not much help with understanding the data. 

The results from the analysis of variance from each model causes rejection of the 
null hypothesis that all of the coefficients in the model are zero, accepting the alternate 
hypothesis that at least one coefficient is not equal to zero (see Appendix H). 
Additionally, the models for PA, CC and CR fail the lack of fit test indicating that the 
shape of the fit is not correct. These models do not do well in describing the data. 

B. MODELS WITH TWO FACTOR INTERACTION 

Two-factor interaction models are constructed for all of the components using S- 
plus to see if more of the variance in the data can be modeled and model checking plots 
are constructed. The scatterplot of the data with the superimposed regression shows a 
linear trend in the response against the factors (see Appendix I), and the distribution of 
the residuals appears to be normal (see Appendix J). The scatterplot of the residuals 
versus the fitted values shows no discemable pattern in the residuals (see Appendix K). 
The QQ-plots show strange behavior at values close to zero, but that they are close to 
being normal for all components (see Appendix L). R 2 and & are given in Table 5. 

These two term interaction models are better at explaining more of the variance in 
the data as indicated by the values for the coefficients of determination, however there is 
very little reduction in the values for the standard error. This improvement in the values 
for R 2 comes at the cost of increased complexity in the models. While the models without 
interaction have 43 terms, the two term interaction models have 343 terms. By adding 
300 terms to the model, 300 degrees of freedom are lost resulting in no significant 
improvement in the standard error. 


32 



Component 

R 2 

<7 

PA 

0.4813 

0.5246 

QA 

0.4832 

0.6034 

RS 

0.5046 

0.5572 

RM 

0.5207 

0.5431 

CC 

0.4968 

0.5996 

CR 

0.4850 

0.6523 


Table 5. R 2 and o for Two Factor Interaction Models. 


Analysis of variance on the two factor interaction models causes rejection of the 
null hypothesis that all of the coefficients are equal to zero. The alternative hypothesis is 
accepted meaning that at least one coefficient is not equal to zero. Present in the anova 
tables is evidence that some of the terms are not necessary in the model and that 
simplification is possible. 

C. REDUCED MODELS WITHOUT INTERACTION 

The step function in S-plus is used to subtract terms from the simple models in an 
effort to simplify the models without losing too much of the information they provide. S- 
plus accomplishes this by using Akaike’s information criterion which is of the form: 

AIC =-2logL(x m+u ...,x n \xi, ...,x m ) + 2r 

where r is the total number of estimated parameters. The AIC is a value that penalizes a 
model for having high complexity when compared to simpler models with fewer terms. 


33 













The step function is applied to the simple models without interaction and the results are 
given in Appendix N. The reduced models have between 15 and 33 terms compared to 43 
terms for the original models. Model checking plots are constructed with no significant 
graphical differences between the reduced simple models and the simple models (see 
Appendices O-R). Values for the coefficient of determination and standard error are 
given in Table 6. As expected, less of the total variance is explained by the reduced 
models, and the value for a increases. 


Component 

R 2 

a 

PA 

0.0915 

0.5532 

QA 

0.1002 

0.6345 

RS 

0.1240 

0.5925 

RM 

0.1703 

0.5711 

CC 

0.1071 

0.6365 

CR 

0.0784 

0.6906 


Table 6. R 2 and a for Reduced Models without Interaction. 


D. REDUCED TWO FACTOR INTERACTION MODELS 

The step function in S-plus is applied to the two factor interaction models and the 
results are given in Appendix S. The data were fit to the new models and model checking 
plots are constructed and given in Appendices T-W. The scatterplot of the data with the 
superimposed regression line indicates that there might be a linear relationship between 
the factors and the score, but most of the data looks like a point cloud. The histograms of 


34 







the residuals have the appearance of a normal distribution. The predicted values versus 
the residuals have no pattern and the QQ-plots look normal. 

The reduced models have significantly decreased values for R 2 with little change 
in values for a. These models use between 33 and 59 terms which is a significant 
simplification over the 343 term models, but a lot of the explanation of the total variance 
is lost in the transition. Table 6 summarizes R 2 and a for the reduced models. 


Component 

R 2 

a 

PA 

0.1098 

0.5496 

QA 

0.1228 

0.6287 

RS 

0.1854 

0.5802 

RM 

0.2453 

0.5534 

CC 

0.1747 

0.6225 

CR 

0.1342 

0.6779 


Table 7. R 2 and a for Reduced Two Factor Models with Interaction. 

E. COMPARING MODELS 

The S-plus anova function is applied to pairs of models to see if they are 
statistically different. The results are given in Appendix U. At a = 0.05, eight pairs of 
models were found to be statistically similar, with the highest similarity between simple 
and reduced two term interaction models for PA and QA (p-Value > 0.99). Most of the 
model pairs are statistically different. 


35 














Choosing the most appropriate model for a complex data set is compromise 
between a model that explains enough of the data while being simple enough to use. The 
two factor interaction models are too complicated to be practical with 343 terms. The rest 
of the models are not different enough to distinguish in practice, so the models with the 
fewest terms are the preferred models, which in this case are the reduced simple models. 
Realistically, none of the models are useful. Not enough of the total variance in the data 
is explained nor is the range of expected scores reduced to a useful level. 

F. INTERVIEWS 

Subject matter experts interviewed agree that while all of the factors in the survey 
are important, additional factors could be added to more effectively group personnel and 
provide more information about the maintenance organization. They also think that 
dividing total years of aviation maintenance experience into two items, years worked in 
MOS and years worked outside of MOS is necessary to clarify that Total years of 
aviation maintenance experience is not simply time on active duty. The experts also 
indicate factors that look at levels of education and training and levels of morale and 
motivation are important and should be included in the demographic items of the survey. 
Responses from subject matter experts are given in Appendix Y. 


36 




V. CONCLUSIONS 


A. FINDINGS 

The results of this thesis show that at the first level of interaction, the 
demographic factors of the MCAS poorly account for the variance in the responses. The 
models constructed using linear regression and analysis of variance do not capture the 
responses of the surveyed population, showing that the demographic factors have low 
utility in data analysis. While analysis of variance shows that the models are preferred to 
no model at all, in use the models are too complex and do not provide enough insight into 
the surveyed group. 

Since the component scores are subjective perceptions, there is no correct score to 
any of the perception items. The reliance on the human component in the scoring is the 
cause of the large amount of variance, and since variance cannot be explained by the 
demographic factors, the MCAS appears to be demographically unbiased. The three 
dimensional bar plots of the component scores versus the levels of factors seem to 
support this (see Figure 4). Either the MCAS has insignificant biasing across factors or 
the present factors do not correctly group respondents to allow the biasing to be 
conclusively measured. 

Although the demographics do not effectively group respondents, they do provide 
information about the demographic composition of the surveyed group. This information 
alone can be useful to commanders in understanding the substance of their squadrons. To 
make these items more useful, changes to the MCAS demographic items based on the 
responses from subject matter experts are recommended below. 


37 



B. RECOMMENDATIONS 


With the MCAS moving to the internet, it will be possible to use more 
demographic factors than what is currently constrained by the layout of scannable 
computer response sheets. The MCAS demographic items should be changed to include 
the following items: 

1) Check the box corresponding to your community: 

2) Type in your unit number. 

3) Type in the number of months have you been with your current squadron. 

4) Type in the total number of maintenance activities to which you have been 

assigned. 

5) Type in the number of deployments you have made. 

6) Check the box corresponding to your rank. 

7) Type in the number years have you worked in your MOS. 

8) Type in the number of years have you worked outside of your MOS. 

9) Type in any supervisory designations that you hold. 

. 10) Check the box corresponding to your work center. 

11) Check the box corresponding to your shift. 

12) Have you attended A School? 

With these new questions, further analysis can be conducted to investigate for 
valid factors that properly describe the data in the responses, in addition to investigating 
if personal performance makes a difference in scoring. Additionally, MCAS could be 
modified to gauge safety climates in other military activities such as military ordnance 


38 




handling facilities and flight deck operations by adjusting the demographic items to suit 
those specific activities. 


39 





THIS PAGE INTENTIONALLY LEFT BLANK 



APPENDIX A. 43-ITEM MAINTENANCE CLIMATE ASSESSMENT SURVEY 

MAINTENANCE CLIMATE ASSESSMENT SURVEY (MCAS) 

Purpose : The MCAS was designed to capture maintainer perceptions of maintenance operations as they 
relate to safety. Your responses help guide Naval Aviation’s on-going efforts to reduce aviation related 
mishaps. Thank you in advance for your participation ! 

Directions : Do not write on this form. Fill in all of your responses using the computer sheet provided. 
Fill in each box that corresponds to your response completely using a pencil. This is not a timed event, so 
answer each question carefully and honestly. Individual responses will not be reported, only compiled 
results will be provided to each squadron. 

Part I - Demographics has six items requesting unit and biographical data. This information will aid in the 
response analysis. NO attempts will be made to identify individuals. 

Part II - Perceptions has 43 questions pertaining to the maintenance operations. Please choose the response 
to each item that most correctly reflects your honest opinion. Responses are: 

A- Strongly Agree B- Agree C- Neutral D- Disagree E- Strongly Disagree 

Part I- Demographics 

Line 1 Fill in the numbered circle corresponding to your community? 

(l)VMGR (2) VMA (3) VMFA (4) HMT (5) HMM 

(6) VMAQ (7) HMH (8) VMH (9) Other 

Line 2-4 Fill in the circles corresponding to your squadron number 

Line 5 Fill in the numbered circle corresponding with you rank 

(1) El-3 (2) E4-5 (3) E6-7 (4) E8-9 (5)W01-4 (6)01-03 (7)04-5 

Line 6 Fill in the numbered circle corresponding to your total years 

of Aviation Maintenance experience 
(1) <1 (2) 1-2 (3)3-5 (4)6-10 (5) 11-15 (6) 15-20 (7) 204- 

Line 7 Fill in the numbered circle corresponding to your work center 

(1) Power Plants (2) Airframes (3) Survival (4) Quality Assurance 
(5) Ordnance (6) Avionics (7) MAINT Control (8) Line (9) Other 

Line 8 Fill in the numbered circle corresponding to your shift 

(1) Day (2) Night 

Part II Perceptions 

Fill in the lettered circle that corresponds with your response to each item. 



SA 

A 

N 

D 

SD 

1 . The command adequately reviews and updates safety. 

(A) 

(B) 

(C) 

(D) 

(E) 


SA 

A 

N 

D 

SD 

2. The command monitors maintainer qualifications and has 

(A) 

(B) 

(C) 

(D) 

(E) 

a program that targets training deficiencies. 

SA 

A 

N 

D 

SD 

3. The command uses safety and medical staff to identify/ 

(A) 

(B) 

(C) 

(D) 

(E) 


41 


manage personnel at risk. 

4. CDIs/QARs routinely monitor maintenance evolutions. 

5. Tool control is taken seriously in the command and 
support equipment licensing is closely monitored. 

6. Signing off personnel qualifications are taken seriously. 

7. Our command climate promotes safe maintenance. 


8. Supervisors discourage SOP, NAMP, or other procedural 
violations and encourage reporting safety concerns. 

9. Peer influence discourages SOP, NAMP, or other 
violations and individuals feel free to report them. 

10. Violations of SOP, NAMP, or other procedures are not 
common in this command. 

11. The command recognizes individual safety achievement 
through rewards and incentives. 

12. Personnel are comfortable approaching supervisors 
about personal problems/illness 

13. Safety NCO, QAR, and CDI, are sought after billets. 


14. Unprofessional behavior is not tolerated in the command 


15. The command has a reputation for quality maintenance 
and sets standards to maintain quality control. 

16. QA and Safety are well respected, and are seen as 
essential to mission accomplishment. 

17. QARs/CDIs sign-off after required actions are complete 
and are not pressured by supervisors to sign-off. 

18. Maintenance on detachments is of the same quality 
as that at home station. 

19. Required publications/tools/equipment are available, 
current/serviceable, and used. 

20. QARs are helpful, and QA is not “feared” in my unit. 


21. Multiple job assignments and collateral duties adversely 
affect maintenance. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 


42 



22. Safety is part of maintenance planning, and additional 
training/support is provided as needed. 

23. Supervisors recognize unsafe conditions and manage 
hazards associated with maintenance and the flight-line. 

24. I am provided adequate resources, time, personnel to 
accomplish my job. 

25. Personnel turnover does not negatively impact the 
command’s ability to operate safely. 

26. Supervisors are more concerned with safe maintenance 
than the flight schedule, and do no permit cutting corners. 

27. Day/Night Check have equal workloads, and staffing is 
sufficient on each shift. 

28. Supervisors shield personnel from outside pressures and 
are aware of individual workload. 

29. Based upon my command’s current assets/manning it 
is not over-committed. 

30. My command temporarily restricts maintainers who are 
having a problems. 

31. Safety decisions are made at the proper levels and 
work center supervisor decisions are respected. 

32. Supervisors communicate command safety goals and 
are actively engaged in the safety program. 

33. Supervisors set the example for following to 
maintenance standards and ensure compliance. 

34. In my command safety is a key part of all maintenance 
operations, and all are responsible/accountable for safety. 

35. Safety education and training are comprehensive and 
effective. 

36. All maintenance evolutions are properly briefed, 
supervised, and staffed by qualified personnel. 

37. Maintenance Control is effective in managing all 
maintenance activities. 

38. Good communication exists up/down the chain of 
command. 

39. I get all the information I need to do my job safely. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 


43 


40. Work center supervisors coordinate their actions. 


41. My command has effective pass-down between shifts. 

42. Maintenance Control troubleshoots/resolves gripes before 
flight. 

43. Maintainers are briefed on potential hazards associated 
with maintenance activities. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 


44 



APPENDIX B. MODEL OF SAFETY EFFECTIVENESS COMPONENTS. 


COMPONENT 1: PROCESS AUDITING 

1. The command adequately reviews and updates safety practices. 

2. The command monitors maintainer qualifications and has a program that targets 
training deficiencies. 

3. The command uses safety and medical staff to identify/manage personnel at risk. 

4. CDIs/QARs routinely monitor maintenance evolutions. 

5. Tool Control is taken seriously in the command and support equipment licensing is 
closely monitored. 

6. Signing personal qualifications are taken seriously. 

COMPONENT 2: Reward System and Safety Climate 

1. Our command climate promotes safe maintenance and flight operations. 

2. Supervisors discourage SOP, NAMP or other procedure violations and encourage 
reporting safety concern. 

3. Peer influence discourages SOP, NAMP or other violations and individuals feel free 
to report them. 

4. Violations of SOP, NAMP or other procedures are not common in this command. 

5. The command recognizes individual safety achievement through rewards and 
incentives. 

6. Personnel are comfortable approaching supervisors about personal problems/illness. 

7. Safety NCO, QAR, and CDI, are sought after billets. 

8. Unprofessional behavior is not tolerated in the command. 

COMPONENT 3: QUALITY ASSURANCE 

1. The command has a reputation for quality maintenance and has set standards to 
maintain quality control. 

2. QA and Safety are well respected, and are seen as essential to mission 
accomplishment. 

3. QARs/CDIs sign-off after required actions are complete and are not pressured by 
supervisors to sign-off. 

4. Maintenance on detachments is the same quality as that at home station. 

5. Required publications/tools/equipment are available, current/serviceable, and used. 

6. QARs are helpful, and QA is not “feared” in my unit. 


45 


COMPONENT 4: RISK MANAGEMENT 


1. Multiple job assignments and collateral duties adversely affect maintenance. 

2. Safety is part of maintenance planning, and additional training/support is provided as 
needed. 

3. Supervisors recognize unsafe conditions and manage hazards associated with 
maintenance and the flight line. 

4. I am provided adequate resources, time, personnel to accomplish my job. 

5. Personnel turnover does not negatively impact the command’s ability to operate 
safely. 

6. Supervisors are more concerned with safe maintenance than the flight schedule, and 
do not permit cutting comers. 

7. Day/Night Check have equal workloads, and staffing is sufficient on each shift. 

8. Supervisors shield personnel from outside pressures and are aware of individual 
workload. 

9. Based upon my command’s current assets/manning it is not over-committed. 

COMPONENT 5: COMMAND AND CONTROL 

1. My command temporarily restricts maintainers who are having problems. 

2. Safety decisions are made at the proper levels, work center supervisors decisions are 
respected. 

3. Supervisors communicate command safety goals and are actively engaged in the 
safety program. 

4. Supervisors set the example for following to maintenance standards and ensure 
compliance. 

5. In my command safety is a key part of all maintenance operations and all are 
responsible/accountable for safety. 

6. Safety education and training are comprehensive and effective. 

7. All maintenance evolutions are properly briefed, supervised, and staffed by qualified 
personnel. 

8. Maintenance control is effective in managing all maintenance activities. 


COMPONENT 6: COMMUNICATION / FUNCTIONAL RELATIONSHIPS 

1. Good communication exists up/down the chain of command. 

2. I get all the information I need to do my job safely. 

3. Work center supervisors coordinate their actions. 

4. My command has effective pass-down between shifts. 

5. Maintenance Control troubleshoots/resolves gripes before flight. 

6. Maintainers are briefed on potential hazards associated with maintenance activities. 


46 










THIS PAGE INTENTIONALLY LEFT BLANK 


48 



APPENDIX D. SIMPLE MODELS: SCATTERPLOTS WITH SUPERIMPOSED 
REGRESSION LINE 


PA 


RS 




Fitted : {lactor(Community) + lactor(Squadron) + factor(Rank) + }actor{Tot.Years) + factor(Shop) + factorfSh Fitted : (factor(Community) + (aclor(Squadron) + factor(Rank) + )actor(Tot.Years) ♦ tactor(Shop) + Jador(Sh 


QA RM 




Fitted : (faetor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Sh Fitted : (factor(Community) + tactor(Squadron) + factor(Rank) + tactor{Tol.Years) + factor(Shop) + factor(Sh 


CC CR 




Fitted: (factor(Community) + tactor(Squadron) + factor (Rank) + factor(Tot. Years) + factor(Shop) + tactor(Sh Fitted : (factor(Community) + (actor(Squadron) + factor(Rank) + lactor(Tot.Years) + factor(Shop) + factor(Sh' 


49 










THIS PAGE INTENTIONALLY LEFT BLANK 


50 



100 200 300 ^ 0 100 200 300 


APPENDIX E. SIMPLE MODELS: HISTOGRAMS OF RESIDUALS 



51 





THIS PAGE INTENTIONALLY LEFT BLANK 


52 

















THIS PAGE INTENTIONALLY LEFT BLANK 


54 














THIS PAGE INTENTIONALLY LEFT BLANK 


56 





APPENDIX H. SIMPLE MODELS: ANALYSIS OF VARIANCE 


PA 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 8.3925 1.198929 3.922101 0.0003198 

factor(Squadron) 13 8.0763 0.621257 2.032343 0.0160499 

factor(Rank) 6 6.0359 1.005978 3.290893 0.0033082 

factor(Tot.Years) 6 3.4403 0.573388 1.875746 0.0821892 

factor(Shop) 9 3.2828 0.364753 1.193231 0.2957039 

factor(Shift) 1 2.7153 2.715335 8.882780 0.0029608 

Residuals 850 259.8325 0.305685 

QA 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 8.3910 1.198710 2.994574 0.0041284 

factor(Squadron) 13 14.3600 1.104618 2.759519 0.0007443 

factor(Rank) 6 12.2469 2.041156 5.099143 0.0000372 

factor(Tot.Years) 6 7.6351 1.272514 3.178950 0.0043234 

factor(Shop) 9 3.9649 0.440543 1.100548 0.3596172 

factor(Shift) 1 0.6428 0.642779 1.605767 0.2054349 

Residuals 850 340.2498 0.400294 

RS 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 3.7699 0.538562 1.549360 0.1471909 

factor(Squadron) 13 12.7604 0.981569 2.823824 0.0005582 

factor(Rank) 6 17.8319 2.971978 8.549925 0.0000000 

factor(Tot.Years) 6 7.6649 1.277490 3.675142 0.0013048 

factor(Shop) 9 5.4436 0.604846 1.740050 0.0761483 

factor(Shift) 1 1.7291 1.729082 4.974302 0.0259869 

Residuals 850 295.4624 0.347603 

RM 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 15.9912 2.284458 7.056521 0.0000000 

factor(Squadron) 13 11.8830 0.914080 2.823525 0.0005590 

factor(Rank) 6 19.2158 3.202630 9.892685 0.0000000 

factor(Tot.Years) 6 10.5421 1.757022 5.427311 0.0000161 

factor(Shop) 9 4.8880 0.543110 1.677627 0.0901813 

factor(Shift) 1 0.7475 0.747503 2.308980 0.1290003 

Residuals 850 275.1766 0.323737 

cc 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 9.8503 1.407189 3.490669 0.0010676 

factor(Squadron) 13 12.2826 0.944816 2.343707 0.0045142 

factor(Rank) 6 15.1026 2.517097 6.243904 0.0000020 

factor(Tot.Years) 6 6.7176 1.119607 2.777295 0.0111256 

factor(Shop) 9 3.5477 0.394188 0.977822 0.4567736 

factor(Shift) 1 2.8159 2.815870 6.985039 0.0083707 

Residuals 850 342.6594 0.403129 


Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 12.4423 1.777478 3.792880 0.0004599 

factor(Squadron) 13 11.0908 0.853140 1.820477 0.0361248 

factor(Rank) 6 18.6478 3.107964 6.631942 0.0000007 

factor(Tot.Years) 6 9.4231 1.570522 3.351265 0.0028616 

factor(Shop) 9 3.7036 0.411516 0.878115 0.5443707 

factor(Shift) 1 0.7389 0.738887 1.576677 0.2095850 

Residuals 850 398.3402 0.468636 


57 


THIS PAGE INTENTIONALLY LEFT BLANK 


58 




APPENDIX I. 


TWO FACTOR INTERACTION MODELS: SCATTERPLOTS 
WITH SUPERIMPOSED REGRESSION LINE 




Fitted : (factor(Commumty) + tactor(Squadron) + factor(Rank) + factor(Tot.Years) + tactor(Shop) + factor{Shil Fitted : (factor(Community} + tactor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shrt 


o oo> o o o o 
o o 

O O CD O O O 

OO O CD O OOOO < 

ODD 00 O 005 O O O 
OO O COdXDCOO OODO .OO 


o o ocomi 

® o ooaa 

O 00 OOOOJ 
O O OODMiOU 

o oo oo g^a— 
o o o o &um 


$ O ODD OCO 
OO <SD OOOOO .O' 
GD O OO O OOOO O 
OOO® O CBCD QO O 
00000)00 <WP OSOEMDO 


o ,ooo«m)ooo®o o 
o owijp OD QDO o to o o 
00 O O 00000 OOODO o 


00 35 ® 4 B 

o ooo oooot 
0^0 o <o < 




Fitted : (factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) faetor($hh Fitted : (faetor(Community) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shop) + tactor(Shil 


8 00 00 o o 

O ODOC 3 X> O .0 

o ottao o ao oca .o' 

o o ® aw^nnaiBOaD 10 

° oocdo°<^ 555 ^°^ o 

O 0 OO II 111 —l|W I m o 


O O tmaamn O 0 ,0' 
OO O OOOo ®D 

OOOO COCO OO O O O 
O MO O © OO 0,0 ' 

od a>8 8 do©o acoooo 


ODD 0.0 anODOOD ODD CD 

o oo o as o o 

oonomo oaoo o 

OOOO OOO OCCD o 

CO O CD O 


O COCOO OODCO OO ooo o o 

5 COB (DO rrri CD CO O O 
0 % OO &BDOOB> OOO 

Ooo 0 CD O O 0 O 




Fitted : (tactor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shi) Fitted: (factor(Community) + factor(Squadron) + factor(Rank) + 1actor{Tot.Years) + factor(Shop) *■ Jactor(Shil 











THIS PAGE INTENTIONALLY LEFT BLANK 


60 




100 300 0 100 300 0 50 100 200 


APPENDIX J. TWO FACTOR INTERACTION MODELS: HISTOGRAMS OF 
RESIDUALS 

PA QA 



-3 -2 -10 12 - 2-1012 

resid(cc.2.aov) resid(cr.2.aov) 


61 







THIS PAGE INTENTIONALLY LEFT BLANK 


62 



APPENDIX K. TWO FACTOR INTERACTION MODELS: RESIDUALS 
VERSUS FITTED VALUES WITH LOESS SMOOTHING 


QA 













THIS PAGE INTENTIONALLY LEFT BLANK 


64 













THIS PAGE INTENTIONALLY LEFT BLANK 


66 



APPENDIX M. TWO FACTOR INTERACTION MODELS: ANALYSIS OF 
VARIANCE 

PA 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 8.3925 1.198929 4.356653 0.0001027 

factor(Squadron) 13 8.0763 0.621257 2.257518 0.0068172 

factor(Rank) 6 6.0359 1.005978 3.655510 0.0014394 

factor(Tot.Years) 6 3.4403 0.573388 2.083571 0.0534953 

factor(Shop) 9 3.2828 0.364753 1.325436 0.2203764 

factor(Shift) 1 2.7153 2.715335 9.866954 0.0017732 

factor(Community):factor(Rank) 27 8.5470 0.316557 1.150301 0.2755360 

factor(Community):factor(Tot.Years) 32 5.5049 0.172029 0.625117 0.9480617 

factor(Community):factor(Shop) 46 16.8120 0.365478 1.328069 0.0778513 

factor(Community):factor(Shift) 5 4.8802 0.976038 3.546715 0.0036412 

factor(Squadron):factor(Rank) 27 7.1875 0.266205 0.967334 0.5134968 

factor(Squadron);factor(Tot.Years) 37 16.5409 0.447050 1.624486 0.0126771 

factor(Squadron):factor(Shop) 56 23.4663 0.419042 1.522709 0.0108588 

factor(Squadron):factor(Shift) 6 3.7496 0.624928 2.270855 0.0356434 

factor(Rank):factor(Tot.Years) 8 3.4152 0.426899 1.551259 0.1367135 

factor(Rank):factor(Shop) 15 5.1111 0.340743 1.238189 0.2382294 

factor(Rank):factor(Shift) 2 0.4713 0.235625 0.856213 0.4253331 

factor(Tot.Years):factor(Shop) 29 8.8454 0.305014 1.108357 0,3199687 

factor(Tot.Years):factor(Shift) 3 0.7165 0.238846 0.867915 0.4574968 

factor(Shop):factor(Shift) 7 3.2273 0.461047 1.675347 0.1124277 

Residuals 550 151.3572 0.275195 

QA 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 8.3910 1.198710 3.292186 0.0019394 

factor(Squadron) 13 14.3600 1.104618 3.033770 0.0002431 

factor(Rank) 6 12.2469 2.041156 5.605916 0.0000116 

factor(Tot.Years) 6 7.6351 1.272514 3.494886 0.0021165 

factor(Shop) 9 3.9649 0.440543 1.209925 0.2860380 

factor(Shift) 1 0.6428 0.642779 1.765354 0.1845090 

factor(Community):factor(Rank) 27 10.0763 0.373196 1.024962 0.4313503 

factor(Community):factor(Tot.Years) 32 10.2932 0.321663 0.883428 0.6538426 

factor(Community):factor(Shop) 46 23.9700 0.521087 1.431136 0.0362521 

factor(Community):factor(Shift) 5 11.2388 2.247751 6.173318 0.0000140 

factor(Squadron):factor(Rank) 27 17.7225 0.656389 1.802734 0.0083195 

factor(Squadron):factor(Tot.Years) 37 15.4891 0,418624 1.149727 0.2540719 

factor(Squadron):factor(Shop) 56 22.7429 0.406123 1.115393 0.2703154 

factor(Squadron):factor(Shift) 6 3.5604 0.593400 1.629737 0.1365990 

factor(Rank):factor(Tot.Years) 8 5.2739 0.659237 1.810557 0.0725066 

factor(Rank):factor(Shop) 15 7.4882 0.499213 1.371060 0.1561216 

factor(Rank):factor(Shift) 2 1.7580 0.879015 2.414162 0.0903895 

factor(Tot.Years):factor(Shop) 29 7.0553 0.243286 0.668170 0.9075797 

factor(Tot.Years):factor(Shift) 3 0.8364 0.278789 0.765679 0.5136070 

factor(Shop):factor(Shift) 7 2.4857 0.355105 0.975276 0.4483208 

Residuals 550 200.2591 0.364107 

RS 

Df Sum of Sq Mean Sq F Value Pr(F) 

factor(Community) 7 3.7699 0.538562 1.734800 0.0984123 

factor(Squadron) 13 12.7604 0.981569 3.161802 0.0001365 

factor(Rank) 6 17.8319 2.971978 9.573249 0.0000000 

factor(Tot.Years) 6 7.6649 1.277490 4.115013 0.0004719 

factor(Shop) 9 5.4436 0.604846 1.948314 0.0432207 

factor(Shift) 1 1.7291 1.729082 5.569667 0.0186225 

factor(Community):factor(Rank) 27 8.6010 0.318554 1.026117 0.4297560 

factor(Community):factor(Tot.Years) 32 5.9900 0.187188 0.602964 0.9597519 

factor(Community):factor(Shop) 46 19.4596 0.423035 1.362667 0.0607806 

factor(Community):factor(Shift) 5 6.8711 1.374220 4.426596 0.0005820 

factor(Squadron):factor(Rank) 27 12.1684 0.450680 1.451719 0.0672413 

factor(Squadron):factor(Tot.Years) 37 14.7707 0.399208 1.285918 0.1242051 

factor(Squadron):factor(Shop) 56 26.1193 0.466416 1.502405 0.0132179 

factor(Squadron):factor(Shift) 6 4.7507 0.791781 2.550461 0.0191331 

factor(Rank):factor(Tot.Years) 8 3.5565 0.444564 1.432017 0.1800429 

factor(Rank):factor(Shop) 15 4.1334 0.275561 0.887628 0.5783215 

factor(Rank):factor(Shift) 2 1.6918 0.845879 2.724720 0.0664497 

factor(Tot.Years):factor(Shop) 29 10.6388 0.366855 1.181702 0.2373524 

factor(Tot.Years):factor(Shift) 3 2.3787 0.792884 2.554013 0.0546501 

factor(Shop):factor(Shift) 7 3.5872 0.512457 1.650712 0.1187383 

Residuals 550 170.7454 0.310446 


67 


RM 


Df 

factor(Community) 7 
factor(Squadron) 13 
factor(Rank) 6 
factor(Tot.Years) 6 
factor(Shop) 9 
factor(Shift) 1 
factor(Community):factor(Rank) 27 
factor(Community):factor(Tot.Years) 32 
factor(Community):factor(Shop) 46 
factor(Community):factor(Shift) 5 

factor(Squadron):factor(Rank) 27 
factor(Squadron):factor(Tot.Years) 37 
factor(Squadron):factor(Shop) 56 
factor(Squadron):factor(Shift) 6 

factor(Rank):factor(Tot.Years) 8 
factor(Rank):factor(Shop) 15 
factor(Rank):factor(Shift) 2 
factor(Tot.Years):factor(Shop) 29 
factor(Tot.Years):factor(Shift) 3 

factor(Shot):factor(Shift) 7 
Residuals 550 

CR 

Df 

factor(Community) 7 
factor(Squadron) 13 
factor(Rank) 6 
factor(Tot.Years) 6 

factor(Shop) 9 
factor(Shift) 1 
factor(Community):factor(Rank) 27 
factor(Community):factor(Tot.Years) 32 
factor(Community):factor(Shop) 46 
factor(Community):factor(Shift) 5 
factor(Squadron):factor(Rank) 27 
factor(Squadron):factor(Tot.Years) 37 
factor(Squadron):factor(Shop) 56 
factor(Squadron):factor(Shift) 6 
factor(Rank):factor(Tot.Years) 8 
factor(Rank):factor(Shop) 15 
factor(Rank):factor(Shift) 2 

factor(Tot.Years):factor(Shop) 29 
factor(Tot.Years):factor(Shift) 3 
factor(Shop):factor(Shift) 7 
Residuals 550 

cc 

Df 

factor(Community) 7 
factor(Squadron) 13 
factor(Rank) 6 
factor(Tot.Years) 6 
factor(Shop) 9 
factor(Shift) 1 
factor(Community):factor(Rank) 27 
factor(Community):factor(Tot.Years) 32 
factor(Community):factor(Shop) 46 
factor(Community):factor(Shift) 5 
factor(Squadron):factor(Rank) 27 
factor(Squadron):factor(Tot.Years) 37 
factor(Squadron):factor(Shop) 56 
factor(Squadron):factor(Shift) 6 
factor(Rank):factor(Tot.Years) 8 

factor(Rank):factor(Shop) 15 
factor(Rank):factor(Shift) 2 
factor(Tot.Years):factor(Shop) 29 
factor(Tot.Years):factor(Shift) 3 

factor(Shop):factor(Shift) 7 

Residuals 550 


Sum of Sq Mean Sq F Value Pr(F) 
15.9912 2.284458 7.74590 0.0000000 

11.8830 0.914080 3.09937 0.0001810 

19.2158 3.202630 10.85914 0.0000000 
10.5421 1.757022 5.95752 0.0000048 

4.8880 0.543110 1.84152 0.0583837 

0.7475 0.747503 2.53455 0.1119530 

6.1172 0.226562 0.76820 0.7946819 

7.0593 0.220602 0.74799 0.8421223 

15.9480 0.346696 1.17554 0.2054702 

12.6939 2.538785 8.60824 0.0000001 

10.8087 0.400322 1.35737 0.1093488 

12.4608 0.336778 1.14191 0.2636165 

20.8355 0.372062 1.26155 0.1036550 

5.6544 0.942395 3.19537 0.0043116 

2.4103 0.301291 1.02158 0.4183172 

4.4618 0.297452 1.00857 0.4442584 

2.6308 1.315404 4.46013 0.0119820 

8.4000 0.289655 0.98213 0.4942530 

1.2055 0.401827 1.36247 0.2533738 

2.2818 0.325976 1.10528 0.3581240 

162.2087 0.294925 


Sum of Sq Mean Sq F Value Pr(F) 
12.4423 1.777478 4.177394 0.0001697 
11.0908 0.853140 2.005033 0.0185347 
18.6478 3.107964 7.304274 0.0000002 
9.4231 1.570522 3.691009 0.0013214 
3.7036 0.411516 0.967136 0.4662855 
0.7389 0.738887 1.736518 0.1881293 
10.0529 0.372329 0.875041 0.6494108 
10.0165 0.313016 0.735645 0.8559696 
30.3247 0.659233 1.549317 0.0137593 
8.1375 1.627495 3.824905 0.0020492 
14.7948 0.547957 1.287797 0.1526769 
19.5299 0.527835 1.240507 0.1600404 
36.1636 0.645778 1.517696 0.0114023 
3.4718 0.578626 1.359876 0.2288249 
4.5586 0.569829 1.339200 0.2212122 
6.5391 0.435941 1.024540 0.4275257 
1.2651 0.632559 1.486627 0.2270415 
14.0630 0.484930 1.139672 0.2827731 
2.2102 0.736717 1.731418 0.1594515 
3.1879 0.455411 1.070298 0.3811256 
234.0246 0.425499 


Sum of Sq Mean Sq F Value Pr(F) 
9.8503 1.407189 3.913888 0.0003535 
12.2826 0.944816 2.627865 0.0014467 
15.1026 2.517097 7.000932 0.0000003 
6.7176 1.119607 3.114022 0.0052210 
3.5477 0.394188 1.096376 0.3634308 
2.8159 2.815870 7.831926 0.0053132 
8.8346 0.327206 0.910074 0.5979349 
8.7848 0.274526 0.763554 0.8237228 
20.7928 0.452016 1.257217 0.1253496 
10.1673 2.033454 5.655750 0.0000426 
11.6354 0.430941 1.198599 0.2264270 
17.7331 0.479274 1.333030 0.0940917 
27.8577 0.497460 1.383610 0.0390184 
5.1754 0.862573 2.399120 0.0268520 
5.4297 0.678707 1.887723 0.0595526 
6.9073 0.460485 1.280770 0.2089887 
1.7807 0.890356 2.476393 0.0849826 
14.6705 0.505881 1.407033 0.0788824 
2.0737 0.691223 1.922534 0.1248178 
3.0709 0.438695 1.220166 0.2894603 
197.7456 0.359537 


68 



APPENDIX N. REDUCED SIMPLE MODELS 


Model: 

PA - factor(Squadron) + factor(Tot.Years) + factor(Shift) 


scale: 0.3056853 

Sum of Sq RSS Cp 

265.0655 281.5725 
19.09854 284.1640 289.0550 
6.98011 272.0456 284.8844 
2.98713 268.0526 283.9482 


Df 

<none> 

factor(Squadron) 19 
factor(Tot.Years) 6 
factor(Shift) 1 


Terms: 

Sum of Squares 
Deg. of Freedom 


factor(Squadron) 
16.4369 
19 


factor(Tot.Years) 
7.2861 
6 


factor(Shift) Residuals 
2.9871 265.0655 

1 866 


Residual standard error: 0.5532452 
Estimated effects may be unbalanced 


QA ~ factor(Squadron) + factor(Tot.Years) + factor(Shift) 


scale: 0.4002939 


Df 

<none> 

factor(Squadron) 19 
factor(Tot.Years) 6 
factor(Shift) 1 


Sum of Sq RSS Cp 

348.6818 370.2977 
23.06513 371.7469 378.1516 
15.45358 364.1354 380.9477 
0.90043 349.5822 370.3975 


Terms: 


factor(Squadron) 
Sum of Squares 22.4130 

Deg. of Freedom 19 


factor(Tot.Years) 
15.4952 
6 


factor(Shift) Residuals 
0.9004 348.6818 

1 866 


Residual standard error: 0.6345352 
Estimated effects may be unbalanced 


Model: 

RS ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shift) 


scale: 0.3476029 


Df 

<none> 

factor(Squadron) 19 
factor(Rank) 6 
factor(Tot.Years) 6 
factor(Shift) 1 


Sum of Sq RSS Cp 

301.9366 324.8784 
16.56125 318.4979 328.2308 
5.50006 307.4367 326.2072 
7.70716 309.6438 328.4143 
1.04953 302.9861 325.2327 


Terms: 


factor(Squadron) 
Sum of Squares 16.0094 

Deg. of Freedom 19 


factor(Rank) 
18.0278 
6 


factor(Tot.Years) 
7.6388 
6 


factor(Shift) Residuals 
1.0495 301.9366 

1 860 


Residual standard error: 0.5925277 
Estimated effects may be unbalanced 


Model: 

RM - factor(Squadron) + factor(Rank) + factor(Tot.Years) 


scale: 0.3237372 


<none> 


Df Sum of Sq RSS Cp 

280.8129 301.5321 


69 



factor(Squadron) 19 23.32159 304.1345 312.5517 
factor(Rank) 6 5.07801 285.8909 302.7253 

factor(Tot.Years) 6 10.54164 291.3546 308.1889 


Terms: 

Sum of Squares 
Deg. of Freedom 


factor{Squadron) 
27.8628 
19 


factor(Rank) 
19.2269 
6 


factor(Tot.Years) 
10.5416 
6 


Residuals 

280.8129 

861 


Residual standard error: 0.5710932 
Estimated effects may be unbalanced 


Model: 

CC - factor(Squadron) + factor(Tot.Years) + factor(Shift) 


scale: 0.4031287 


Df Sum of Sq RSS Cp 

<none> 350.8909 372.6598 

factor(Squadron) 19 22.16263 373.0535 379.5036 

factor(Tot.Years) 6 17.43463 368.3255 385.2569 

factor(Shift) 1 2.80445 353.6953 374.6580 


Terms: 

Sum of Squares 
Deg. of Freedom 


factor(Squadron) 
22.0953 
19 


factor(Tot.Years) 
17.1855 
6 


factor(Shift) 
2.8045 
1 


Residuals 

350.8909 

866 


Residual standard error: 0.636542 
Estimated effects may be unbalanced 


Model: 

CR ~ factor(Community) + factor(Tot.Years) + factor(Shift) 


scale: 0.4686355 


Df Sum of Sq RSS Cp 

<none> 418.7697 432.8288 

factor(Community) 7 10.24208 429.0118 436.5100 

factor(Tot.Years) 6 21.33850 440.1082 448.5437 

factor(Shift) 1 1.89642 420.6661 433.7879 


Terms: 

Sum of Squares 
Deg. of Freedom 


factor(Community) 
12.4423 
7 


factor(Tot.Years) 
21.2783 
6 


factor(Shift) 
1.8964 
1 


Residuals 

418.7697 

878 


Residual standard error: 0.690622 
Estimated effects may be unbalanced 


70 



APPENDIX O. REDUCED SIMPLE MODELS: SCATTERPLOTS WITH 
SUPERIMPOSED REGRESSION LINE 



Fitted : factor(Squadron) factor{Tot.Years) + tactor(Shitt) 


Fitted : fador(Community) + factor(Tot.Years) + factor{Shift) 






































































THIS PAGE INTENTIONALLY LEFT BLANK 


72 




100 200 300 0 100 200 300 0 100 200 300 


APPENDIX P. REDUCED SIMPLE MODELS: HISTOGRAMS OF RESIDUALS 





73 




THIS PAGE INTENTIONALLY LEFT BLANK 


74 



APPENDIX O. REDUCED SIMPLE MODELS: RESIDUALS VERSUS FITTED 


VALUES 



3.4 3.6 3.8 4.0 4.2 4.4 


Fitted: lactor(Squadron) + factorfTot. Years) + factor(Shift) 



3.0 3.2 3.4 3.6 3.8 4,0 4.2 

Fitted: factor(Squadron) + faetor(Rank) + factor(Tot.Years) + factor(Shift) 


cc 



3.2 3.4 3.6 3.8 4.0 4.2 

Fitted : faetor(Squadron) + factor(Tol.Years) + factor(Shift) 



3.2 3.4 3.6 3.8 4.0 

Fitted : factor(Community) + factorfTot.Years) + factor(Shift) 


75 





















THIS PAGE INTENTIONALLY LEFT BLANK 


76 



APPENDIX R. REDUCED SIMPLE MODELS: QQ-PLOTS 



of Standard Normal 


Quantiles of Standard Normal 












THIS PAGE INTENTIONALLY LEFT BLANK 


78 



APPENDIX S. REDUCED TWO FACTOR INTERACTION MODELS 


Model: 

PA - factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) + 
factor(Community):factor(Shift) 

scale: 0.2751949 


Df 

<none> 

factor(Squadron) 12 
factor(Tot.Years) 6 
factor(Community):factor(Shift) 5 


Sum of Sq RSS 

259.7296 
8.235337 267.9649 
7.737269 267.4669 
5.319236 265.0488 


Cp 

277.8925 

279.5231 

282.3274 

280.4597 


Model: 

QA - factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) + 
factor(Community):factor(Shift) 


scale: 0.3641075 


<none> 
factor(Squadron) 
factor(Tot.Years) 
factor(Community):factor(Shift) 


Df Sum of Sq RSS Cp 

339.8904 363.9215 
12 12.36975 352.2601 367.5526 

6 16.34776 356.2381 375.8999 

5 8.42356 348.3139 368.7039 


Model: 

RS - factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) + 
factor(Squadron):factor(Shift) + factor(Tot.Years):factor(Shift) 

scale: 0.3104461 


<none> 
factor(Rank) 
factor(Shop) 
factor(Squadron):factor(Shift) 
factor(Tot.Years):factor(Shift) 


Df Sum of Sq RSS 

280.7776 
6 5.38955 286.1672 

9 6.45093 287.2286 

12 11.55724 292.3349 

5 4.31919 285.0968 


Cp 

317.4103 

319.0745 

318.2732 

321.5168 

318.6250 


Model: 

RM - factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) + 
factor(Squadron):factor(Shift) + factor(Tot.Years):factor(Shift) 

scale: 0.2949249 


<none> 
factor(Rank) 
factor(Shop) 
factor(Squadron):factor(Shift) 
factor(Tot.Years):factor(Shift) 


Df Sum of Sq RSS 

255.4226 
6 3.99087 259.4134 

9 5.55127 260.9738 

12 17.43743 272.8600 

5 3.57653 258.9991 


Cp 

290.2237 

290.6755 

290.4663 

300.5829 

290.8510 


Model: 

CC - factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + 
factor(Shift) + factor(Community):factor(Shift) + factor(Squadron):factor(Shift) + 

factor(Rank):factor(Shift) + factor(Tot.Years):factor(Shift) 

scale: 0.3595374 


<none> 

factor(Community):factor(Shift) 
factor(Squadron):factor(Shift) 
factor(Rank):factor(Shift) 
factor(Tot.Years):factor(Shift) 


Df Sum of Sq RSS 

324.3285 
0 0.000000 324.3285 

7 5.543296 329.8718 

5 4.227122 328.5556 

5 5.275932 329.6045 


Cp 

364.5967 

364.5967 

365.1065 

365.2285 

366.2773 


79 



Model: 

CR ~ factor(Community) + factor(Rank) + factor(Tot.Years) + factor(Shift) + 
factor(Community):factor(Shift) + factor(Rank):factor(Shift) + factor(Tot.Years) 
factor(Shift) 

scale: 0.4254993 


<none> 

factor(Community):factor(Shift) 
factor(Rank):factor(Shift) 
factor(Tot.Years):factor(Shift) 


Df Sum of Sq 

6 9.058145 
5 7.047982 
5 7.483000 


RSS Cp 
393.4122 424.8992 
402.4703 428.8513 
400.4602 427.6921 
400.8952 428.1272 


80 



APPENDIX T. REDUCED TWO FACTOR INTERACTION MODELS: 

SCATTERPLOTS WITH SUPERIMPOSED REGRESSION 
LINE 


PA 


QA 




Fitted : factor(Community) + factor(Squadron) + 1actor(Tot. Years) + lactor(Shift) + factor(Community):1actor(S Fitted : (actor(Community) + factor(Squadron) + factor{Tot.Years) + factor(Shift) + 1actor(Community):<aetor(S 

RS RM 




iadron) + tactor(Rank) + factor(Tot.Years) + factor(Shop) + faetor(Shitt) + faclor{Squadron):lac1or(Shitt) + faclonadron) + factor(Rank) + faetor(Tot. Years) + factor(Shop) + factor(Shift) + iaclor(Squadron):factor(Shift) + factor 


cc 


CR 




lity) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shitt) + factor(Community):factor(Shift) + faoity) + factor(Rank) + factor(Tot.Years) + factor(Shift) + factor{Community):factor(Shift) + factor(Rank):factor(Sh 


81 








THIS PAGE INTENTIONALLY LEFT BLANK 


82 



APPENDIX U. REDUCED TWO FACTOR INTERACTION MODELS 
HISTOGRAMS OF RESIDUALS 























THIS PAGE INTENTIONALLY LEFT BLANK 


84 



APPENDIX V. REDUCED TWO FACTOR INTERACTION MODELS: 
RESIDUALS VERSUS FITTED VALUES 

PA QA 



2.0 2.5 3.0 3.5 4.0 4.S 1 2 3 4 

Fitted : factor(Community) + faetor(Squadron) + factor(Tot.Years) + lactor(Shift) + (aclor(Conimunity):»actor{S Fitted : factor(Comrminity) + factor(Squadron) + factor(Tot.Years) + factor(Shrft) + factor(Community):faetor(S 


RS RM 



2 3 4 5 1 2 3 4 

ladron) + factor(Rank) + tactor(Tot.Years) + tactor(Shop) + factor(Shift) + factor(Squadron):factor(Shrft) + factonadron) + factor(Rank) + (actor{Tot.Years) + factor(Shop) + factor(Shift) + factor(Squadron):factor(Shift) + factor 


CC CR 



itty) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shitl) + factor{Community):factor(Shift) +faoity) + faetor(Rank) + factor(Tot.Years) + factor(Shift) + factor(Community):tactor(Shift) * factor(Rank):factor(Sh 



85 












THIS PAGE INTENTIONALLY LEFT BLANK 


86 













THIS PAGE INTENTIONALLY LEFT BLANK 


88 



APPENDIX X. COMPARISONS ON MODELS USING ANOVA 


The values listed in the following tables are the probabilities that the models are 
equivalent. Values printed in bold type indicate similar models. 


Component 

Simple vs 
Two Factor 
Interaction 

Simple vs. 
Reduced 
Simple 

Simple vs. 
Reduced Two 
Factor 
Interaction 

PA 

0.00316 

0.37994 

0.99999 

QA 

0.00657 

0.17934 

0.99990 

RS 

0.00174 

0.04694 

0.00029 

RM 

0.00730 

0.09862 

1.86442e-7 

CC 

0.00156 

0.20517 

0.00001 

CR 

0.00580 

0.03360 

0.10600 


Component 

Two Factor 
Interaction 
vs. Reduced 
Simple 

Two Factor 
Interaction 
vs. Reducd 
Two Factor 
Interaction 

PA 

0.00324 

0.00788 

QA 

0.00488 

0.01596 

RS 

0.00087 

0.01474 

RM 

0.00470 

0.14662 

CC 

0.00121 

0.02202 

CR 

0.00200 

0.02111 


Component 

Reduced 
Simple vs. 
Reduced Two 
Factor 
Interaction 

PA 

0.00752 

QA 

0.00120 

RS 

0.00011 

RM 

3:50999e-7 

CC 

0.00008 

CR 

0.00016 


89 


























THIS PAGE INTENTIONALLY LEFT BLANK 


90 



APPENDIX Y. SUBJECT MATTER EXPERT RECOMMMENDED CHANGES 
TO MCAS DEMOGRAPHIC FACTORS 

Factors currently included in MCAS (no change): 

1. Community 

2. Squadron 

3. Rank 

4. Work Center 

5. Shift 

Items to modify in MCAS: * 

Total years of aviation maintenance experience (single item) 
modified to: 

6. Years worked in MOS 

7. Years worked outside MOS 

Factors to add to MCAS: 

education/training level 

8. attended an A school 

9. highest level of education attained 

10. number of maintenance activities assigned to during career 
morale/motivation indicators 

11. command advanced 

12. number of personal awards 

13. level of job satisfaction 

14. time to attain qualifications 

15. past performance on personal evaluations 

16. assigned to B tour 

other factors 

17. Age 

18. number of months in current squadron 

19. number of deployments 

20. supervisory designations earned 

21. level of confidence in Maintenance Control Officer 


91 



THIS PAGE INTENTIONALLY LEFT BLANK 


92 



LIST OF REFERENCES 


Allen, J., Rankin, B. & Sargent, B. (1998). Human Factors Process for Reducing 
Maintenance Errors [On-line]. Available: 

http://www.boeing.coni/commerciaI/aeromagazine/aero 03/textonlv/m01 txt.html 

Baker, R.H. (1998). Climate Survey Analysis for Aviation Maintenance Safety . Master’s 
Thesis, Naval Postgraduate School, Monterey, California. 

Brown, F.G. (1976). Principles of Educational and Psychological Testing . Holt, Rinehart 
and Winston. 

Booth, R. (1993). Safety Culture: Concept, Measurement and Training Implications. 
Proceedings of British Health and Safety Society Spring Conference: Safety Culture and 
the Management of Risk . 

Ciavarelli, A.P. (Nov 1997). Organizational Safety Effectiveness Model [On-line]. 
Available: http://web.nps.navv.mil/~avsafety/research/qmbmodel.htm 

Ciavarelli, A.P. & Figlock, R.C. (Nov 1997). Organizational Factors in Naval Aviation 
Accidents: Command Safety Assessment [On-line], Available: 
http://web.nps.navv.mil/~avsafetv/research/Qmbrept.htm 

Federal Aviation Administration Strategic Plan, 1998. 

Figlock, R. C. (1998). Qualitative Evaluation of a Notional Model of Organizational 
Safety Effectiveness and a Prototype Assessment Questionnaire . Unpublished Doctoral 
dissertation. 

Goodrum, B.W. (1999). Assessment of Maintenance Safety Climate in U.S. Navy Fleet 
Logistics Support Wing Squadrons . Masters Thesis, Naval Postgraduate School, 
Monterey, California. 

History [On-line], (April 2000). Available: http://www.faa.gov/historv.htm 

Human Factors in Aviation Maintenance and Inspection [On-line], (April 2000) 
Available: http://hfskyway.faa,gov/ 

Interview between C. Harris, Lieutenant, USN, Naval Postgraduate School, Monterey, 
California, and the author, 14 March 2000. 

Interview between J. Schmidt, Commander, MSC, USN, School of Aviation Safety, 
Naval Postgraduate School, Monterey, California, and the author, 17 December 1999. 


93 











Komamiski, A. (April 2000). Human Factors in Aircraft Maintenance. Microsoft 
Powerpoint presentation presented at the Advances in Aviation Safety Conference and 
Exposition, Daytona Beach, Florida. 

Libuser, C. B. (1994). Organizational structure and risk mitigation. Unpublished doctoral 
dissertation. University of California, Los Angeles. 

Maintenance and Inspection Human Factors Program [On-line]. (April 2000). Available: 
http://hfskyway.faa.gov/researchv3.htm 

Naval Safety Center Brief, Fiscal Year 1997. 

Naval Safety Center Brief, Fiscal Year 2000. 

Nutwell, R. & Sherman, K. (1997, March-April). Safety: Changing the Way We Operate. 
Naval Aviation News , 12-15. 

Oneto, T. J. (1999). Safety Climate Assessment in Naval Reserve Aviation Maintenance 
Operations. Master’s Thesis, Naval Postgraduate School, Monterey, California. 

OPNAV INSTRUCTION 3750.60, The Naval Aviation Safety Program. Department of 
the Navy, 1989. 

Perrow, C. (1984). Normal Accidents. New York: Basic Books. 

Reason, J. (1990). Human Error . Cambridge: University Press. 

Reason, J. (1997). Managing the Risks of Organizational Accidents . Brookfield: Ashgate 
Publishing Company. 

Redmill, F. & Rajan, J. (1997). Human Factors in Safety-critical Systems . Oxford: 
Butterworth-Heinemann. 

Roberts, K.H. (1990). Some characteristics of one type of high reliability organization. 
Organizational Science . 1(2), 160-176. 

Roberts, K.H. (1990). Managing high-reliability organizations. California Management 
Review . 32(4), 101-113. 

Safety, Security and System Efficiency [On-line]. (April 2000). Available: 
http://www.faa. gov/safetyinfo2. htm 


94 



Schmidt, J., Schmorrow, D., & Hardee, M. (1998). A Preliminary Human Factors 
Analysis of Naval Aviation Maintenance Related Mishaps. Proceedings of the Society of 
Automotive Engineers, Airframes. Engines, Maintenance, and Repair Conference . (Paper 
Number 983 111.) 

Schmorrow, D. D. (1998). Human Error Analysis and Model of Aviation Maintenance 
Related Mishaps , Master’s Thesis, Naval Postgraduate School, Monterey, California. 

United States General Accounting Office (1993). Program Evaluation and Methodology 
Division. Developing and Using Questionnaires. 

Uttal, B. The corporate culture vultures. Fortune , 17 October 1983. 

Weick, Karl E. (1987). Organizational culture as a source of high reliability. California 
Management Review . 29(2), 112-127. 

Wogalter, M., Allison, S. & McKenna, N. (1989). Effects of cost and social influence on 
warning compliance. Human Factors . 31(2), 133-140. 

Wickens, C. D., Gordon, S. E. & Lui, Y. (1997). An Introduction to Human Factors 
Engineering . New York: Longman, Inc. 


95 



THIS PAGE INTENTIONALLY LEFT BLANK 


96 



INITIAL DISTRIBUTION LIST 


1. Defense Technical Information Center.2 

8725 John J. Kingman Rd„ STE 0944 

Ft. Bel voir, Virginia 22060-6218 

2. Dudley Knox Library.2 

Naval Postgraduate School 

411 Dyer Rd. 

Monterey, California 93943-5101 

3. Defense Logistic Studies Information Exchange..1 

U.S. Army Logistics Management College 

Fort Lee, Virginia 23801-6043 

4. Professor Robert R. Read, Code OR/Re.2 

Department of Operations Research 

Naval Postgraduate School 
Monterey, California 93943 

5. CDR John K. Schmidt.3 

School of Aviation Safety 

Naval Postgraduate School 
Monterey, California 93943 

6. Professor Lyn R. Whitaker, Code OR/Wh.1 

Department of Operations Research 

Naval Postgraduate School 
Monterey, California 93943 

7. LT Bruce R. Stanley, Jr.2 

3753 Stanford Drive 

Oceanside, California 92056 


97