Verification, Validation, and Testing of Engineered Systems

ISBN-13: 9780470527511 / Angielski / Twarda / 2010 / 720 str.

Avner Engel; A. Engel

Opis

Systems' Verification Validation and Testing (VVT) are carried out throughout systems' lifetimes. Notably, quality-cost expended on performing VVT activities and correcting system defects consumes about half of the overall engineering cost. Verification, Validation and Testing of Engineered Systems provides a comprehensive compendium of VVT activities and corresponding VVT methods for implementation throughout the entire lifecycle of an engineered system. In addition, the book strives to alleviate the fundamental testing conundrum, namely: What should be tested? How should one test? When should one test? And, when should one stop testing? In other words, how should one select a VVT strategy and how it be optimized? The book is organized in three parts: The first part provides introductory material about systems and VVT concepts. This part presents a comprehensive explanation of the role of VVT in the process of engineered systems (Chapter-1). The second part describes 40 systems' development VVT activities (Chapter-2) and 27 systems' post-development activities (Chapter-3). Corresponding to these activities, this part also describes 17 non-testing systems' VVT methods (Chapter-4) and 33 testing systems' methods (Chapter-5). The third part of the book describes ways to model systems' quality cost, time and risk (Chapter-6), as well as ways to acquire quality data and optimize the VVT strategy in the face of funding, time and other resource limitations as well as different business objectives (Chapter-7). Finally, this part describes the methodology used to validate the quality model along with a case study describing a system's quality improvements (Chapter-8). Fundamentally, this book is written with two categories of audience in mind. The first category is composed of VVT practitioners, including Systems, Test, Production and Maintenance engineers as well as first and second line managers. The second category is composed of students and faculties of Systems, Electrical, Aerospace, Mechanical and Industrial Engineering schools. This book may be fully covered in two to three graduate level semesters; although parts of the book may be covered in one semester. University instructors will most likely use the book to provide engineering students with knowledge about VVT, as well as to give students an introduction to formal modeling and optimization of VVT strategy.

Spis treści

Preface xvii

Part I Introduction 1

1. Introduction 3

1.1 Opening 3

1.1.1 Background 4

1.1.2 Purpose 5

1.1.3 Intended audience 5

1.1.4 Book structure and contents 6

1.1.5 Scope of application 8

1.1.6 Terminology and notation 9

1.2 VVT Systems and Process 9

1.2.1 Introduction—VVT systems and process 9

1.2.2 Engineered systems 10

1.2.3 VVT concepts and defi nition 12

1.2.4 The fundamental VVT dilemma 19

1.2.5 Modeling systems and VVT lifecycle 20

1.2.6 Modeling VVT and risks as cost and time drivers 24

1.3 Canonical Systems VVT Paradigm 32

1.3.1 Introduction—Canonical systems VVT paradigm 32

1.3.2 Phases of the system lifecycle 34

1.3.3 Views of the system 37

1.3.4 VVT aspects of the system 39

1.4 Methodology Application 39

1.4.1 Introduction 39

1.4.2 VVT methodology overview 40

1.4.3 VVT tailoring 43

1.4.4 VVT documents 50

1.5 References 56

Part II VVT Activities and Methods 61

2. System VVT Activities: Development 63

2.1 Structure of Chapter 63

2.1.1 Systems development lifecycle phases and VVT activities 63

2.1.2 VVT activity aspects 64

2.1.3 VVT activity format 65

2.2 VVT Activities during Defi nition 65

2.2.1 Generate Requirements Verifi cation Matrix (RVM) 65

2.2.2 Generate VVT Management Plan (VVT–MP) 67

2.2.3 Assess the Request For Proposal (RFP) document 69

2.2.4 Assess System Requirements Specifi cation (SysRS) 71

2.2.5 Assess project Risk Management Plan (RMP) 72

2.2.6 Assess System Safety Program Plan (SSPP) 74

2.2.7 Participate in System Requirements Review (SysRR) 77

2.2.8 Participate in System Engineering Management Plan (SEMP) review 77

2.2.9 Conduct engineering peer review of the VVT–MP document 79

2.3 VVT Activities during Design 80

2.3.1 Optimize the VVT strategy 80

2.3.2 Assess System/Subsystem Design Description (SSDD) 83

2.3.3 Validate system design by means of virtual prototype 85

2.3.4 Validate system design tools 86

2.3.5 Assess system design for meeting future lifecycle needs 87

2.3.6 Participate in the System Design Review (SysDR) 90

2.4 VVT Activities during Implementation 91

2.4.1 Preparing the test cycle for subsystems and components 91

2.4.2 Assess suppliers’ subsystems test documents 96

2.4.3 Perform Acceptance Test Procedure—Subsystems/Enabling products 97

2.4.4 Assess system performance by way of simulation 100

2.4.5 Verify design versus implementation consistency 102

2.4.6 Participate in Acceptance Test Review—Subsystems/Enabling products 103

2.5 VVT Activities during Integration 104

2.5.1 Develop System Integration Laboratory (SIL) 104

2.5.2 Generate System Integration Test Plan (SysITP) 106

2.5.3 Generate System Integration Test Description (SysITD) 108

2.5.4 Validate supplied subsystems in a stand–alone configuration 111

2.5.5 Perform components, subsystem, enabling products integration tests 112

2.5.6 Generate System Integration Test Report (SysITR) 114

2.5.7 Assess effectiveness of the system Built In Test (BIT) 116

2.5.8 Conduct engineering peer review of the SysITR 120

2.6 VVT Activities during Qualification 120

2.6.1 Generate a qualification/acceptance System Test Plan (SysTP) 121

2.6.2 Create qualification/acceptance System Test Description (SysTD) 123

2.6.3 Perform virtual system testing by means of simulation 125

2.6.4 Perform qualification testing/Acceptance Test Procedure (ATP)—System 126

2.6.5 Generate qualification/acceptance System Test Report (SysTR) 129

2.6.6 Assess system testability, maintainability and availability 131

2.6.7 Perform environmental system testing 137

2.6.8 Perform system Certification and Accreditation (C&A) 140

2.6.9 Conduct Test Readiness Review (TRR) 144

2.6.10 Conduct engineering peer review of development enabling products 146

2.6.11 Conduct engineering peer review of program and project safety 148

2.7 References 149

3. Systems VVT Activities: Post–Development 153

3.1 Structure of Chapter 153

3.2 VVT Activities during Production 154

3.2.1 Participate in Functional Confi guration Audit (FCA) 154

3.2.2 Participate in Physical Confi guration Audit (PCA) 157

3.2.3 Plan system production VVT process 159

3.2.4 Generate a First Article Inspection (FAI) procedure 161

3.2.5 Validate the production–line test equipment 165

3.2.6 Verify quality of incoming components and subsystems 165

3.2.7 Perform First Article Inspection (FAI) 166

3.2.8 Validate pre–production process 167

3.2.9 Validate ongoing–production process 168

3.2.10 Perform manufacturing quality control 170

3.2.11 Verify the production operations strategy 172

3.2.12 Verify marketing and production forecasting 174

3.2.13 Verify aggregate production planning 176

3.2.14 Verify inventory control operation 177

3.2.15 Verify supply chain management 180

3.2.16 Verify production control systems 181

3.2.17 Verify production scheduling 183

3.2.18 Participate in Production Readiness Review (PRR) 184

3.3 VVT Activities during Use/Maintenance 186

3.3.1 Develop VVT plan for system maintenance 187

3.3.2 Verify the Integrated Logistics Support Plan (ILSP) 191

3.3.3 Perform ongoing system maintenance testing 200

3.3.4 Conduct engineering peer review on system maintenance process 204

3.4 VVT Activities during Disposal 208

3.4.1 Develop VVT plan for system disposal 209

3.4.2 Assess the system disposal plan 212

3.4.3 Assess system disposal strategies by means of simulation 214

3.4.4 Assess on–going system disposal process 215

3.4.5 Conduct engineering peer review to assess system disposal processes 219

3.5 References 221

4. System VVT Methods: Non–Testing 223

4.1 Introduction 223

4.2 Prepare VVT Products 223

4.2.1 Requirements Verification Matrix (RVM) 223

4.2.2 System Integration Laboratory (SIL) 226

4.2.3 Hierarchical VVT optimization 230

4.2.4 Defect management and tracking 234

4.2.5 Classification Tree Method 239

4.2.6 Design of Experiments (DOE) 243

4.3 Perform VVT Activities 256

4.3.1 VVT process planning 256

4.3.2 Compare images and documents 262

4.3.3 Requirements testability and quality 265

4.3.4 System test simulation 272

4.3.5 Failure mode effect analysis 280

4.3.6 Anticipatory Failure Determination 286

4.3.7 Model–based testing 293

4.3.8 Robust design analysis 302

4.4 Participate in Reviews 312

4.4.1 Expert team reviews 312

4.4.2 Formal technical reviews 326

4.4.3 Group evaluation and decision 331

4.5 References 346

5. Systems VVT Methods: Testing 351

5.1 Introduction 351

5.2 White Box Testing 356

5.2.1 Component and code coverage testing 356

5.2.2 Interface testing 360

5.3 Black Box—Basic Testing 365

5.3.1 Boundary value testing 365

5.3.2 Decision table testing 367

5.3.3 Finite State Machine testing 368

5.3.4 Human–system interface testing (HSI) 373

5.4 Black Box—High–Volume Testing 378

5.4.1 Automatic random testing 378

5.4.2 Performance testing 381

5.4.3 Recovery testing 385

5.4.4 Stress testing 386

5.5 Black Box—Special Testing 388

5.5.1 Usability testing 388

5.5.2 Security vulnerability testing 393

5.5.3 Reliability testing 402

5.5.4 Search–based testing 410

5.5.5 Mutation testing 418

5.6 Black Box—Environment Testing 422

5.6.1 Environmental Stress Screening (ESS) testing 422

5.6.2 EMI/EMC testing 424

5.6.3 Destructive testing 426

5.6.4 Reactive testing 431

5.6.5 Temporal testing 436

5.7 Black Box—Phase Testing 443

5.7.1 Sanity testing 444

5.7.2 Exploratory testing 445

5.7.3 Regression testing 447

5.7.4 Component and subsystem testing 452

5.7.5 Integration testing 455

5.7.6 Qualification testing 461

5.7.7 Acceptance testing 463

5.7.8 Certification and accreditation testing 466

5.7.9 First Article Inspection (FAI) 473

5.7.10 Production testing 477

5.7.11 Installation testing 481

5.7.12 Maintenance testing 484

5.7.13 Disposal testing 487

5.8 References 488

Part III Modeling and Optimizing VVT Process 495

6. Modeling Quality Cost, Time and Risk 497

6.1 Purpose and Basic Concepts 497

6.1.1 Historical models for cost of quality 498

6.1.2 Quantitative models for cost/time of quality 499

6.2 VVT Cost and Risk Modeling 500

6.2.1 Canonical VVT cost modeling 500

6.2.2 Modeling VVT strategy as a decision problem 502

6.2.3 Modeling appraisal risk cost 505

6.2.4 Modeling impact risk cost 511

6.2.5 Modeling total quality cost 516

6.2.6 VVT cost and risk example 517

6.3 VVT Time and Risk Modeling 521

6.3.1 System/VVT network 521

6.3.2 Modeling time of system/VVT lifecycle 524

6.3.3 Time and risk example 528

6.4 Fuzzy VVT Cost Modeling 530

6.4.1 Introduction 530

6.4.2 General fuzzy logic modeling 530

6.4.3 Fuzzy modeling of the VVT process 532

6.4.4 Fuzzy VVT cost and risk estimation example 541

6.4.5 Fuzzy logic versus probabilistic modeling 544

6.5 References 548

7. Obtaining Quality Data and Optimizing VVT Strategy 550

7.1 Systems’ Quality Costs in the Literature 550

7.2 Obtaining System Quality Data 554

7.2.1 Quality data acquisition 554

7.2.2 Quality data aggregation 555

7.3 IAI/Lahav Quality Data—An Illustration 557

7.3.1 IAI/Lahav pilot project 557

7.3.2 Obtaining raw system and quality data 559

7.3.3 Anchor system and quality data 560

7.3.4 Generating the VVT model database 561

7.4 The VVT–Tool 562

7.4.1 Background 562

7.4.2 Tool availability 563

7.5 VVT Cost, Time and Risk Optimization 564

7.5.1 Optimizing the VVT process 565

7.5.2 Loss function optimization—VVT cost 569

7.5.3 Weight optimization—VVT cost 576

7.5.4 Goal optimization—VVT cost 580

7.5.5 Genetic algorithm optimization—VVT time 584

7.5.6 Genetic multi–domain optimization—VVT cost and time 596

7.6 References 600

8. Methodology Validation and Examples 604

8.1 Methodology Validation Using a Pilot Project 604

8.1.1 VVT cost model validation 605

8.1.2 VVT time model validation 610

8.1.3 Fuzzy VVT cost model validation 617

8.2 Optimizing the VVT Strategy 618

8.2.1 Analytical optimization of cost 619

8.2.2 Cost distribution by phase 626

8.2.3 Weight optimization of cost 627

8.2.4 Goal optimization of cost 631

8.2.5 MPGA optimization for time 635

8.2.6 SSGA optimization of cost and time 637

8.3 Identifying and Avoiding Signifi cant Risks 639

8.3.1 Avoiding critical risks 640

8.3.2 Conjecture on future risk scenarios 642

8.4 Improving System Quality Process 644

Appendix A SysTest Project 646

A.1 About SysTest 646

A.2 SysTest Key Products 648

A.3 SysTest Pilot Projects 649

A.4 SysTest Team 653

A.5 EC Evaluation of SysTest Project 655

References 656

Appendix B Proposed Guide: System Verification, Validation and Testing Master Plan 657

B.1 Background 657

B.2 Creating the VVT–MP 658

B.3 Chapter 1: System Description 659

B.3.1 Project applicable documents 659

B.3.2 Mission description 659

B.3.3 System description 659

B.3.4 Critical technical parameters 660

B.4 Chapter 2: Integrated VVT Program Summary 660

B.4.1 Integrated VVT program schedule 660

B.4.2 VVT program management 661

B.5 Chapter 3: System VVT 662

B.5.1 VVT strategy 662

B.5.2 Planning VVT activities 665

B.5.3 VVT limitations 668

B.6 Chapter 4: VVT Resource Summary 669

B.6.1 Test articles 669

B.6.2 Test sites and instrumentation 669

B.6.3 Test support requisition 669

B.6.4 Expendables for testing 669

B.6.5 Operational force test support 670

B.6.6 Simulations, models and test beds 670

B.6.7 Manpower/personnel needs and training 670

B.6.8 Budget summary 670

Appendix C List of Acronyms 671

Index 679

O autorze

Dr. Avner Engel holds a PhD from the Industrial Engineering Department of Tel–Aviv University. For the past twenty years, he has worked for Israel Aerospace Industries, where he has managed large software projects. Dr. Engel was involved with several research projects funded by the European Commission. He is currently teaching systems engineering courses at the Holon Institute of Technology in Holon, Israel.

Blurb

A comprehensive collection of VVT activities and methods for system–wide implementation

Verification, Validation, and Testing (VVT) is of extreme importance to systems engineering, where up to 60 percent of systems development cost is expended on VVT activities or correcting system defects. Verification, Validation, and Testing of Engineered Systems is the first resource to explain this process in a comprehensive, implementable manner. Through a practical approach, the text presents VVT activities performable throughout a system′s lifetime, from system definition and design to system use and disposal.

Beginning with a thorough explanation of the role of VVT in the process of engineered systems, the book provides a much–needed quantitative, credible model that answers the key questions of what, how, and when one should test, as well as when to stop testing. It equips both new and experienced readers with statistical and fuzzy logic paradigms for quantitative VVT cost, time, and risk models to minimize uncertainties and risks in systems development.

• Explains the essence of systems′ VVT and the linkage between VVT and systems development, manufacturing, use/maintenance and retirement

• Includes systems′ development and post–development VVT activities, as well as non–testing and testing systems′ methods for engineered systems

• Reveals how to acquire quality data and optimize the VVT strategy in the face of limitations and in accordance with different business objectives

• Describes the methodology used to validate the quality model along with examples outlining a system′s quality improvements

• Presents actual quality data related to engineered systems as measured in various industries

Verification, Validation, and Testing of Engineered Systems aids systems and test engineers as well as first– and second–line managers working in systems development and manufacturing industries, civilian agencies, or the military. It can be used as a textbook in graduate–level courses in systems, electrical, aerospace, mechanical, and industrial engineering.