Title Details

Mark A. Levin is the Reliability Manager for Product Development at Teradyne, Inc, USA. He has over 36 years of electronics experience working in manufacturing, design and research.

Ted T. Kalal is a retired reliability Manager. He has held many positions as a contract engineer and consultant where he focused on design, quality and reliability tasks.

About The Authors

Series Foreword Second Edition

Preface Second Edition

Series Foreword First Edition

Foreword First Edition

Preface First Edition

List Of Acronyms

1The Need For A New Paradigm For Hardware Reliability And Software Quality 36

1.1 Gaining Competitive Advantage 4

1.2 Competing In The Next Decade – Winners Will Compete On Reliability 4

1.3 Concurrent Engineering 5

1.4 Reducing The Number Of Engineering Change Orders (ECOS) At Product Release 7

1.5 Time-To-Market Advantage 7

1.6 Accelerating Product Development 9

1.7 Identifying And Managing Risks 10

1.8 ICM, A Process To Mitigate Risk 10

1.9 Software Quality Overview 11

1.9.1 Software Glossary 13

2 Barriers To Implementing Hardware Reliability And Software Quality 14

2.1 Lack Of Understanding 14

2.2 Internal Barriers 15

2.3 Implementing Change And Change Agents 16

2.4 Building Credibility 18

2.5 Perceived External Barriers 19

2.6 It Takes Time To Gain Acceptance 20

2.7 External Barrier 21

2.8 Barriers To Software Process Improvement 22

3 Understanding Why Products Fail 24

3.1 Why Things Fail 24

3.2 Parts Have Improved, Everyone Can Build Quality Products 26

3.3 Hardware Reliability And Software Quality – The New Paradigm 27

3.4 Reliability VS Quality Escapes 28

3.5 Why Software Quality Improvement Programs Are Unsuccessful 28

4 Alternative Approaches To Implementing Reliability 31

4.1 Hiring Consultants For HALT Testing 31

4.2 Outsourcing Reliability Testing 31

4.3 Using Consultants To Develop And Implement A Reliability Program 32

4.4 Hiring Reliability Engineers 32

5 The Product Life Cycle 35

5.1 Six Phases Of The Product Life Cycle 35

5.1.1 Mitigate Risk 36

5.2 The ICM Process For A Small Company 42

5.2.1 DFX – Design For Manufacturability (DFM), Design For Test (DFT), Design For Serviceability (DFS) And Maintainability, And Design For Reliability (DFR) 43

5.2.2 Warranty 43

6 Reliability Concepts 46

6.1 The Bathtub Curve 47

6.2 Mean Time Between Failure  48

6.2.1 Mean Time Between Repair 49

6.2.2 Mean Time Between Maintenances (MTBM) 49

6.2.3 Mean Time Between Incident (MTBI) 49

6.2.4 Mean Time To Failure (MTTF) 49

6.2.5 Mean Time To Repair (MTTR) 49

6.2.6 Mean Time To Restore System (MTTRS) 49

6.3 Warranty Costs 50

6.4 Availability 51

6.4.1 On-Site Manufacturer Service Personnel 53

6.4.2 Customer Trained Service Personnel 53

6.4.3 Manufacturer Training For Customer Service Personnel 53

6.4.4 Easy-To-Use Service Manuals 53

6.4.5 Rapid Diagnosis Capability 53

6.4.6 Repair And Spare Parts Availability 53

6.4.7 Rapid Response To Customer Requests For Service 54

6.4.8 Failure Data Tracking 54

6.5 Reliability Growth 55

6.6 Reliability Demonstration Testing 56

6.7 Maintenance And Availability 59

6.7.1 Preventative Maintenance  60

6.7.2 Predictive Maintenance 61

6.7.3 Prognostic And Health Management (PHM) 62

6.8 Component Derating 66

6.9 Component Uprating 68

7 FMEA 71

7.1 The FMEA Process 71

7.1.1 The Functional Block Diagram (FBD) 72

7.1.2 The Fault Tree Analysis 76

7.1.3 Failure Modes And Effects Analysis Spreadsheet 79

7.1.4 Preparing For The FMEA 83

7.1.5 Barriers To The FMEA Process 85

7.1.6 FMEA Ground Rules 88

7.1.7 Ways To Improve FMEA Efficiency And Effectiveness 90

7.1.8 Software FMEA 92

7.1.9 Software Fault Tree Analysis (SFTA) 96

7.1.10 Process FMEA 97

7.1.11 FMMEA 98

8 The Reliability Toolbox 100

8.1 The HALT Process 100

8.1.1 Types Of Stresses Applied In HALT 103

8.1.2 The Theory Behind The HALT Process 104

8.1.3 HALT Testing Liquid Cooled Products 107

8.1.4 HALT Testing 108

8.2 Highly Accelerated Stress Screening (HASS) 120

8.2.1 Proof Of Screen (POS) 122

8.2.2 Burn-In 123

8.2.3 Environmental Stress Screening (ESS) 124

8.2.4 Economic Impact Of Hass 125

7.3.5 The HASA Process 125

8.3 Summary Of HALT, HASS, HASA And POS Benefits 126

8.4 HALT And Hass Test Chambers 126

8.5 Accelerated Reliability Growth (ARG) 127

8.6 Accelerated Early Life Test (ELT) 131

8.7 Spc Tool 131

8.8 Fifo Tool 132

9 Software Quality Goals And Metrics 137

9.1 Software Metrics 139

9.2 Lines of Code (LOC) 140

9.3 Defect Density 141

9.4 Defect Models 143

9.5 Defect Run Chart 144

9.6 Escaped Defect Rate 146

9.7 Code Coverage 147

10 Software Quality Analysis Techniques 149

10.1 Root Cause Analysis 149

10.2 The 5 Whys 149

10.3 Cause And Effect Diagrams 150

10.4 Pareto Charts 151

10.5 Defect Prevention, Defect Detection And Defensive Programming 153

10.6 Effort Estimation 156

11 Software Lifecycles 158

11.1 Waterfall 158

11.2 Agile 160

11.3 CMMI 163

11.4 How To Choose A Software Lifecycle 166

12 Software Procedures And Techniques 168

12.1 Gathering Requirements 168

12.2 Documenting Requirements 170

12.3 Documentation 173

12.4 Code Comments 175

12.5 Reviews And Inspections 178

12.6 Traceability 182

12.7 Software And Hardware Integration 182

13 Why Hardware Reliability And Software Quality Improvement Efforts Fail 185

13.1 Lack Of Commitment To The Reliability Process 185

13.2 Inability To Embrace And Mitigate Technologies Risk Issues 187

13.3 Choosing The Wrong People For The Job 188

13.4 Inadequate Funding 188

13.5 Inadequate Resources 193

13.6 Mil-Std 217 - Why They Are Obsolete 194

13.7 Finding But Not Fixing Problems 197

13.8 Nondynamic Testing 198

13.9 Vibration Testing Too Difficult To Implement 198

13.10 The Impact Of Late Hardware Or Late Software Delivery 198

13.11 Supplier Reliability 198

14 Supplier Management 200

14.1 Purchasing Interface 200

14.2 Identifying Your Critical Suppliers 201

14.3 Develop A Thorough Supplier Audit Process  201

14.4 Develop Rapid Nonconformance Feedback 202

14.5 Develop A Materials Review Board (MRB) 203

14.6 Counterfeit Parts And Materials 203

15 Establishing A Reliability Lab 207

15.1 Staffing For Reliability 207

15.2 The Reliability Lab 208

15.3 Facility Requirements 209

15.4 Liquid Nitrogen Requirements 209

15.5 Air Compressor Requirements 211

15.6 Selecting A Reliability Lab Location 211

15.7 Selecting A HALT Test Chamber 212

15.7.1 Chamber Size 213

15.7.2 Machine Overall Height 214

15.7.3 Power Required And Consumption 216

15.7.4 Acceptable Operational Noise Levels 216

15.7.5 Door Swing 216

15.7.6 Ease Of Operation 216

15.7.7 Profile Creation, Editing, And Storage 216

15.7.8 Temperature Rates Of Change 217

15.7.9 Built-In Test Instrumentation 217

15.7.10 Safety 217

15.7.11 Time From Order To Delivery 217

15.7.12 Warranty 217

15.7.13 Technical/Service Support 217

15.7.14 Compressed Air Requirements 222

15.7.15 Lighting 222

15.7.16 Customization 222

16 Hiring And Staffing The Right People  223

16.1 Staffing For Reliability 223

16.2 Staffing For Software Engineers 227

16.3 Choosing The Wrong People For The Job 228

17 Implementing The Reliability Process 230

17.1 Reliability is Everyone’s Job 230

17.2 Formalizing The Reliability Process 231

17.3 Implementing The Reliability Process 232

17.4 Rolling Out The Reliability Process 232

17.5 Developing A Reliability Culture 236

17.6 Setting Reliability Goals 237

17.7 Training 238

17.8 Product Life Cycle Defined 239

17.9 Concept Phase 240

17.10 Design Phase 241

17.11 Production Phase 242

17.12 End-Of-Life And Obsolesence Phase 242

17.13 Proactive And Reactive Reliability Activities 242

18 Product Concept Phase 249

18.1 Establish The Reliability Organization 251

18.2 Define The Reliability Process 251

18.3 Define The System Reliability Requirement 251

18.4 Capture And Apply Lessons Learned 252

18.5 Risk Mitigation 254

18.5.1 Filling Out The Risk Mitigation Form 255

18.5.2 Risk Mitigation Meeting 257

19 Design Concept Phase 258

19.1 Setting Reliability Requirements And Budgets 259

19.2 Define Reliability Design Guidelines  263

19.3 Risk Mitigation In The Design Concept Phase 264

19.3.1 Identifying Risk Issues 264

19.3.2 Reflecting Back (Capturing Internal Lessons Learned) 265

19.3.3 Looking Forward (Capturing New Risk Issues) 266

19.4 Reliability Capital Budget And Activity Scheduling 268

19.5 Risk Mitigation Meeting 270

19.6 Reflection 270

20 Product Design Phase 271

20.1 Product Design Phase 271

20.2 Reliability Estimates 273

20.3 Implementing Risk Mitigation Plans  273

20.3.1 Mitigating Risk Issues Captured Reflecting Back 274

20.3.2 Mitigating Risk Issues Captured Looking Forward 276

20.4 Design For Reliability Guidelines (DFR) 281

20.4.1 Derating Guidelines 283

20.5 Design FMEA 284

20.6 Installing A Failure Reporting Analysis And Corrective Action System 286

20.7 HALT Planning 287

20.8 HALT Test Development 288

20.9 Risk Mitigation Meeting 290

21 Design Validation Phase 293

21.1 Design Validation 294

21.2 Using HALT To Precipitate Failures 295

21.2.1 Starting The HALT Test 297

21.2.2 Room Ambient Test 299

21.2.3 Tickle Vibration Test 299

21.2.4 Temperature Step Stress Test And Power Cycling 300

21.2.5 Vibration Step Stress Test 301

21.2.6 Combinational Temperature And Vibration Test 301

21.2.7 Rapid Thermal Cycling Stress Test  302

21.2.8 Slow Temperature Ramp 303

21.2.9 Combinational Search Pattern Test 303

21.2.10 Additional Non-Environmental Stress Tests 304

21.2.11 HALT Validation Test 304

21.3 Proof Of Screen (POS) 306

21.4 Highly Accelerated Stress Screen (HASS) 308

21.5 Operate Fracas 309

21.5 Design FMEA 310

21.6 Closure Of Risk Issues 310

22 Software Testing And Debugging 313

22.1 Unit Tests 313

22.2 Integration Tests 314

22.3 System Tests 315

22.4 Regression Tests 318

22.5 Security Tests 319

22.6 Guidelines For Creating Test Cases 320

22.7 Test Plans 321

22.8 Defect Isolation Techniques 322

22.8.1 Simulation 322

22.9 Instrumentation And Logging 325

23 Applyng Software Quality Procedures 329

23.1 Using Defect Model To Create Defect Run Chart 330

23.2 Using Defect Run Chart To Know When You Have Achieved The Quality Target 330

23.3 Using Root Cause Analysis On Defects To Improve Organizational Quality Delivery 332

23.4 Continuous Integration And Test 333

24 Production Phase 334

24.1 Accelerating Design Maturity 334

24.1.1 Product Improvement Tools 337

24.2 Reliability Growth 340

24.2.1 Accelerated Reliability Growth (ARG) 342

24.2.2 Accelerated Early Life Testing (ELT) 344

24.3 Design And Process FMEA 344

24.3.1 Quality Control Tools 345

25 End Of Life Phase 351

25.1 Product Termination 352

25.2 Project Assessment 352

26 Field Service 353

26.1 Design For Ease Of Access 353

26.2 Identify High Replacement Assemblies (FRU’s) 353

26.3 Wear Out Replacement 354

26.4 Preemptive Servicing 355

27.5 Servicing Tools 355

26.6 Service Loops 356

26.7 Availability Or Repair Time Turnaround 357

26.8 Avoid System Failure Through Redundancy 357

26.9 Random Versus Wear Out Failures 357

Appendix A 359

Reliability Consultants 359

Graduate Reliability Engineering Programs & Reliability Certication Programs 362

Reliability Professional Organizations & Societies  366

Reliability Training Classes 367

Environmental Testing Services 369

HALT Test Chambers 371

Reliability Web Sites 371

Reliability Software 372

Reliability Seminars & Conferences 374

Reliability Journals 375

Appendix B 376

MTBF, Fit, and PPM Conversions 376

Mean Time Between Failure (MTBF) 376

Estimating Field Failures 383

Table of Figures

Figure 1‑1 Product Cost is Determined Early In Development 6

Figure 1‑2 Cost To Fix A Design Increases An Order Of Magnitude With Each Subsequent Phase. Courtesy Of Teradyne, Inc. 7

Figure 1‑3 The Reliability Process Reduces The Number Of ECOS Required After Product Release 8

Figure 1‑4 Including Reliability In Concurrent Engineering Reduces Time To Market 8

Figure 1‑5 Product Introduction Relative To Competitors. 9

Figure 1‑6 The ICM Process 10

Figure 2‑1 Overcoming Reliability Hurdles Bring Significant Rewards. Courtesy Of Teradyne, Inc. 15

Figure 5‑1 The Six Phases Of The Product Life Cycle 35

Figure 5‑2 The ICM Process 37

Figure 5‑3 A Risk Mitigation Program (ICM) Needs To Address Risk Issues In All Aspects Of The Development Program. Courtesy Of Teradyne, Inc. 41

Figure 6‑1 The Bathtub Curve (Timescale Is Logarithmic) 47

Figure 6‑2 Cumulative Failure Curve 47

Figure 6‑3 Light Bulb Theoretical Example 48

Figure 6‑4 Availability As A Function Of MTBF And MTTR 55

Figure 6‑5 Design Maturity Testing – Accept/Reject Criteria 58

Figure 6‑6 Number Of Fan Failures Vs. Run Time 61

Figure 6‑7 Mechanism That Can Cause Degradation And Failure 63

Figure 6‑8 PHM Data Collection And Processing To Detect Degradation (Courtesy Anto Peter) 66

Figure 7‑1 Functional Block Diagram 72

Figure 7‑2 Filled Out Functional Block Diagram 72

Figure 7‑3 Schematic Diagram Of A Flashlight 74

Figure 7‑4 Functional Block Diagram Of A Flashlight 75

Figure 7‑5 Functional Block Diagram Of A Flashlight Using Post-Its^® 76

Figure 7‑6 Fault Tree Logic Symbols 77

Figure 7‑7 Fault Tree Diagram For Flashlight Using Post-Its^® 78

Figure 7‑8 Logic Flow Diagram 79

Figure 7‑9 Fault Tree Logic Diagram 79

Figure 7‑10 Flash Light Fault Tree Logic Diagram 85

Figure 7‑11 Functional Block Diagram For The Flashlight Process 86

Figure 7‑12 Example Of A SFTA For An Execution Flow Failure 96

Figure 8‑1 Pareto Of Failures 101

Figure 8‑2 HALT Failure Percentage By Stress Type 103

Figure 8‑3 Product Design Specification Limits 104

Figure 8‑4 Design Margin 104

Figure 8‑5 Some Products Fail Product Spec 104

Figure 8‑6 HALT Increases Design Margin 105

Figure 8‑7 Soft And Hard Failures 106

Figure 8‑8 Impact Of HALT On Design Margins 106

Figure 8‑9 Two Heat Exchangers Placed In Front Of Chamber Forced Air 107

Figure 8‑10 Test Setup Profile To Checkout Connections And Functionality 110

Figure 8‑11 Temperature Step Stress With Power Cycle And End Of Each Step 112

Figure 8‑12 Vibration Step Stress 113

Figure 8‑13 Temperature And Vibration Step Stress 115

Figure 8‑14 Rapid Thermal Cycling 116

Figure 8‑15 Slow Temperature Ramp 117

Figure 8‑16 Slow Temperature Ramp With Constantly Varying Vibration Level 118

Figure 8‑17 HASS Stress Levels 121

Figure 8‑18 The Bathtub Curve 122

Figure 8‑19 HASA Plan. Courtesy Of James Mclinn 124

Figure 8‑20 A HALT Chamber Has Six Simultaneous Degrees Of Freedom (Movement) 126

Figure 8‑21 ARG Process Flow 128

Figure 8‑22 Accelerated Reliability Growth 129

Figure 8‑23 ARG And ELT Acceleration Test Plans 129

Figure 8‑24 The Arrhenius Acceleration Model 130

Figure 8‑25 Selective Process Control. Courtesy Of James Mclinn 131

Figure 9‑1 Quality ROI Chart (Financial Impact Of Escapes Is Low) 138

Figure 9‑2 Quality ROI Chart (Financial Impact Of Escapes Is High) 138

Figure 9‑3 Defect Run Chart 1 143

Figure 10‑1 Generic Fishbone Diagram 149

Figure 10‑2 Sample Fishbone Diagram 150

Figure 10‑3 Sample Pareto Chart 151

Figure 10‑4 Code Review Root Cause Pareto 152

Figure 10‑5 Try-Catch Code Example 154

Figure 11‑1 Waterfall Lifecycle 158

Figure 11‑2 Quality Processes In A Waterfall Lifecycle 159

Figure 11‑3 Sprint Activities 160

Figure 11‑4 Sprint Activities In An Epic 161

Figure 12‑1 Sample Requirements 170

Figure 12‑2 Sample User Stories 171

Figure 12‑3 Code Comments Example 176

Figure 12‑4 Sample UART HAL Code 182

Figure 15‑1 ESPEC/Qualmark HALT Chamber 214

Figure 17‑1 The Six Phases Of The Product Life Cycle 239

Figure 17‑2 The Hardware Reliability Process 244

Figure 17‑3 Proactive Activities In The Product Life Cycle 245

Figure 18‑1 Risk Severity Scale 255

Figure 18‑2 ICM Sign-Off Required Before Proceeding To Design Concept 256

Figure 19‑1 Opportunity To Affect Product Cost 257

Figure 19‑2 The Bathtub Curve 259

Figure 19‑3 System MTBF Requirement 261

Figure 19‑4 Subsystem MTBF Requirement 261

Figure 19‑5 180^o Of Reliability Risk Mitigation 264

Figure 19‑6 Where To Look For New Reliability Risks 265

Figure 19‑7 The Reliability Risk Mitigation Process 267

Figure 19‑8 The ICM Is An Effective Gate To Determine If The Project Should Proceed 269

Figure 20‑1 The First Phase Of The Product Life Cycle 271

Figure 20‑2 Looking Forward To Identify Risk Issues 275

Figure 20‑3 Risk Mitigation Strategies For Reliability And Performance 275

Figure 20‑4 Risk Growth Curve Shows The Rate At Which Risk Issues Are Identified And Mitigated 280

Figure 20‑5 DFR Guideline For Electrolytic Capacitor Usage. Courtesy Of Teradyne, Inc. 283

Figure 20‑6 HALT Planning Flow 287

Figure 20‑7 HALT Planning Check List 287

Figure 20‑8 HALT Development Phase 288

Figure 21‑1 Reliability Activities In The Validation Phase 294

Figure 21‑2 HALT Process Flow 297

Figure 21‑3 HALT Test Setup Verification Test 299

Figure 21‑4 Temperature Step Stress 300

Figure 21‑5 Vibration Step Stress 300

Figure 21‑6 Temperature And Vibration Step Stress 301

Figure 21‑7 Rapid Thermal Cycling (60°C/Min) 301

Figure 21‑8 Slow Temperature Ramp 302

Figure 21‑9 Slow Temperature Ramp And Sinusoidal Amplitude Vibration 303

Figure 21‑10 HALT Form To Log Failures 304

Figure 21‑11 HALT Graph Paper For Documenting Test 305

Figure 21‑12 HASS Stress Levels 306

Figure 21‑13 HASS Profile 308

Figure 22‑1 Sample Test Plan 316

Figure 22‑2 Sample Log Code 326

Figure 22‑3 Example Log File Extract 327

Figure 24‑1 Achieving Quality In The Production Phase 336

Figure 24‑2 Design Issue Tracking Chart 338

Figure 24‑3 Reliability Growth Chart 339

Figure 24‑4 Reliability Growth Chart Versus Predicted 340

Figure 24‑5 Duane Curve 341

Figure 24‑6 Phase 5 ARG Process Flow 342

Figure 24‑7 Typical SPC Chart 345

List of Tables

Table 5.1 Functional Activities For Cross-Functional Integration Of Reliability 38

Table 6.1 Failures In The Warranty Period W/Different MTBFS 51

Table 6.2 Advantages Of Proactive Reliability Growth 55

Table 6.3 RDT Multiplier For Failure-Free Runtime 57

Table 6.4 Sensors To Monitor For Overstress In Wearout Degradation 64

Table 6.5 FMMEA For Fan Bearings (Detection Omitted) 65

Table 6.6 Sensors To Monitor Bearing Degradation 66

Table 6.7 Component Grade Temperature Classifications 69

Table 7.1 The FMEA Spreadsheet 80

Table 7.2 RPN Ranking Table 82

Table 7.3 FMEA Parking Lot For Important Issue That Are Not Part Of The FMEA 89

Table 7.4 Common Software Failure Modes 93

Table 7.5 Common Causes For Software Failure 93

Table 7.6 Failure Modes And Associated Possible Causes 94

Table 8.1  Agreed Upon HALT Limits 108

Table 8.2 HALT Profile For Test Setup Checkout 109

Table 8.3 Temperature Step Stress With Power Cycle And End Of Each Step 112

Table 8.4 Vibration Step Stress 113

Table 8.5 Temperature And Vibration Step Stress 114

Table 8.6 Rapid Thermal Cycling 116

Table 8.7  Slow Temperature Ramp 117

Table 8.8 Slow Temperature Ramp With Constantly Varying Vibration Level 118

Table 9.1 Sample Line Counts 140

Table 11.1 CMMI Process Areas 163

Table 11.2 CMMI Maturity Levels 164

Table 11.3 Lifecycle Comparison 165

Table 15.1 Annual Sales Dollars Relative To Typical Warranty Costs 212

Table 15.2 HALT Facility Decision Guide 218

Table 15.3 HALT Machine Decision Matrix 219

Table 16.1 Reliability Skill Set For Various Positions 223

Table 17.1 Reliability Activities For Each Phase Of The Product Life Cycle 232

Table 17.2 Reliability Activities – What’s Required, Recommended And Nice To Have 235

Table 18.1 Product Concept Phase Reliability Activities 249

Table 18.2 Product Concept Phase Risk Mitigation Form 254

Table 19.1 Design Concept Phase Reliability Activities 258

Table 20.1 Reliability Activities For The Product Design Phase 271

Table 20.2 Common Accelerated Life Test Stresses 277

Table 20.3 Environmental Stress Tests 278

Table 21.1 Reliability Activities In The Design Validation Phase 293

Table 24.1 Reliability Activities In The Production Ramp Phase 5 334

Table 24.2 Reliability Activities In The Production Release Phase 6 335