A Simulationist's Framework
for Business Analysis

Part 07:

Testing and Acceptance

(Verification, Validation, and Accreditation)


R.P. Churchill

CBAP, IIBA-CBDA, PMP, CSPO, CSM
Lean Six Sigma Black Belt

www.rpchurchill.com/presentations/BAseries/07_Testing www.rpchurchill.com | Portfolio | Presentations
30 Years of Simulation

Continuous simulation of the heating of a square billet and Discrete-Event simulation of a multi-phase process.
30 Years of Simulation

Industries

  • Paper (Chemical/Process)
  • Nuclear (Power Generation)
  • Metals (Steel, Non-Ferrous)
  • HVAC (Building Control)
  • Insurance, Banking, Legal
  • Security, Inspections
  • Passenger Processing
  • Medical Facilities
  • Evacuations
  • Area Control
  • Threat Response
  • Logistics, Supply
  • Maintenance and Reliability
  • Staff Level Determination
  • Fleet Management

           

Applications

  • Design and Sizing
  • Operations Research
  • Real-Time Control
  • Operator Training
  • Risk Analysis
  • Economic Analysis
  • Impact Analysis
  • Process Improvement (BPR)

Architectural Considerations

  • Continuous vs. Discrete-Event
  • Interactive vs. Fire-and-Forget
  • Real-Time vs. Non-Real-Time
  • Single-Platform vs. Distributed
  • Deterministic vs. Stochastic
Context for Testing

Testing and Acceptance, aka Verification, Validation, and Acceptance, isn't just tacked on to the end of an engagement.

 

It actually takes place throughout the entire engagement...

...and should be planned for and incorporated from the beginning.

 

Most of this discussion will be software-centric, but the ideas may be applied more generally.

 

Link to detailed discussion.

 

The Framework:
  • Project Planning
  • Intended Use
  • Assumptions, Capabilities, Limitations, and Risks and Impacts
  • Conceptual Model (As-Is State)
  • Data Sources, Collection, and Conditioning
  • Requirements (To-Be State: Abstract)
    • Functional (What it Does)
    • Non-Functional (What it Is, plus Maintenance and Governance)
  • Design (To-Be State: Detailed)
  • Implementation
  • Test
    • Operation, Usability, and Outputs (Verification)
    • Outputs and Fitness for Purpose (Validation)
  • Acceptance (Accreditation)
  • Project Close
The Framework - Simplified
  •   Intended Use
  •   Conceptual Model (As-Is State)
  •   Data Sources, Collection, and Conditioning
  •   Requirements (To-Be State: Abstract)
    • Functional (What it Does)
    • Non-Functional (What it Is, plus Maintenance and Governance)
  •   Design (To-Be State: Detailed)
  •   Implementation
  •   Test
    • Operation, Usability, and Outputs (Verification)
    • Outputs and Fitness for Purpose (Validation)

 

 

IVV&A Inspiration for the Framework

Tell 'em what you're gonna do. Do it. Tell 'em what you did.

Accreditation Plan V&V Plan V&V Report Accreditation Report
Executive Summary Executive Summary Executive Summary Executive Summary
1. Problem Statement 1. Problem Statement 1. Problem Statement 1. Problem Statement
2. M&S Requirements and Acceptability Criteria 2. M&S Requirements and Acceptability Criteria 2. M&S Requirements and Acceptability Criteria 2. M&S Requirements and Acceptability Criteria
3. M&S Assumptions, Capabilities, Limitations, and Risks & Impacts 3. M&S Assumptions, Capabilities, Limitations, and Risks & Impacts 3. M&S Assumptions, Capabilities, Limitations, and Risks & Impacts 3. M&S Assumptions, Capabilities, Limitations, and Risks & Impacts
4. Accreditation Methodology 4. V&V Methodology 4. V&V Task Analysis 4. Accreditation Assessment
5. Accreditation Issues 5. V&V Issues 5. V&V Recommendations 5. Accreditation Recommendations
6. Key Participants 6. Key Participants 6. Key Participants 6. Key Participants
7. Planned Accreditation Resources 7. Planned V&V Resources 7. Actual V&V Resources Expended 7. Actual Accreditation Resources Expended
8. V&V Lessons Learned 8. Accreditation Lessons Learned
Suggested Appendices Suggested Appendices Suggested Appendices Suggested Appendices
A. M&S Description A. M&S Description A. M&S Description A. M&S Description
B. M&S Requirements Traceability Matrix B. M&S Requirements Traceability Matrix B. M&S Requirements Traceability Matrix B. M&S Requirements Traceability Matrix
C. Basis of Comparison C. Basis of Comparison C. Basis of Comparison C. Basis of Comparison
D. References D. References D. References D. References
E. Acronyms E. Acronyms E. Acronyms E. Acronyms
F. Glossary F. Glossary F. Glossary F. Glossary
G. Accreditation Programmatics G. V&V Programmatics G. V&V Programmatics G. Accreditation Programmatics
H. Distribution List H. Distribution List H. Distribution List H. Distribution List
I. Accreditation Plan I. V&V Plan I. Accreditation Plan
J. Test Information J. V&V Report
Lessons From Six Sigma

Six Sigma is technically about reducing variation, so fewer outcomes will be outside of the acceptable range.

However, it's more colloquially understood as eliminating causes of error, and therefore waste.

That is, the best way to fix errors is to prevent them from happening in the first place.

To that end, the entire process of iteratively working through the framework phases is about getting it right the first time.

Basic Engagement Structures

Link to detailed discussion.

Engagement Structure Variations

Link to detailed discussion.

Requirements Traceability Matrix

Keeping track of all relationships through the course of an engagement helps ensure completeness — and identify things that may have been missed.

Link to detailed discussion.

Tests Conducted In Each Phase
  •   Intended Use: Correctly identify the problem
  •   Conceptual Model: Understand what's going on
  •   Data: GIGO - Garbage In, Garbage Out
  •   Requirements: Understand what's needed
    • Functional: Make sure the solution does the right things
    • Non-Functional: Make sure it does it the right way
  •   Design: Develop effective solutions
  •   Implementation: Solid design, local test/debug, unit tests
  •   Test: After the solution is developed
    • Verification: Regression / Integration / UI / Output / Deployment testing
    • Validation: Comparison to expected/historical outputs, Expert judgment
    • Accreditation: Acceptance based on negotiated standards
Verification vs. Validation

Operation, Usability, and Outputs (Verification)

  • Tests to ensure the system operates as intended and produces outputs accurately from the inputs and calculations.
  • This process ensures that the system:
    • makes sense to the users
    • enables manipulation of all relevant elements
    • prevents and corrects errors
    • maintains users' situational awareness
    • includes documentation, training, and help
  • These types of tests are most able to be automated.

Outputs and Fitness for Purpose (Validation)

  • Tests to confirm that the results produced are targeted to the problem under investigation.
  • Ensures simulation behavior matches real-world system for known cases.
  • Validation of results may be:
    • Objective, e.g., measured comparisons to known values in a simulation or calculation
    • Subjective, e.g., judged as "correct" by SMEs for novel situations and realizations of business value
  • These types of tests are most likely to require expert judgment.

Link to detailed discussion.

Models of Testing

 

 

Models of Testing continued
Design Documentation
  • Implementation (coding) shouldn't start without design direction.
  • Design specification can happen at multiple levels, and implementation can proceed independently for each.
  • Examples of levels can be infrastructure, framework, and details.
  • Different teams can work on different levels.
  • They say slicing should be vertical and not horizontal, but this is a guideline, and is situationally dependent.
Types of Tests: Linting and Coding Style Standards
  • Linting and similar tools enforce standards for spacing, indentation, variable naming, and so on.
  • Naming conventions, both general (e.g., Hungarian or camelCase) and domain-specific (useful in simulation).
  • Some standards vary by language (C vs. Node vs. SQL vs. Python vs. Assembly vs. HTML/CSS), others are the source of almost religious arguments (2-spaces is the ONLY way to indent, dammit!).
  • Organizations may favor coding for clarity over coding for maximum operational efficiency in all cases.
  • Requirements for internal and external documentation may be specified.
  • Code can be structured to not require tests that are never needed — if the possible inputs are properly constrained.
  • All such choices should be made consciously and purposefully.
Types of Tests: Unit Tests
  • Automated tests that assess the existence and correctness of specific blocks of code.
  • Involve integrated libraries that analyze functions from the target code independently as a separate process.
  • Multiple tests may be run on each section of code, and a single project may involve thousands.
  • Used in regression testing.
  • Should be written by the same people who write the code.
  • How much coverage is needed?
    • 100% coverage is almost certainly pushing the point too far.
    • 0% coverage can work in smaller or single-practitioner projects.
    • This method may be best for large projects with many interactions and moving parts, where unexpected interactions and dependencies occur.
Types of Tests: Interface Tests continued
  • Many types of interfaces are possible.
    • Direct function calls in external libraries, units, or modules.
    • Message-based requiring packages of data to be sent and received (TCP/IP, XML, JSON).
    • Shared resource (memory, file, database)
    • Sensors measuring physical phenomena.
    • User interfaces (human-in-the-loop)
  • Some can be spoofed (mocked) within code, in interpreted languages.
  • Connections can be created to alternate hardware or software systems.
  • Switches can be set to execute alternate code.

Link to detailed discussion.

Types of Tests: Interface Tests continued
Types of Tests: Interface Tests continued
Types of Tests: Interface Tests continued
Types of Tests: Desktop/Local Debugging
  • This is the process of examining the effects of executing individual lines of code.
  • The original developers are best placed to do this during initial development.
  • Debugging tools can be software or hardware, and internal/integrated or external.
  • Can examine changes in memory, the content of messages, and detailed changes of state.
  • Debugging by third parties (maintainers, dedicated testers) is greatly aided by comments and documentation.
  • Debugging must sometimes be accomplished using write statements and other indirect methods (sound, lights, etc.)
  • Careful examination of standard output can illuminate details of internal operations.
Types of Tests: Standalone Test Bed, Spike/Demo
  • Experiments and tests can often best be conducted separate from the main system.
  • When a component works on its own, it may not be necessary to test it once integrated.
  • I developed entire test beds to test models that would be integrated into larger systems. Sometimes they were written in entirely different languages.
Types of Tests: User Interface Testing
  • UI/UX testing can be manual or automated.
  • Reviews can be conducted for technical operation, usability, understandability, theme/consistency/look-and-feel, and so on. (think functional vs. non-functional)
  • UI elements should exist to display or manipulate every relevant data item, and perform every desired action.
  • Automated testing tools exercise the UI using external scripts or recorded macros.
  • Examples include UFT/QTP, Selenium, CodeUI, UIA, TestComplete, and others.
  • Tests should be thorough to exercise every possible element on every screen.
  • May need language experts to review screens for modular internationalization.
Types of Tests: Technical, Performance, and Reliability Testing
  • Calculation Test: Determines whether predicted outcomes are correctly generated.
  • Smoke Test: Rough test to see whether anything obvious is broken.
  • Profiling: Measures percentage of time spent in each section of code.
  • Standalone Performance Test: Testing individual components or systems to see how long operations take or how many cycles can be achieved in a given time period.
  • Real-Time Test: Determines whether necessary activities can be completed within a specified time period.
  • Recovery-From-Error Test: Sees if system can recover gracefully from specifically induced errors.
  • Long-Term Robustness Test: Shows ability of system to run for long periods of time without maintenance, restarts, memory leaks, hardware failures, and so on.
Microservices CI/CD Pipeline
  • Individual services (components) are sent to testing by the developers.
  • Different screens and tests are run in each environment, and components are advanced to the next environment when they pass each environment's suite of tests.
  • Environments may test for Lint/Coverage/Documentation, Smoke, Interface, User Interface, Performance, and more before being deployed to production environment.
  • Test environments may run at smaller scale and may not include all services.
  • Specialized testers may run (and automate) each environment, while a release train manager oversees the entire process.
  • Components can be rolled back to previous versions if errors are found in production.
  • Most environments may form the initial release train, but there may be one or more dedicated environments for hotfixes.

Link to detailed discussion.

Other Test Contexts

Regression testing determines whether current changes broke any existing stuff.

Alpha and Beta (and similar) tests are preliminary releases of all or part of the final system to a limited set of users to assess robustness and fitness for final release. It is a form of crowdsourced testing.

Business Acceptance and Final Acceptance tests check against user and business requirements and intended uses established by the customer. These are linked through the Requirements Traceability Matrix (RTM).

Industry tests and Regulatory tests check against published standards established for various reasons.

The Ultimate Tests

The ultimate tests of whether your changes, systems, and business and organizational models work are

 

...the survival of your organization in a competitive landscape.

 

...economic profit, as measured by your accounting system.

 

...unit sales, customers served, volume, or market share.

 

...customer satisfaction, as measured by surveys and ongoing engagement.

This presentation and other information can be found at my website:

rpchurchill.com

E-mail: bob@rpchurchill.com

LinkedIn: linkedin.com/in/robertpchurchill