CISA Information Systems Implementation: Testing, Release, Changeover, Data Conversion and Post-implementation Review

Implementation testing runs in levels, each building on a stable result from the one before: unit testing, then integration testing, then system testing, then final acceptance testing. Each level answers a different question, is run by a different party and has its own entry and exit criteria. When a later level starts before the earlier one is stable, it spends its time finding faults the earlier level should have caught, those faults are harder to diagnose at that stage, the later level's results stop measuring what it was meant to measure, and every late fix forces the later tests to be repeated. The order matters because each level assumes the parts below it already work.

Two other terms belong here. Sociability testing confirms that the new system can coexist with the other systems already running in its target environment without harming them. Pilot testing runs the system for a limited group or location before full release. An IS auditor reviewing readiness checks that each level has a documented plan, test cases that trace back to requirements, recorded results and resolved or formally accepted defects before the next level begins.

Non-functional tests show whether the system will perform, survive and recover in real conditions, and regression testing shows that a change has not broken something that worked before. The performance tests are near-twins, and the difference is what is being pushed and how far:

The requirement names the test. A requirement written as a service level at a known level of demand calls for load testing; a question about headroom or failure behaviour calls for stress testing; a question about data growth calls for volume testing.

Regression testing reruns a broad set of tests that passed earlier, using the same test data where possible, after any fix or change. Because the inputs are identical, any difference in the results can be traced straight to the change. Its purpose is side effects: a change to one component can break the other functions that call it, so the regression set must reach beyond the component that changed. Checking only that the reported defect is gone proves the fix, not the absence of new faults.

When there is not time to rerun the full suite, impact analysis traces which programs, interfaces and reports depend on the changed component, and the regression tests for those are rerun first. This is risk-based selection: the scope shrinks to what the change can affect, but regression testing still happens before the change reaches production.

User acceptance testing (UAT) is the business confirming, independently of the builders, that the system meets its requirements, so it is designed from the business requirements and acceptance criteria, run or witnessed by business users and signed off by the business owner. Approval to move to production rests primarily on that sign-off. Assurance from other parties supports it but cannot stand in for it, because only the business can say that the system does what the business needs.

Independence is the point. When the party that built or configured the system also designs and runs the acceptance tests, it tends to test what it built rather than what the business asked for, so gaps in understanding pass unnoticed. The remedy is to put the business back in charge: users derive the scenarios from their own acceptance criteria and run or witness them, with the builder supporting rather than leading.

Test data must not create a new exposure. Test environments usually have wider access and weaker controls than production, so loading live personal or confidential records (salaries, national identifiers, medical details, payment data) into test is a significant weakness. The preventive control is to mask, substitute or synthesize sensitive fields before the data reaches the test environment; controls that only deter or detect misuse leave the exposure in place.

Substitute data must still serve the test objective. If the application validates a field with a format or checksum rule (an identifier with a check digit, a bank account format, a postcode pattern), the substitutes must be generated to satisfy that rule, alongside deliberately invalid values for the negative tests. Format-preserving masking or synthetic generation meets both the privacy requirement and the test requirement at once.

The test environment itself should mirror production (same software versions, configuration and component versions) so that a pass in test predicts behaviour in production, and it should be separate from development and production.

A system is ready for production when its business owner has accepted it against agreed go-live (go/no-go) criteria and every open defect has been either fixed or assessed and formally accepted by the accountable owner. That sign-off is the best single piece of evidence of readiness because it covers the system as a whole. Evidence that reports on the project's progress, or on one component or one readiness area, is useful but narrower.

Going live with known defects, even serious ones, can be legitimate when the owner understands the risk and accepts it with compensating controls, such as manual checks or extra review for a defined period. The IS auditor's role is to make sure that the decision is informed, taken by the accountable person and documented. The go/no-go decision and the choice of workaround remain management's.

Readiness typically covers completed and signed-off testing, converted and reconciled data, trained users, updated procedures and documentation, operations and support handover, security controls in place, a cutover plan with fallback, and a scheduled post-implementation review.

For systems that need formal security authorization, two terms are distinct:

CISA uses the terms certification and accreditation. Newer NIST guidance (SP 800-37 Revision 1 onwards) calls the same pair assessment and authorization, and the outcome an authorization to operate; the split between technical evaluation and management acceptance of risk is unchanged.

Attestation (an assertion or report that a statement is true) and validation (confirming that a system meets its intended use) are related assurance terms, but neither is the management authorization to operate.

Release management defines each release as a versioned package of specific, approved components, and configuration management records those components and settings in an approved baseline so that every later change can be requested, approved and verified against it. Before a new system's first production release, an approved configuration baseline of each component's version and settings is what most helps the organization control later change and detect unauthorized drift. Records that describe history or access do not define what the approved state is.

A release whose contents are not pinned to exact versions cannot be reproduced, cannot be tested meaningfully (the test may exercise something other than what is deployed) and cannot be verified after deployment. When an auditor finds several release weaknesses together, the useful question is which one the others depend on: comparisons between environments and checks after deployment only work when there is a defined, versioned release to compare against, so the missing definition is the root cause and the rest are symptoms.

A sound release package contains the exact versions of code, libraries, scripts and configuration items; the test evidence for that package; approval to release; deployment and backout instructions; and a record of where it was installed. Ongoing IT change, patch and emergency-change management after go-live are covered in Domain 4.

Code should reach production only through a party independent of development, from a controlled library, and the deployed build should be verified to be exactly the version that passed acceptance. The person who writes a change should not also be able to put it into production, because nothing then stops untested or unauthorized code from going live. The preventive control is that a separate group (operations or release management) migrates only approved releases, and that development staff have no write access to production. Detective controls find an unauthorized change after it is live; separation stops it.

Separation is only as good as the integrity of what is handed over. Between acceptance and deployment, the accepted version must not be able to change. The control has two parts:

• Lock the accepted build in a controlled library (or artifact repository) that development staff cannot change.
• Verify integrity at deployment, for example by comparing a cryptographic hash of the deployed build with the hash recorded for the accepted one.

Automated pipelines (CI/CD) change how segregation is achieved, not whether it is needed. When a pipeline deploys automatically, the merge into the release branch is effectively the promotion to production, so that is where the second person must sit: an independent reviewer approves each change, and the pipeline is configured to refuse to deploy anything that lacks the approval. Segregation in a pipeline means two people, and a technical rule that enforces it.

Changeover is the way users move from the old system to the new one, and the four approaches trade risk against cost and effort.

The right choice depends on the scenario's constraints. Parallel running normally gives the most assurance for a high-impact system, but it needs the people and capacity to process everything twice. When a constraint rules out the textbook choice, pick the method that best limits how many users an undetected error can reach before it is found and corrected, and that still leaves a way back.

A cutover plan must say when the organization will decide to proceed or stop, and how it will return to the old system if the new one fails. Its core elements are a timed sequence of steps (final data conversion, reconciliation, smoke tests, switch-over), go/no-go criteria at defined checkpoints, named decision-makers, and a backout procedure that has been tested.

The fallback matters most with a direct cutover, and it matters more the faster the route back disappears: when the old hardware is to be reused, the old licence ends or the old database stops receiving updates, a failure after the switch can leave the organization unable to operate. Missing go/no-go criteria and an untested way back are therefore among the most serious cutover gaps; most other gaps can be fixed after go-live, but a failed cutover with no route back cannot.

The best evidence that fallback would work is evidence of capability rather than intent: the team has actually carried out the whole reversal end to end in a test environment, and timed it against the window available. Plans, approvals and individual ingredients such as backups show that a fallback was considered, not that it works.

Data conversion controls prove that every record moved from the old system to the new one (completeness), arrived with the right values (accuracy) and is accepted by the people who own it. Conversion usually follows data cleansing in the source system, mapping of old fields to new ones and trial conversions; the controls below are then applied to the final run.

Record-level reconciliation is the most effort, so it is usually automated or applied to the critical fields.

Match the control to the error. For completeness of transfer, comparing record counts and control totals between source and target is the standard evidence, because it covers the whole population. When values may have been attached to the wrong records, summary figures cannot help, and only a record-level comparison of key fields detects it.

Final sign-off on converted data belongs to the business data owner, after reviewing the reconciliation results: the converted data becomes the business's records, and the owner is accountable for them. The party who ran the conversion is checking its own work, so its confirmation is supporting evidence only.

New infrastructure must be built to the organization's approved hardening (secure configuration) baseline and verified against it before it enters production. Vendor-default builds of operating systems, databases and network devices typically ship with unnecessary services, sample accounts and permissive settings. The fix is to apply the whole baseline, not to remediate individual findings one at a time, which leaves the rest of the configuration at its defaults. The best evidence that servers meet the baseline is an automated configuration-compliance scan comparing each deployed server's actual settings with the approved baseline; statements that describe intent or a sample are weaker than a test of every machine actually going live.

A post-implementation review (PIR) determines whether the system meets its requirements, whether the project's deliverables and controls were achieved, and whether the benefits set out in the business case are being delivered. Project measures such as cost and timeliness are inputs to it, not its primary objective.

• Timing: the review takes place once the system has run long enough for benefits and control operation to be measured, often months after go-live. Lessons learned can be captured at project closure, while the team is still together, without bringing the review itself forward.
• Independence: the PIR needs objective evidence from people who did not deliver the project. The IS auditor may perform or review the PIR, but does not approve the system or its results on management's behalf.
• Root cause before recommendation: when a PIR finds a shortfall, the auditor first establishes why it happened. For example, if a new expense system has not cut processing costs and staff are submitting claims by email instead, the workaround is itself a control weakness and may explain the missing benefit, so the auditor finds out why staff avoid the system before recommending any remedy.

The auditor reports findings and recommendations to management; correcting the system or changing the process is management's job.