CISA Audit Execution: Sampling, Evidence, Data Analytics, Reporting and Follow-up

An IS audit is run as a controlled project: an approved plan sets the scope, objectives, resources and timeline, and any change to them is assessed and approved before the work changes. The audit lead tracks progress against the plan, keeps audit management and the engagement sponsor informed, and treats scope, schedule and resources as things that are re-planned openly rather than adjusted quietly.

Scope changes during fieldwork

Fieldwork often turns up something the plan did not anticipate. A data-centre review might reveal that a facilities contractor has standing remote access to building-management systems, or a stakeholder might ask for coverage of a process nobody budgeted for. The controlled path is the same in every case:

1. Raise the matter with audit management promptly.
2. Assess its effect on resources, timing and the existing objectives.
3. Agree a decision: revise the scope and plan (with approval), schedule separate work, or record it for future audit planning.

The principle is that scope decisions belong to audit management, and a significant exposure is communicated even when it falls outside the current scope. Whatever is decided is documented, so the report can state accurately what was covered.

Communicating progress and collecting feedback

Delays, resource shortfalls and late evidence are reported to audit management and the sponsor with their effect on the timeline and the options available, and the response is agreed openly: extend the timeline, re-prioritise, or report a limitation. The level of assurance the plan promised is never lowered silently; if the evidence standard or coverage has to change, audit management agrees it and the report says so.

Regular status meetings with auditee management exist so that emerging issues are raised early and facts are confirmed before reporting, leaving no surprises at the closing meeting. Management's input corrects facts and shapes action plans; the conclusions remain the auditor's.

Communication runs both ways. The auditor collects stakeholder feedback on how the engagement was run, through discussion at the closing meeting and post-engagement surveys of auditee management, and the audit function uses it in its quality program and in planning later audits.

A compliance test (test of controls) gathers evidence that a control operated as designed; a substantive test examines the transactions, balances or data themselves to find errors. Checking that sampled changes to the vendor master file were independently reviewed is a compliance test of that review control. Recalculating depreciation for sampled fixed assets is a substantive test of the asset values.

The two are linked through reliance. When compliance testing shows a key control works throughout the period, the auditor may rely on it and reduce the extent of substantive testing in the area it protects. When compliance testing shows a key control failed, that reliance is gone. If the objective is to conclude on the data the control protects, the evidence must now come from the data itself, so the auditor expands substantive testing. For example, if an inventory system's requirement for a second count on large stock adjustments was skipped, the auditor tests more adjustments directly to conclude on inventory values. The control failure itself is also reported as a finding.

Reliance has limits: a control that tested well this year does not carry its assurance into next year untested, and good results for one control say nothing about the other controls in the application.

Statistical sampling selects items randomly and uses probability to measure the risk that the sample's conclusion is wrong, so its result can be projected to the population at a stated confidence level; judgmental (non-statistical) sampling selects items by the auditor's judgment, often targeting risk, and its result cannot be projected statistically. The defining advantage of statistical sampling is that sampling risk is quantified objectively. Professional judgment is still needed in both approaches, to set the parameters and to evaluate the deviations found.

Judgmental selection is useful for aiming at the riskiest items, such as all access granted by a particular administrator account. The limit is in what the result supports: a conclusion drawn from items chosen because they were risky applies to those items. Saying anything about the whole population needs a representative (statistical) sample or a full-population test.

Choosing a method

Stop-or-go and discovery are both attribute methods for low expected rates, but their goals differ: stop-or-go minimises the sample when errors are rare, while discovery is designed to find a rare, significant event.

Stratification

Stratification divides a population into groups of similar items so each group can be sampled efficiently. When a small share of the items carries a large share of the value, as when a handful of high-value stock lines dominate an inventory, the auditor can examine every item in the high-value stratum and sample the rest. Because the large items no longer inflate the variability of the sample, this gives a more precise estimate for the same effort than an unstratified mean-per-unit sample.

For an attribute sample, the required size rises with the confidence level and with the expected deviation rate, and falls as the tolerable deviation rate rises. Tolerable rate is the maximum deviation rate the auditor can accept and still rely on the control; the closer the expected rate is to it, the more items are needed to tell them apart.

Population size matters far less than people expect. Once a population is large, the sample is driven by the confidence level and the expected and tolerable rates; growing the population, even many times over, raises the sample only a little. Sample size does not scale in proportion to the population.

Sampling risk

Sampling risk is the risk that the conclusion from a sample differs from the conclusion the auditor would reach by testing the whole population. It exists because only part of the population was examined, so even a properly drawn random sample can, by chance, be unrepresentative. It has two directions:

• Risk of overreliance: concluding a control is effective when it is not. This threatens audit effectiveness, because a wrong opinion is issued, and is the greater concern. In substantive testing the equivalent is the risk of incorrect acceptance (beta risk), and CISA material often uses the terms interchangeably.
• Risk of underreliance: concluding a control is ineffective when it works. This affects efficiency, because extra testing follows, but the opinion is not wrongly favourable. The substantive-testing equivalent is incorrect rejection (alpha risk).

Non-sampling risk

Non-sampling risk is the risk of a wrong conclusion for any reason unrelated to sampling: the auditor overlooks or misjudges a deviation in an item that was examined, misunderstands what counts as a deviation, or applies a procedure unsuited to the objective. Because it does not come from which items were selected, it is not affected by sample design. It is reduced by planning, clear guidance on what counts as an exception, training, and supervision and review of the work.

Audit evidence must be sufficient (enough of it) and appropriate (relevant and reliable) to support each conclusion, and its reliability depends mainly on its source and how it was obtained. ISACA's ITAF evidence standard requires the auditor to obtain this evidence before concluding.

What makes evidence more reliable

• Independent source: a balance confirmed by the bank directly to the auditor is more reliable than a copy of the bank statement provided by the client.
• Obtained directly by the auditor: evidence the auditor extracts from the system itself, such as a configuration exported from a firewall during the audit, is more reliable than a printout the auditee prepares, which could be altered or incomplete.
• Documentary or system-generated rather than oral: what someone says or signs about a control is a representation; it does not show the control operates.
• Covers the period: evidence spanning the whole period under review is more reliable than evidence from a single date.

Collection techniques compared

Documents and settings show what a control is meant to do; testing its behaviour shows what it does. To confirm that an application locks an account after repeated failed log-ons, for instance, the auditor attempts the failed log-ons on a test account. Independently performing a three-way match of purchase order, goods receipt and invoice for sampled payments, and comparing the outcome with the system's, is also reperformance.

Observation's limit matters when the conclusion covers a period. Watching a guard check badges at a data-centre entrance one morning shows the control worked that morning. To conclude on the year, the auditor inspects evidence that spans it, such as access logs for a sample of dates across the period.

Audit data analytics uses computer-assisted audit techniques (CAATs), such as generalized audit software, scripts and analytics platforms, to test large volumes of data directly; the results are only as good as the data, so the auditor first confirms the data is complete and accurate.

Before analysing an extract

• Verify completeness and accuracy first: record counts and control totals (for example, total transaction value) are reconciled to the source system before any analysis begins. Someone's assurance that the extract is complete is a representation; reconciliation is verification.
• Protect production and independence: the auditor keeps control of the queries and their logic, uses read-only access, and ideally works on a verified copy of production data. That keeps production data safe from accidental change and keeps the evidence in the auditor's hands rather than the auditee's.
• Document the data source, extraction method, scripts and parameters in the working papers so the work can be reviewed and repeated.

What full-population testing does and does not prove

Testing every item removes sampling risk for the attribute tested, because nothing is left unexamined. It does not remove the need to confirm data completeness, to evaluate the controls over the system, or to follow up exceptions with management. Nor does it prove more than the test logic checks: an exact-match test finds only exact matches, so records that differ slightly (a reformatted reference, a transposed digit) pass unnoticed. Before concluding that a condition is absent, the auditor considers fuzzy or near-match tests and other ways the condition could be hidden.

Common analytics tests include duplicate and gap detection, matching between systems (such as HR leavers against active accounts), recalculation, stratification and ageing, outlier detection, and Benford's-law digit analysis.

Continuous (online) audit techniques collect or test evidence while a system processes live transactions, and each type suits a different objective.

The near-twins are snapshot (follows a transaction through processing steps) and SCARF (selects and logs qualifying transactions for later review), and ITF (the auditor's own test data in the live system) and the others, which work on real transactions. Running inside production is what lets an ITF test live processing logic, and it is also why its specific risk is contaminating production results.

When evidence essential to an audit objective is withheld, the auditor first tries alternative procedures; if none can meet the objective, the scope limitation is reported to the audit committee and disclosed in the audit report.

Alternative procedures first

A refusal of one route to evidence is not automatically a scope limitation. If the same evidence can be obtained another way, the auditor uses it, provided the auditor either controls how it is produced (for example, watching a report being generated) or verifies it independently (for example, against record counts, control totals or sequence checks). Evidence the auditor has neither controlled nor verified carries little weight. A service organization's SOC report is useful in its own right, but it gives assurance about the provider's controls, not about the client's own transactions or configurations.

When no alternative exists

If access is refused, escalation does not resolve it, and no other source provides equivalent evidence, the objective cannot be met. The auditor reports the limitation to the audit committee and discloses it in the report, so that readers know exactly what the audit could and could not conclude on. Evidence produced by the auditee about its own activity is not a substitute for independent evidence.

Indicators of fraud or irregularities

An IS auditor who finds indicators of possible fraud communicates them promptly to an appropriate level of management above those involved and, where needed, to those charged with governance. Indicators include unexplained overrides, missing documents, and undisclosed conflicts of interest, such as an approver with a personal interest in the transaction approved.

The auditor's role is to be alert to indicators, document them, consider their effect on the audit and report them. Investigation, any action against individuals or their access, and any referral to authorities are decisions for management and legal counsel, unless law places a reporting duty on the auditor. Timeliness matters more than a complete picture, and discretion matters too: alerting the people involved can compromise evidence. The auditor may consult legal counsel through audit management. Detecting all fraud is not the auditor's responsibility, but being alert to its indicators is.

A well-formed audit finding states four attributes: the condition (what was found), the criteria (what should be), the cause (why the gap exists) and the effect (the risk or impact). The recommendation addresses the cause.

Recommendations that address root cause

A recommendation aimed at the cause is the one most likely to stop the condition recurring. Correcting the specific instances fixes today's condition, and a periodic detective check catches recurrence after the fact, but only removing the cause prevents it. Where the cause is that a duty has no owner, as when servers are patched late because no team is responsible for the patch schedule, the root-cause recommendation is to assign accountability with a defined timeframe.

From draft to final report

• Discuss draft findings with auditee management (typically at an exit or closing meeting) before the report is final, to confirm facts and obtain management's responses and action plans.
• The auditor owns the content. Management comments on findings and responds to them; the wording, ratings and conclusions are the auditor's.
• Supported findings are reported even when management disagrees. The report records management's view, or its refusal to respond. Independence means the evidence, not agreement, decides what is reported.
• Issues corrected during the audit are still reported, noting the corrective action taken. They existed during the period, and governance should know the control failed. Results are communicated in a written report so there is a permanent record.
• The report goes to the parties set by the audit charter and engagement plan, and states scope, objectives, period, any scope limitation, findings, conclusions and recommendations.

Post-audit follow-up verifies with evidence that management carried out the actions it agreed and that the risk is sufficiently addressed; a quality assurance and improvement program checks that the audit function's own work meets standards.

Follow-up

• Timing is driven by the finding's risk and the agreed implementation date: higher-risk items are followed up sooner, scheduled when the action should be in place.
• Verify, do not accept assertions. Management's statement that an action is complete, however formal, is a representation. Follow-up asks whether the new or changed control is operating, so the auditor tests evidence of it, for example for a sample of the periods since it began.
• Risk acceptance must be real. Management may decide not to implement an agreed action. That is acceptable only if the residual risk was formally accepted by someone with the authority to accept that exposure, so the auditor's first step is to confirm that. If the accepted risk appears to exceed the organization's appetite, the auditor informs senior management or the audit committee. The decision itself stays with management.

Supervisory review of working papers

The primary purpose of a supervisor's review of working papers before the report is issued is to ensure that the evidence documented supports the findings and conclusions. Budget, schedule and retention rules are secondary. A conclusion that rests on inquiry alone needs corroborating evidence before it is reported, whatever the deadline, because inquiry remains inquiry however carefully it is recorded. Review, together with clear guidance on what counts as an exception and training, is also how an audit manager reduces non-sampling risk across the team.

Quality assurance and improvement program (QAIP)

An external assessment is credible only if the assessor is both qualified in audit practice and independent of the organization, which is why it must come from outside it. It is also a review of the audit function's conformance with standards, a different engagement from an audit of the organization's own controls. Results feed improvement actions and are reported to the audit committee.