CISA IT Management: Resources, Vendors, KPIs and Quality

An IS auditor evaluates IT resource management by checking that money, people and technology are committed to the work that best serves the organization's strategy, and that the organization later confirms it received the value it paid for. The official task is to evaluate IT resource and project management for alignment with the organization's strategies and objectives. Alignment is not shown by a strategy document; it is shown by how funding decisions are actually made and reviewed.

Alignment is decided when money is committed

The strongest evidence of alignment is an IT portfolio management process: every proposal, from every business unit, is submitted with a business case and ranked on the same criteria, such as expected benefit, contribution to strategic objectives, risk and cost. Because the ranking is enterprise-wide, scarce resources go to the investments that contribute most, not to the department that asks loudest. Each governance artefact answers a different question:

Projects in flight

Alignment also covers projects already running: portfolio and programme oversight re-checks active projects against strategy, stops or re-scopes those whose business case no longer holds, and reallocates staff and budget. Project governance itself (steering committees, stage gates, project management methods) is taught in the CISA Systems Acquisition and Development lesson in Domain 3.

Benefits realization closes the loop

A business case promises benefits; approval is granted on that promise. Benefits realization is the practice of measuring, after a system is in use, whether the promised benefits were achieved, and reporting the result to the body that approved the investment. Without it, governance never learns whether its decisions created value, and inflated business cases go unchallenged. Keep three post-delivery checks apart:

Good estimating before approval and value tracking after delivery work together: one improves the decision, the other tests it. How business cases and feasibility studies are built is covered with system acquisition in Domain 3; here the concern is whether the organization's IT resources as a whole are directed and reviewed against strategy.

IT personnel controls reduce the risk that the people who run information systems misuse their access, leave without warning, or take knowledge with them that the organization cannot replace. Human resources are an IT resource, and the IS auditor looks at controls across the employment life cycle.

Mandatory leave detects fraud

Many frauds require the person to be present every day to suppress exceptions, rework reconciliations or answer queries. When the employee is away for a continuous block and a colleague actually takes over the work, those schemes tend to surface. That is why mandatory vacation and job rotation are classed as detective controls, while controls applied at hiring set obligations and keep unsuitable people out but cannot reveal what happens later.

Timely termination is the main access risk

If IT learns of leavers only from a periodic HR report, every departed employee can keep working credentials until the next report arrives. That is the greatest concern because it is direct unauthorized-access exposure, most serious for involuntary terminations, where access should be removed at the moment the person is told. The control that closes the exposure is an HR-to-IT notification triggered by the leaving event itself (ideally automated through identity management), so that access removal does not depend on a reporting cycle. Transfer reviews and asset recovery complete the leaver process.

Key-person dependency

When one administrator has performed every critical procedure on a core system for years, the organization cannot operate or recover that system if the person is unavailable. Continuity depends on at least one other person being able to perform the work, which means skills that have been practised, not just written down. Documented procedures and succession plans support that; controls aimed at misuse, such as monitoring privileged activity, address a different risk.

An organization can outsource the performance of an IT service, but it cannot outsource accountability for the data, risk and regulatory compliance attached to it; that stays with the organization's board and senior management. This holds for every sourcing model: insourcing, outsourcing to a managed-service provider, offshore delivery, or cloud services (IaaS, PaaS, SaaS).

The provider is responsible for performing what the contract specifies. The organization remains accountable to regulators, customers and its board for the outcome. Two consequences follow:

• A contractual duty is a control only if the organization can verify it. A clause requiring the provider to follow the organization's retention schedule or security standards transfers the task; without a right to audit, independent assurance reports or regular reporting, nobody can tell whether the provider complies, and a breach is discovered by the regulator rather than by management.
• "The provider breached the contract" is not a defence. The organization may recover damages from the provider, but the regulatory finding is against the organization, and the IS auditor reports both that accountability remains in-house and that the means of monitoring were missing.

Other parties have defined roles but not accountability: a service auditor gives an opinion on controls, and the internal IS audit function provides independent assurance, which it could not do if it owned the controls. A sourcing decision should therefore be preceded by a risk assessment that asks what the organization must still be able to see and control once the service is moved, and the contract and monitoring plan should be designed around that answer.

A sound vendor selection process defines requirements and evaluation criteria before proposals are opened, scores every bidder against them, carries out due diligence on the shortlisted provider, and does not allow a contract to be signed until security and legal review are complete. The official task is to evaluate whether vendor selection and contract management meet business, legal and regulatory requirements.

The steps the auditor expects to see

1. Requirements and criteria first. Business, technical, security and regulatory requirements are documented; evaluation criteria and their weightings are agreed before the request for proposal (RFP) is issued or at the latest before responses are opened.
2. Consistent evaluation. Each proposal is scored against the same criteria, ideally by a panel, with the scoring retained as evidence.
3. Due diligence on the preferred provider, matched to the risk being assessed (see the table below).
4. Security and legal gate. The contract is signed only after the security assessment and legal review are finished and their findings resolved or formally accepted.
5. Approval at the level the procurement policy requires.

Agreeing criteria after proposals are opened is a serious finding: the evaluation can be shaped around a preferred bidder, which undermines the integrity of the decision and may breach procurement rules. The process is judged on whether criteria were fixed in advance and applied consistently, so that the decision can be explained and defended afterwards.

Match the due diligence to the risk

Fix the process, not just the instance

When a contract is signed before review and gaps surface later, the individual case can be remediated, but the auditor also asks why the process allowed it. A recommendation that changes the procurement process prevents the next occurrence; one that fixes a single provider does not. Decisions about the service itself, such as whether to continue it, belong to management.

The contract is the organization's main lever over a provider, so it must give the organization the right to verify controls, measurable service commitments, and a way to leave with its data. The IS auditor reviews whether these terms exist and whether they fit the type of service bought.

Escrow versus data return

Source code escrow deposits the application's code with a third party and releases it to the customer if the vendor goes out of business or stops supporting the product. It suits software the customer licenses and runs itself. For a standard multi-tenant software-as-a-service application, escrowed code is of little use: the customer would also need the provider's infrastructure, configuration and operations to run it. What lets the business continue is its data, so exit terms covering data ownership, the format and timing of its return, and transition assistance carry the continuity protection for SaaS. Remedies that depend on an operating provider, such as credits, lose their value if the provider stops trading.

Third-party assurance reports let an organization rely on a service provider's controls without auditing the provider itself, but only if it chooses the right report, checks the period and scope it covers, and performs the controls the report expects of the client. These reports are issued under the AICPA's System and Organization Controls (SOC) framework; the international equivalent for financial-reporting controls is ISAE 3402.

The Type I/Type II distinction applies to SOC 1 as well. To know whether controls operated effectively throughout a period, the auditor needs a Type II report covering that period. An ISO/IEC 27001 certificate attests that an information security management system meets the standard; it does not report tests of specific controls.

Bridge letters cover a time gap

When a report's period ends before the client's own period end, the first step is to obtain a bridge letter (also called a gap letter): a statement by the provider's management that there have been no significant changes to the controls since the report period ended, or describing any that occurred. Because it is management's assertion rather than tested evidence, the auditor may add procedures depending on risk. A bridge letter addresses a gap in time only.

Complementary user entity controls

Reports usually list complementary user entity controls (CUECs): controls the client must operate for the provider's control objectives to be achieved, for example reviewing its own users' access to the provider's portal. An unmodified opinion covers the provider's controls; it cannot cover a control the client was supposed to perform. Reliance on the report therefore depends on the client's own CUECs operating, and testing them is part of the client's audit.

Monitoring a provider's performance

Vendor management continues after signature. Two principles are tested:

• Corroborate provider-produced reports. Reports produced by the party being measured are evidence of lower reliability. The organization should compare them with its own incident, monitoring and outage records and understand how the provider measures each target, because the measurement method decides what the reported figure means.
• Shortfalls must lead to corrective action. Service credits compensate but do not make a critical service reliable. Vendor management uses an escalation path: root-cause analysis, a corrective action plan agreed with the provider and tracked through service review meetings, escalation when it slips, and ultimately exit or replacement.

Supply-chain risk is the exposure an organization inherits from the components, suppliers and subcontractors behind the products and services it buys, and the IS auditor evaluates whether the organization knows what is in its supply chain and has assurance over each link it depends on. The official task is to evaluate supply chains for IT risk factors and integrity issues.

Software components: the SBOM

A software bill of materials (SBOM) is a machine-readable inventory of the components, libraries and versions inside a piece of software (common formats are SPDX and CycloneDX). When a vulnerability is announced in a widely used open-source library, the organization's first question is where that component is used; an SBOM for each application answers it at once, so patching can be prioritized. Other controls work at a different level: scanning finds exposed systems, warranties allocate liability, and asset inventories record products rather than the components inside them.

Subservice organizations: inclusive versus carve-out

Providers often depend on their own suppliers, such as a data-centre operator. A SOC report presents such a subservice organization in one of two ways:

A carve-out does not mean the subcontractor's controls are weak; it means the report is silent on them, so the client has to know which of its relied-on controls sit with a subservice organization.

Hardware and firmware

Integrity concerns also apply to equipment: counterfeit or tampered components, modified firmware, and devices that pass through unvetted resellers. Controls include buying from authorized channels, verifying provenance and signed firmware, inspecting and testing equipment on receipt, and requiring suppliers to notify changes in their own supply chain.

A key performance indicator (KPI) measures how well IT is achieving its goals, usually after the fact; a key risk indicator (KRI) is a leading signal that exposure to a risk is rising, with a threshold that triggers escalation and action before a loss occurs. The official task is to evaluate the monitoring and reporting of IT KPIs and KRIs.

What makes a KRI effective

A KRI earns its place by being predictive: it moves while there is still time to respond. An indicator that changes only once the damage is done is a loss measure, however well it is documented. Good design also links each KRI to a risk in the register, calibrates thresholds, and gives each one an owner and an agreed response.

Detection without action is the common gap

Monitoring and response are separate parts of the control. An indicator can be measured and reported accurately and still achieve nothing if no one is obliged to respond when it breaches. The auditor therefore checks for defined ownership and escalation, and for evidence that breaches led to risk treatment. Setting risk appetite and maintaining the risk register belong to enterprise risk management, covered in the CISA Enterprise Risk Management, Privacy and Data Governance lesson; here the auditor evaluates whether IT's indicators are monitored, reported and acted on.

IT performance reporting is only useful if the measures are both reliable (correctly defined and calculated) and relevant (tied to the outcomes the business needs), and the IT balanced scorecard is a framework that makes reporting cover more than IT's internal operations.

Understand the definition before relying on a figure

A metric is only as reliable as its definition. A reported availability figure may exclude planned maintenance, partial degradation, or outages outside measured hours. Re-performing a calculation confirms the arithmetic but not whether the definition fits the decision being made. An auditor evaluating a metric therefore looks at its definition, scope and exclusions as well as the accuracy of the source data; comparisons and wider distribution add value only once the figure means what readers think it means.

Relevance: measure business outcomes

When dozens of indicators are green but business units say IT does not support them, IT is measuring its own activity rather than the outcomes the business relies on. Relevant measures start from what the business needs and work down to IT. COBIT 2019's goals cascade follows this logic: stakeholder drivers and needs lead to enterprise goals, enterprise goals to alignment goals, and alignment goals to governance and management objectives with their metrics. Accuracy and relevance are different properties: a precise measure of the wrong thing is still the wrong thing.

The four perspectives of the IT balanced scorecard

The IT balanced scorecard adapts Kaplan and Norton's business scorecard to IT (notably in work by Van Grembergen, which ISACA's guidance uses). It complements financial measures with four perspectives:

A measure belongs to a perspective by what it measures, not by whether it is a current result or a future target; future orientation is about building capability, such as skills and new technology readiness. A scorecard that reports only uptime, ticket times and delivery dates covers operational excellence alone, so management cannot judge IT's contribution to business goals, user satisfaction or future capability.

Quality assurance (QA) gives confidence that work is done according to defined processes and standards, while quality control (QC) examines particular outputs for defects; QA must be independent of the work it reviews, and maturity models give it a structured way to improve processes. The related official task is to provide guidance to the organization to improve the quality and control of information systems.

QA independence and enforcement

If QA reports to the head of the function whose projects it reviews, that manager can influence what QA reports, so findings may not be reported objectively. QA should report to a level above the projects it reviews, for example the CIO or a quality or steering committee. Independence must also extend to how findings are closed: if the project manager can sign a waiver for a skipped review and QA then closes the finding, the reviewed party decides whether its own deviation matters. Independent QA therefore needs authority over exceptions: deviations from the methodology go to someone outside the project for approval, and those still open are visible to management. QA reviews and reports; it does not take over the project's work.

Maturity models and gap assessment

Capability and maturity models rate how well a process is defined, managed and improved. CMMI (V2.0 and V3.0, now maintained by ISACA's CMMI Institute) rates organizational maturity from 0 Incomplete to 5; the commonly cited levels 1 to 5 are 1 Initial (ad hoc), 2 Managed (planned and tracked at project level), 3 Defined (standardized across the organization), 4 Quantitatively Managed (measured and controlled) and 5 Optimizing (continuous improvement). COBIT 2019 rates process capability on a CMMI-based scale from 0 to 5. Two rules guide their use:

• Baseline first. An improvement programme starts by assessing each process's current capability against its target level; the gaps found drive the plan and its investments in tools, training and measures.
• Targets follow business value and risk. Higher levels cost more to reach and sustain, so targets differ between processes: a process critical to the business or carrying high risk justifies a higher level than a routine one. Uniform top-level targets spend effort where it adds little, and other organizations' levels reflect their own priorities.

Quality assurance of the audit function itself is covered in Domain 1; operational service-level and capacity management processes are covered in Domain 4.