CISA IT Service Operations: Availability, Incident, Problem, Change, Patch, Log, Service Level and Database Management

Capacity management makes sure IT resources meet current and forecast business demand cost-effectively, and availability management makes sure services are usable when the business needs them. Capacity management monitors utilization and performance, forecasts demand and puts resources in place before they are needed, without paying for idle capacity. Availability management sets availability requirements, measures them and improves the factors that cause downtime. Restoring a failed service is incident management, and recording configuration items is configuration management; keep those neighbouring processes apart.

Peaks, not averages

An average hides the periods that matter. A payroll system can look lightly loaded across a month and still saturate on the two days each month when salaries are calculated, which is exactly when delays hurt. Capacity should be judged against peak periods and the business cycle. When reported utilization and user experience disagree, the auditor's first move is to gather evidence that explains the gap, such as performance data for the busiest periods, before forming a view on the remedy.

Forecasts need business input

Historical trends describe the past. A planned step change, such as an acquisition, entry into a new market or a regulatory reporting requirement, will not appear in any trend line, so capacity forecasts must take input from business plans and strategy. A forecast built on history alone is a weakness even when today's measurements look comfortable, because the risk lies in demand that has not arrived yet.

MTBF, MTTR and availability

Availability can be expressed as MTBF / (MTBF + MTTR), where MTBF is the mean time between failures (reliability) and MTTR is the mean time to repair (recoverability).

Redundancy is different again: it does not make a component fail less often, but it raises service availability by masking a component failure. The factor to improve is the one that is weak. Using the formula, a component that fails once a year and takes a week to repair loses far more availability to repair time than to failure frequency, so work on MTTR pays off most; a component that fails weekly but recovers in minutes needs better reliability. Redundancy and recovery design are covered in the Business resilience lesson.

Incident management restores normal service as quickly as possible; problem management finds and removes the underlying causes of incidents so they stop recurring. They work on the same failures with different goals and different time horizons, and CISA expects you to tell them apart and to recognise when one is doing the other's job badly.

Recurrence is the signal

Repeated incidents with the same symptoms and the same quick fix mean incident management is working and problem management is not. Picture a print service that hangs every Monday morning and is cleared each time by clearing the queue: every ticket is resolved on time, yet nobody asks why it happens. Problem management should open a problem record, analyse the root cause, document a known error and its workaround, and raise a permanent fix through change management.

Known errors and workarounds

A known error is a problem whose cause is understood and for which a workaround exists. The record only helps if the people diagnosing new incidents can find it, so it must sit where they search, such as a known-error database or knowledge base linked to incident diagnosis. A workaround that exists on paper but is not reachable by the service desk delivers none of its value. A problem should stay open as a known error until the permanent fix is implemented; treating the workaround as the end of the problem leaves the cause in place.

Measures and incentives

Measure each process by its own outcome. Problem management succeeds when repeat incidents fall, not when it produces more records or analyses. Good workarounds improve incident measures without removing any cause, so healthy incident statistics say little about problem management. Incentives shape behaviour too: a function rewarded for speed of restoration will tend to postpone slow root-cause work, so problem management works best with an owner whose objectives reward removing causes, while incident management stays free to restore service first.

Incidents should be classified and prioritized by business impact and urgency using defined criteria, escalated by defined rules when targets are at risk, and closed only when resolution is confirmed. Working tickets in arrival order, or giving every incident the same target, ignores how much each incident matters to the business.

• Impact is how widely or seriously the business is affected (number of users, criticality of the service).
• Urgency is how quickly the harm grows if nothing is done.
• Priority combines the two through a defined matrix, which sets target response and resolution times.

Functional and hierarchical escalation

Both should be written into the incident procedure with triggers tied to resolution targets, so that an incident at risk of breaching moves on in time. Escalation is about adding capability or authority while the incident is still open; reporting breaches afterwards is useful for management but does not help the incident in hand.

Closure and evidence of resolution

An incident is closed when the service is actually restored, best confirmed by the affected user. To test whether closure reflects real resolution, the strongest evidence is what the sampled tickets themselves show about confirmation from the user, because it tests the control operating. Speed statistics can look good while tickets are closed prematurely, procedures show only design, and staff explanations are testimonial. The general rule is to prefer evidence that is direct, independent of the people performing the process and drawn from operation rather than design.

The incidents in this topic are IT service incidents. Security incident response, forensics and evidence collection are covered in Domain 5.

Change management makes sure every change to production is assessed, tested, approved, implemented by the right people and reversible if it fails. A change advisory board (CAB) or change manager approves changes on evidence that bears on the risk to production: the business need, an impact assessment, test results and, especially for high-risk changes, a tested backout (rollback) plan that returns production to its previous state. Release management packages approved changes into scheduled releases and deploys them with that backout path, so operational releases stay inside change control.

Emergency changes

Emergency procedures let urgent fixes be made before formal approval so that service can be restored. A change made without prior approval is therefore not itself a finding; what makes it acceptable is that it is logged, reviewed and formally approved after the fact within a defined period, and normally regularised through the standard process. The auditor first confirms this compensating control exists and operates.

The emergency route must stay exceptional. When emergency changes cluster on one system or recur for the same cause, the route has become a way round normal change control, and the recurring cause belongs with problem management and a planned permanent fix. Watch the trend in emergency volume per system, not only whether each individual change was approved.

Segregating development from production

Programmers should not be able to move their own code into production. The preventive control is to give only an independent group (operations or release management) the right to update production libraries, moving approved, tested versions. Detective controls such as later review of what changed are valuable, but they act after the fact, and while developers keep write access to production every other control can be bypassed.

Emergency (fire-call) IDs

Where programmers need production access for urgent out-of-hours fixes, a temporary or emergency ID is an accepted approach because it breaks normal segregation only under control. The key compensating control addresses what is done with the access: every action under the ID is logged and reviewed after each use by someone independent of the person who used it, and tied to an emergency change record. Controls over how the ID is issued matter, but they restrict who gets access and when, not what happens once it is in use.

The direction of an auditor's sample depends on the assertion being tested: sample from change records to test that recorded changes followed the process, and start from production to test that no change bypassed it. Unauthorized changes by definition have no approved record, so a test that begins with the change tickets can never find them.

Following up exceptions

Reconciling production with change records can produce exceptions that have innocent explanations, such as a configuration file whose modification date changed when a vendor tool rewrote it without altering the settings. The auditor does not accept an explanation from the people involved without corroboration, and does not report a finding before verifying it either. The next step is to obtain evidence that settles the question, by comparing what is in production with the last authorized version. For programs, that comparison is made on source code: recompiled object code often differs from the original (embedded timestamps, compiler versions), so comparing object code is unreliable, and the source is what must match the last authorized version. Recommendations come after the facts are established.

Configuration management records each configuration item (CI), its approved baseline and its relationships to other items, so the organization knows what it runs and what a change or failure will affect. The configuration management database (CMDB) holds that record: which application runs on which server, which service depends on which database.

The relationships are the main benefit. Change management uses them to assess the impact of a proposed change on dependent services, and incident management uses them to trace the likely components behind an outage. The CMDB is not the asset register (financial data such as cost and depreciation belongs to IT asset management, covered in the sibling lesson on IT components and assets), nor the program library that holds approved code.

Accuracy and drift

The greatest risk of an inaccurate CMDB is operational: impact assessments built on wrong relationships let approved changes break services nobody knew were dependent. Financial or licensing inefficiency is a smaller concern. Periodic reconciliation, often by automated discovery, compares live configurations with recorded baselines.

Reading findings together

Interpret configuration findings by combining them. Evidence that approved changes are reliably recorded tells you the update process works for changes that go through change management. Live configurations that differ from their baselines with no related change record tell you something else: changes are being made outside the process, whether by staff fixing things directly or by unauthorized activity. The conclusion follows from the two facts together, and the remedy is to detect and investigate drift, not just to tidy the records.

Patch management identifies the systems that need a patch, prioritizes deployment by risk, tests patches and deploys them through change management with a way to back out. It starts from an accurate inventory, because a patch can only be applied to a system the organization knows it has.

Risk-based deadlines

Deployment deadlines should be tiered by vulnerability severity and the exposure of each asset, so that the riskiest combinations, such as a severe flaw on an internet-facing system, are fixed first. A single deadline for every patch, whether long or short, mismatches effort and risk: generous enough for the critical case is too slow, and tight enough for everything overloads testing and teams.

Test before production

Even urgent patches are tested in an environment that mirrors production, scheduled through change management and deployed with a confirmed rollback path. That balances closing the vulnerability quickly against the risk of breaking a core system; delay leaves the exposure open, while skipping testing trades a security risk for an availability one.

Relying on patch compliance reports

A compliance percentage describes only the systems the tool knows about. Unmanaged systems are often the least patched and never appear in the figure, so before relying on such a report the auditor tests the completeness of its population against an independent source of what actually exists, such as the asset inventory or an independent discovery scan.

Systems that cannot be patched

Some systems cannot be patched: an unsupported legacy operating system, or a certified medical or industrial device whose manufacturer forbids changes. If the fix is unavailable, the next best control limits the opportunity to exploit the weakness, without breaking the constraint that prevented patching in the first place:

• Compensating controls: isolate the system on a restricted network segment that permits only required traffic, and monitor it more closely.
• A documented exception to the patch policy, approved by the risk or business owner (accepting risk is not the technical team's decision), for a fixed term with an expiry date, after which the risk, the compensating controls and the remediation plan are reassessed.
• A retirement or replacement plan, so the exception has an end.

Risk acceptance or an exception on its own records the decision without reducing the risk, and exceptions that stay open for years with no expiry have become permanent and unowned.

Operational log management makes sure systems record enough of the right events, protect those records from tampering, keep them as long as obligations require and have them reviewed by someone independent. A log is only useful if it is complete, trustworthy, correlated with other logs and available when an investigation needs it.

Integrity: out of reach of the people being logged

Anyone who controls a system can usually alter its local logs. The preventive answer is to send log entries as they are written to a central repository that the people being logged cannot modify, so an independent record survives even if local entries are erased. Rules and deterrents help, but only separation of the log from the logged person's control actually prevents tampering.

Time synchronization

Correlating events across a firewall, an application server and a database requires every log to use the same time reference. Synchronizing all clocks with an authoritative time source (for example through the Network Time Protocol) prevents timestamps drifting apart. Without a common clock, no amount of formatting or detail lets investigators put events in the right order.

Content fit for purpose

Judge log content against what the logs must support. To investigate unauthorized changes to data, an investigator needs who made the change, what was changed, when, from where, and the before and after values. A log that records that a change happened but not what it changed cannot reconstruct the event.

Retention

Retention periods are set by legal, regulatory and investigative requirements, because investigations can start long after the events; storage is then sized to meet the requirement, not the other way round. Older logs can sit in cheaper archival storage provided they stay protected and retrievable.

Independent review of privileged activity

Privileged users, such as administrators and DBAs, can change data and controls directly, so their activity needs review by someone independent of that function. Review by someone who holds the same privileges is self-review in effect, and frequency or diligence cannot make up for the lack of independence. Security event monitoring tools such as SIEM are covered in Domain 5.

Service level management agrees service targets with the business, makes sure the provider can deliver them, and measures performance in a way that reflects what users experience. The service level agreement (SLA) is the agreement between the IT service provider and its customer.

An SLA commitment is only credible if every team and supplier it depends on has an agreed target that fits within it. If the service desk promises to restore email within a working day but the storage team and the hardware supplier it relies on have no agreed response times, the SLA is unsupported; the gap to report is the missing OLA or underpinning contract.

Metrics from the user's perspective

A metric can be met while the business suffers. A response-time figure measured inside the data centre, for instance, can look excellent while branch users wait on a slow network link. Availability and response metrics should be defined end to end, as users experience the service; a metric that measures the wrong thing stays wrong however its target or reporting is adjusted.

Validating provider reports

A provider has an incentive to report favourably, so the strongest validation comes from a source independent of it: the organization's own availability monitoring, service desk incident records and user complaints. Anything the provider itself produces or confirms, and anything that examines only how a report is prepared, is weaker evidence of whether the reported figures are true. Vendor selection and contract management are covered in the IT management lesson in Domain 2.

Database management controls keep stored data accurate and consistent, and keep the powerful database administrator (DBA) role accountable. An IS auditor evaluates both the integrity features the database enforces and the controls over the people who can bypass them.

Integrity controls

Match the control to the moment: locking and constraints prevent the error as it happens, rollback cleans up an interrupted transaction precisely, and reconciliations or restores from backup detect or recover after the event, usually with more disruption.

Normalization and denormalization

Normalization removes redundancy so each fact is stored once, at the cost of more joins. Denormalization deliberately reintroduces redundancy to cut joins and speed up reads, which means updates must keep several copies in step. The integrity risk of denormalized tables is therefore inconsistency between the copies after updates; the performance cost of joins belongs to normalization.

Controlling the DBA

• Accountability: each DBA uses a named privileged account with activity logged. A shared default administrator account means no action can be traced to a person; it should be locked away for break-glass use.
• Segregation: the DBA should not also enter business transactions or perform application duties. Where a small organization cannot separate them, the best compensating control is the one aimed at the specific conflict: independent review of the changes the DBA makes to that data. Controls aimed at other risks, or that cover only occasional periods, leave the conflict largely in place.
• Direct data corrections: changes made with SQL outside the application bypass its controls, so they need a preventive control: authorization by the data owner before the DBA acts. That authorization is also what any later review compares the change against.
• Independent review of DBA activity, by someone who does not hold DBA privileges.

Database backup and restoration are covered in the Business resilience lesson; data governance and classification belong to Domain 2.