SC-200 Sentinel Data Ingestion: Syslog, CEF, Azure Activity, Threat Intelligence and Custom Logs

Use the Common Event Format (CEF) via AMA connector when a device sends CEF, and the Syslog via AMA connector when it sends plain syslog. CEF events land parsed in the CommonSecurityLog table; plain syslog lands in the Syslog table as text. Both connectors are installed from the Content hub (the Common Event Format and Syslog solutions) and both run on the Azure Monitor Agent (AMA) with a data collection rule (DCR).

CEF is a syslog message with a structured payload. After the syslog header comes CEF:0|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity| followed by key=value extensions such as src=, dst= and act=. The CEF connector's DCR uses the Microsoft-CommonSecurityLog stream, so those values become columns: DeviceVendor, DeviceProduct, SourceIP, DestinationPort, DeviceAction and so on. If the same message were collected by the Syslog connector instead, it would be one string in SyslogMessage beside generic fields such as Facility, SeverityLevel, HostName and ProcessName.

The rest of this topic covers other sources with their own paths. Azure activity events go to AzureActivity through diagnostic settings managed by Azure Policy. Azure resource logs go through resource diagnostic settings. Threat indicators go to ThreatIntelIndicators and ThreatIntelObjects. Anything without a built-in table goes to a custom _CL table. Windows Security events (Windows Security Events via AMA and Windows Event Forwarding) are covered in their own lesson.

A log forwarder is a Linux machine that receives syslog and CEF from devices that can't run an agent, and hands the messages to the Azure Monitor Agent, which sends them to the workspace. Firewalls and switches can't host AMA, so they send to the forwarder, and the forwarder is the only machine the DCR targets.

The forwarder must be an Azure resource so a DCR can select it. An Azure VM already is one. An on-premises or other-cloud server becomes one only after you install the Azure Connected Machine agent to onboard it to Azure Arc; until then it doesn't appear on the DCR's Resources tab. It needs Python 3 and a syslog daemon, rsyslog or syslog-ng. The retired Log Analytics agent (MMA) is not part of this design; AMA and DCRs replace it.

The setup runs in this order:

1. Install the Syslog and/or Common Event Format solution from the Content hub.
2. On the connector page, create a DCR: choose the forwarder on the Resources tab and the facilities and levels to collect. Creating the DCR deploys AMA to the machine.
3. Run the installation script from the connector page (Forwarder_AMA_installer.py) on the forwarder. It configures the daemon to listen for remote messages on port 514, UDP and TCP, and restarts it.
4. Point each device at the forwarder on port 514.

Step 3 matters because a syslog daemon doesn't accept remote messages by default. A DCR on its own collects only the forwarder's own local syslog; the script is what opens the daemon to the network. Inside the forwarder, the daemon passes messages to the agent on local TCP port 28330 (AMA 1.28.11 and later). That port is internal: only 514 needs to be open from the devices. When troubleshooting, tcpdump on port 514 shows whether messages reach the daemon, and on 28330 whether the daemon hands them to the agent.

A Syslog or CEF DCR chooses which syslog facilities to collect and, for each one, which severities. Facilities name the source subsystem (auth, authpriv, daemon, kern, local0–local7 and others); severities rank how serious each message is.

On the connector page you pick a minimum log level per facility, and the DCR collects that level and every more severe one. LOG_DEBUG therefore collects everything for the facility, LOG_CRIT collects Critical, Alert and Emergency, and LOG_EMERG collects only Emergency. Any requirement that is a contiguous range ending at Emergency fits the portal, which is the least-effort route.

A minimum level can't express a set that leaves out the more severe end, such as "Info and Notice only". For that, create or edit the DCR through the API or its JSON (an ARM template, for example) and list the exact severities in the facility's logLevels array. Use this route only when a requirement needs it, because the connector page itself works in minimum levels.

When the Syslog and CEF DCRs on one forwarder collect the same facility, CEF messages on that facility are stored twice: parsed in CommonSecurityLog and as text in Syslog. Microsoft documents two fixes.

• Separate the facilities. If the device lets you choose its facility, send CEF on a facility (say local5) that only the CEF DCR collects, and remove that facility from the Syslog DCR.
• Filter CEF out of the Syslog stream. If the device can't change its facility, add a transformation to the Syslog DCR that drops CEF messages, so only the CEF DCR keeps them.

The filter should test two fields. ProcessName usually contains "CEF" for CEF messages, but for sources that don't follow the RFC 3164 or 5424 header format (AMA 1.41 and later) it may not, so the message body is checked as well:

source
| where ProcessName !contains "CEF" and SyslogMessage !contains "CEF:0"

The transformation keeps a row only if neither field marks it as CEF. Either fix works by removing the overlap itself: as long as both rules collect a facility with no filter, both store its CEF messages.

The Azure Activity data connector collects subscription-level activity events by creating a diagnostic setting on each subscription through an Azure Policy assignment, and the events land in the AzureActivity table. Activity events record control-plane operations: role assignments, resource creation and deletion, policy changes, service health and so on.

From the connector page you launch the Azure Policy assignment wizard. On the Basics tab you choose the scope: the subscription whose activity log you want to stream. On the Parameters tab you choose the Microsoft Sentinel workspace as the destination. The policy (a DeployIfNotExists definition) then creates the diagnostic setting that streams the activity log to the workspace.

Two prerequisites are tested:

• Permissions. Besides read and write permissions on the workspace, you need the Owner role on the policy assignment scope, as the connector documentation requires.
• Legacy connections. The original method linked subscriptions directly to the workspace. If any subscriptions are still connected that way, disconnect them on the connector page (it lists them and offers Disconnect All) before you set up the policy-based connection.

The Azure Activity connector doesn't use AMA or a DCR on machines: the activity log is a platform log, so diagnostic settings are the only collection path.

Azure resources such as key vaults, storage accounts and public IP addresses send their resource logs to Microsoft Sentinel through diagnostic settings, created either by hand on each resource (standalone connectors) or for every resource of a type in a scope by Azure Policy (policy-managed connectors). PaaS resources can't host an agent, so a DCR that installs AMA never applies to them.

Remediation. A DeployIfNotExists policy applies automatically to resources created or updated after the assignment. Resources that already existed stay without a setting until you remediate: tick Create a remediation task in the wizard, or create one later for the assignment. A compliance scan only evaluates and reports non-compliance; it doesn't deploy anything. Full coverage of a scope therefore takes two things: the assignment for what comes later, and remediation for what is already there.

Parent and child resources. Some services split their logs across child resources. For the Azure Storage Account connector, Microsoft has you configure the account (parent) resource for the Transaction metric and each storage-type child (blob, queue, table, file) for its logs. Blob reads, writes and deletes are in the blob service's logs, so a setting on the account alone sends no blob operations.

Connector status. These connectors show as connected only if data was ingested at some point in the past 14 days, and they turn green again as soon as data arrives. A source that is quiet is not broken: the Azure DDoS Protection connector, for example, shows Connected only when protected resources are under attack.

Microsoft Sentinel takes threat intelligence from Microsoft (the Defender Threat Intelligence connector), from TAXII servers it polls (the TAXII connector), from platforms that push STIX objects (the upload API) and from analysts who create indicators by hand. Choose by who holds the data, whether it is pulled or pushed, and which STIX objects you need.

TAXII details. Providers usually publish the API root and collection ID; if only a discovery endpoint is given, query it to find the API root. The import runs from Microsoft's service, not from your network, and Microsoft publishes the client IP addresses it uses. Some servers, such as FS-ISAC's, accept only allowlisted clients, so the provider must allowlist those published Microsoft addresses, because they are the only source of the connection. The connector authenticates with whatever credentials the TAXII provider issues, if any.

Upload API details. It suits a platform that can push but can't host a TAXII server, and it is the only one of the push methods that carries non-indicator objects. The API authorizes each call against the app's Microsoft Sentinel Contributor role on the workspace; obtaining a token proves identity, not permission. Don't confuse it with Monitoring Metrics Publisher, which is the role for the Logs Ingestion API.

Matching indicators against your logs is done by analytics rules, such as the threat intelligence matching rule, which belong to the detections lesson; they don't ingest anything.

Ingestion rules filter or modify threat intelligence objects from data connectors before they are stored, and the objects are stored in ThreatIntelIndicators (indicators) and ThreatIntelObjects (other STIX objects).

An ingestion rule has conditions (such as Source, Confidence or object type) and an action. Delete skips matching objects so they aren't stored from then on (versions already ingested stay in the table); Edit changes them, for example extending Valid until or adding tags. Rules run in their configured order and need no code or extra Azure resources. A new or edited rule can take up to 15 minutes to take effect. Their scope is the key limit: they apply only to data from data connectors. Objects added through the upload API or created manually are not affected, so those sources need their own filtering. Automation rules and playbooks act on incidents and alerts, after the fact, and are not the tool for shaping a feed.

Tables. Microsoft introduced ThreatIntelIndicators and ThreatIntelObjects in 2025 and stopped populating the legacy ThreatIntelligenceIndicator table after July 31, 2025. Custom queries, analytics rules, workbooks and automation that still read the legacy table show no new data and must be rewritten against the new tables. Indicators (IP addresses, domains, file hashes, URLs) are in ThreatIntelIndicators; threat actors, attack patterns, identities and relationships are in ThreatIntelObjects.

Querying the latest state. Each update to an indicator adds a new row, and deletion is recorded with IsDeleted. To list live indicators, take the latest row per indicator first, then drop deleted ones:

ThreatIntelIndicators
| where ObservableKey == "ipv4-addr:value"
| summarize arg_max(TimeGenerated, *) by Id
| where IsDeleted == false

The order matters. Filtering IsDeleted == false before arg_max would keep an older, not-yet-deleted row for an indicator that has since been deleted.

A custom log table is a table you define in the Log Analytics workspace, with a name ending in _CL, for data that has no built-in Microsoft Sentinel table. Use it for line-of-business applications and other sources with their own fields.

In the portal, open the workspace, go to Tables > Create and choose a new custom log table that uses a data collection rule. You then:

1. Name the table; the portal adds the _CL suffix.
2. Choose the table plan.
3. Select an existing DCR or create one (and a data collection endpoint if your setup needs one).
4. Upload a JSON sample file; the portal builds the schema from it.
5. Optionally write a transformation in KQL that shapes each record before it is stored.

Every table needs a TimeGenerated column of type datetime. If the sample has none, the portal says it will add a transformation that creates one, and you can edit it. If the default (the processing time) isn't what detections need, assign the event's own timestamp instead, converting it from a string:

source
| extend TimeGenerated = todatetime(EventTime)

The column's type is what counts: a string value must be converted to datetime, whatever the column is called. Retention and table plans in general are covered in the platform configuration lesson.

Data reaches a custom table through the Logs Ingestion API when an application can send it over HTTPS, or through the Custom Logs via AMA connector when the source only writes text files.

Logs Ingestion API. Register an Entra application, then on the DCR's Access control (IAM) assign the app the Monitoring Metrics Publisher role; that is the least privilege the endpoint checks, and it is checked on the DCR, not on the workspace. The client posts to the DCR's own logs ingestion endpoint with the DCR immutable ID and the stream name. A data collection endpoint (DCE) is needed only when you connect over private link or the DCR has no logs ingestion endpoint. For private link, create the DCE in the same region as the DCR and workspace, add it to the Azure Monitor Private Link Scope (AMPLS), reference it in the DCR, and have the client post to the DCE's logs ingestion URI.

Custom Logs via AMA (in preview) creates a DCR that tells the agent which files to tail through a file pattern, which can include wildcards for rotating files, and sends each line to a _CL table. Create the table first, with only TimeGenerated (datetime) and one string column: RawData when the agent reads files on the machine itself, or Message when logs come from appliances through a Linux log forwarder (they send syslog to the forwarder on port 514, and the daemon writes the file that the agent collects). As with Syslog, a non-Azure forwarder needs the Azure Arc Connected Machine agent so the DCR can select it. When you pick a device type from the connector's list, the DCR comes with a transformation for that format; leave it as populated.