CCNA SNMP and Syslog: SNMPv3, Traps, Informs and Syslog Severity Levels

SNMP (Simple Network Management Protocol) lets a central network management system read and change management data on network devices and lets those devices report events to it. It is the standard way an operations team collects interface counters, CPU and memory load, and device status from hundreds of routers and switches, and the way devices raise an alarm when something changes.

SNMP has three building blocks:

• SNMP manager — software on the NMS (network management station). It sends requests, receives notifications, stores the results and draws the graphs and alarms.
• SNMP agent — a process on each managed device (router, switch, firewall, server). It answers the manager's requests and sends notifications. The agent, not the manager, holds the device's live values.
• MIB (Management Information Base) — the structured set of objects the agent exposes, such as interface status, byte counters, CPU load and the hostname.

Every MIB object is identified by an OID (object identifier), a dotted number in a global tree, for example 1.3.6.1.2.1.2.2.1.10 for interface inbound octets (ifInOctets). Standard objects live under the mib-2 branch (1.3.6.1.2.1); vendor-specific objects live under the private enterprises branch 1.3.6.1.4.1, with each vendor holding its own number (Cisco's is 9).

The OID alone is just a number. To show an object by name, with its type and meaning, the NMS needs the MIB file that defines it. Standard MIBs ship with every NMS; for vendor objects you load (compile) the vendor's MIB files into the NMS. A value that comes back as a bare numeric OID still proves that polling works: naming and interpreting objects is purely a job of the MIB files loaded on the NMS, while communities, versions and notification settings decide only whether the exchange happens at all.

The manager reads and writes with Get, GetNext, GetBulk and Set on UDP port 161 of the agent; the agent sends notifications (traps and informs) to the manager on UDP port 162. SNMP normally runs over UDP.

Two consequences matter in practice:

• Firewall rules. Polling is a flow the NMS starts toward the device on UDP 161; a stateful firewall lets the responses back automatically. Notifications are a separate flow the device starts toward the NMS on UDP 162, so they need their own rule. UDP 514 is syslog, not SNMP.
• Walking big tables. SNMPv1 has only Get and GetNext, so walking the interface table of a large switch costs one exchange per object. GetBulk, added in SNMPv2, retrieves many rows at once, so the number of request/response exchanges needed to read a table depends on the operation used, which in turn depends on the SNMP version; the polling interval only changes how often that work is repeated.

A trap is sent once with no acknowledgment; an inform is acknowledged by the manager and resent by the agent until it is acknowledged or its retries run out. Both are notifications from agent to manager on UDP 162.

On Cisco IOS you enable the notification types with snmp-server enable traps and name each receiver with snmp-server host. The keyword after the address chooses the message type; traps is the default when it is left out.

snmp-server enable traps
! v1 traps (the default type)
snmp-server host 10.9.9.9 version 1 NOC-STR
! acknowledged notifications need informs, which need v2c or v3
snmp-server host 10.9.9.9 informs version 2c NOC-STR

If notifications are being lost over a congested link and must be confirmed, two things have to change together: the message type (informs) and, if the host is set to version 1, the version (2c or 3), because SNMPv1 has no inform message. The snmp-server host command only defines where notifications go; it does not grant or control polling access.

SNMPv1 and SNMPv2c authenticate requests only with a community string sent in clear text, while SNMPv3 uses named users with optional message authentication and encryption. Anyone who captures v1 or v2c traffic can read the community string and every value polled.

SNMPv3 has three security levels:

A capture tells you the level: an authentication parameter with readable OIDs and values means authNoPriv; unreadable data means authPriv. Community-string hygiene limits who can use the string and what they can do with it, but the string and the data still cross the wire in clear text; only SNMPv3 at authPriv provides authenticated and encrypted SNMP.

On IOS, v3 is configured in two parts. The group sets the minimum security level and the views (what may be read, written or sent in notifications); the user belongs to a group and carries the keys:

snmp-server group NMS-RO v3 priv
snmp-server user nms1 NMS-RO v3 auth sha Auth#Key1 priv aes 128 Priv#Key2

A group configured with priv requires authPriv, so its users need both an auth algorithm and key and a priv algorithm and key. A group with no write view gives read-only access. A group with auth stops at authNoPriv, and a user defined as v2c or a snmp-server community line provides no per-user authentication or encryption at all.

Restricting access with communities and ACLs

A community string grants either read-only (RO) or read-write (RW) access, and a standard ACL attached to it limits which NMS source addresses the agent will answer.

access-list 15 permit host 10.10.5.20
snmp-server community Ops-Read RO 15

• RO allows Get, GetNext and GetBulk. RW also allows Set, which can change the configuration — give it out sparingly, if at all.
• The number after RO or RW applies that ACL to the community. Requests from addresses the ACL permits are answered; anything else hits the implicit deny and is silently dropped, so the NMS sees a timeout, not an error.
• Because the drop is silent, a timeout from one source while other sources are answered points to the ACL attached to the community: every NMS address that should poll needs its own permit.
• snmp-server host lines only name notification receivers. A missing host line stops traps or informs reaching that address; it does not stop the address polling the device.

The ACL limits who can talk to the agent but does not hide the community string on the wire, so it complements, rather than replaces, SNMPv3.

Use SNMP polling for values that change continuously and must be trended, and use event-driven messages — SNMP notifications or syslog — when you need to know quickly that something happened.

The two methods are usually combined. The general rule: polling detects a change no sooner than the next poll, and polling faster multiplies the load on every device, so fast detection comes from the device pushing the event rather than from a shorter interval. Pushing only helps if the message actually leaves the device — a local log buffer records events but alerts nobody.

Syslog has eight severity levels numbered 0 to 7, where a lower number means a more serious condition: 0 emergencies is the worst and 7 debugging the least severe.

A common memory aid is "Every Awesome Cisco Engineer Will Need Ice-cream Daily". Be careful with the order of the top three: alerts (1) is more severe than critical (2), even though "critical" sounds worse. Also note that the examples are typical IOS levels, not fixed rules: the severity is whatever number the message carries.

A Cisco IOS log message is made of an optional sequence number, an optional timestamp, a %FACILITY-SEVERITY-MNEMONIC tag and a description of the event.

000123: *Mar  1 00:12:45.331: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

"Facility" therefore has two meanings in syslog work: the IOS message facility in the tag (the subsystem, such as OSPF), and the syslog protocol facility (such as local7) that a server uses for filing — covered below.

Reading a sequence of messages

Messages tell a story when read together. Knowing a few common ones lets you tell an administrative action from a fault:

• %SYS-5-CONFIG_I: Configured from console by jsmith on vty0 (10.2.2.8) — a configuration change; vty0 with an address means a remote session (Telnet/SSH), while console as the line means the physical console port.
• %LINK-5-CHANGED: … changed state to administratively down — someone entered shutdown on the interface.
• %LINK-3-UPDOWN: … changed state to down — the physical link was lost (cable, far end, optics); note the higher severity (3) for a fault.
• %LINEPROTO-5-UPDOWN — the line protocol followed the link state.
• %OSPF-5-ADJCHG: … from FULL to DOWN, Neighbor Down: Interface down or detached — the adjacency dropped because its interface went down; a dead-timer expiry gives a different reason ("Dead timer expired").

Read a sequence in time order and ask what each message implies about the next: a configuration message names who changed the device and from which line, a link message says whether the interface was shut or failed, and protocol messages that follow are usually consequences of that link change rather than separate faults.

Each logging destination has a severity threshold: the device sends that level and every more severe (lower-numbered) level, and drops the rest. Setting logging trap warnings sends levels 0–4 to the server and holds back 5, 6 and 7.

IOS can send log messages to four destinations, each with its own command and threshold:

logging host 10.1.1.50
logging trap notifications

With this configuration an OSPF adjacency change (level 5) reaches the server but ACL log hits (level 6) do not — a simple way to stop a noisy informational message flooding the server while keeping significant events. Note which command controls which destination: logging trap is the server threshold (the name is historical and has nothing to do with SNMP traps); changing logging buffered or logging console does not change what the server receives.

Two practical points:

• When you run a debug command over SSH and see nothing while a colleague on the console does, enter terminal monitor in your session. logging synchronous under the line only stops messages breaking up what you are typing.
• Debug output is level 7, so it reaches a destination only if that destination's threshold is debugging.

The syslog protocol facility is a header field that tells the receiving server what kind of sender produced the message, and IOS stamps messages it sends to a server with facility local7 by default. Servers commonly sort or file messages by facility, so if routers and firewalls both arrive as local7 they end up together.

• Change it with logging facility local5 (any of local0–local7, the facilities reserved for local use, plus a few system facilities).
• The facility is a separate setting from which severities are sent to the server and from which address they are sent from; changing the facility changes only how the receiving server classifies the messages.

On the wire, facility and severity are combined into one PRI (priority) value at the start of the message, in angle brackets:

PRI = facility number × 8 + severity

The local facilities are numbered 16 (local0) to 23 (local7). To decode a PRI, divide by 8: the quotient is the facility and the remainder is the severity.

Logs from several devices can only be compared when every device stamps messages with calendar time, from a clock synchronised to a common source, at enough precision to separate events.

• service timestamps log datetime msec stamps log messages with date and time to the millisecond. Without datetime a device can stamp with time since boot (uptime, such as 2w3d:), which cannot be placed in calendar time. Without msec the stamp stops at whole seconds, so events in the same second cannot be ordered. Optional keywords localtime and show-timezone use the configured time zone.
• service timestamps debug … is the separate command for debug output; it does not change normal log messages.
• A leading * on the timestamp means the clock is not authoritative — it has not been set from a reliable source. A date such as Mar 1 on a device that was never set is a typical sign. (A leading . means the clock was synchronised but has lost synchronisation.) Times from such clocks cannot be compared across devices.
• NTP fixes accuracy: pointing every device at the same NTP time source makes their clocks agree, so their log times line up.
• service sequence-numbers numbers messages on each device. It orders messages from one device (and shows if any were lost), but the numbers are per device and say nothing about time, so they cannot order events across devices.

To order events across two switches you therefore need both: a common NTP source for accuracy and millisecond datetime stamps for precision.