Kubernetes Scheduling: Requests, Node Affinity, Taints and Tolerations

The kube-scheduler is the control plane component responsible for placement. It continuously watches the API server for Pods that have no nodeName set, and for each one it runs a scheduling cycle with two phases. First comes filtering: the scheduler eliminates every node that cannot run the Pod at all. The nodes that survive are called feasible nodes. Second comes scoring: each feasible node is ranked, and the scheduler binds the Pod to the highest-scoring node by writing that node's name into the Pod object.

Both phases are made of plugins, and knowing which checks are hard and which are soft is the key to most placement questions:

Each score plugin rates every feasible node, the scheduler multiplies each plugin's score by that plugin's weight, and it sums them. No single signal wins on its own: a node that caches the image can still lose to a node that matches a heavily weighted preferred affinity.

Two boundaries matter for the exam. If filtering leaves zero feasible nodes, the Pod is not scheduled anywhere: it remains Pending and the scheduler retries as the cluster changes. And the scheduler only decides placement; it never starts containers. Once a Pod is bound, the kubelet on the chosen node notices the assignment, pulls the images through the container runtime, and starts the containers. Scheduler picks, kubelet runs.

Because the scheduler only handles Pods whose nodeName is empty, a manifest that sets spec.nodeName directly bypasses the scheduler entirely: no scheduler filtering or scoring, and NoSchedule taints are ignored. Only the kubelet on the named node checks the Pod, with its own admission: if the node lacks room it rejects the Pod with a reason such as OutOfcpu or OutOfmemory, a nodeSelector or required node affinity the node does not match is rejected too, and an untolerated NoExecute taint still ejects the Pod. Setting nodeName is rarely the right tool; nodeSelector or affinity keep the scheduler's safety checks.

Every container in a Pod can declare CPU and memory requests and limits. A request is the amount the container is guaranteed and, critically, the number the scheduler uses: during filtering, a node is feasible only if its allocatable capacity minus everything already requested on it covers the sum of the Pod's requests. Live usage is ignored, so a node full of idle-but-greedy Pods still counts as full. A limit plays no part in scheduling and is instead enforced at runtime on the node.

CPU is measured in cores or millicores, where 500m means half a core; memory is measured in bytes with suffixes such as Mi and Gi. The two resources behave differently when a container reaches its limit. CPU is compressible: the kernel throttles the container's CPU time to stay within the limit, slowing it down but not killing it. Memory is not compressible: the allocation is not simply refused; when the container exceeds its memory limit the kernel's OOM killer terminates the process, the container status reports OOMKilled, and the Pod's restart policy decides whether it is restarted.

A typical declaration looks like this:

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

Two validation rules catch people out. First, if a container sets a limit but no request, and no LimitRange in the namespace supplies a default request, Kubernetes sets the request equal to the limit. The full limit then counts against node capacity at scheduling time, so a container with only limits: cpu: 2 needs a node with 2 unrequested CPUs. Second, a limit lower than its request is invalid and the API server rejects the Pod.

Requests and limits also determine a Pod's Quality of Service class: Guaranteed when every container has CPU and memory requests equal to its limits, BestEffort when no container sets any request or limit, and Burstable for everything in between. Under node memory pressure the kubelet ranks Pods for eviction by whether their usage exceeds their requests, then by Pod priority, then by how far usage exceeds the request; in practice BestEffort Pods (request zero) and Pods using more than they requested go before Guaranteed ones.

The most basic way to steer a Pod toward particular nodes is nodeSelector. Nodes carry labels, which are plain key-value pairs; some are added automatically by Kubernetes, such as kubernetes.io/hostname and the topology labels for region and zone, and administrators add their own to describe hardware or roles:

kubectl label nodes worker-2 disktype=ssd

A Pod then declares the labels a node must have:

spec:
  nodeSelector:
    disktype: ssd

The semantics are strict and simple. Every key-value pair in the nodeSelector must match the node's labels exactly, and when you list several pairs they are ANDed: a node must carry all of them, so a node matching only one of two listed labels is filtered out. It is purely a hard filter: a node either qualifies or it does not. There is no way to express a preference, no OR (a label's value being one of several options, or matching either of two labels), and no way to say a label must be absent. If no node in the cluster carries the required labels, the Pod is not placed on some second-best node; it simply stays Pending until a matching node appears.

For the exam, recognize nodeSelector as the entry-level mechanism: exact label match, hard requirement only. When a question mentions needing expressions like membership in a set of values, or a soft preference that the scheduler should try to honor but may ignore, the answer is node affinity, which exists precisely because nodeSelector cannot express those things.

Node affinity is the expressive successor to nodeSelector. It lives under spec.affinity.nodeAffinity and comes in two flavors whose long names the exam expects you to recognize. requiredDuringSchedulingIgnoredDuringExecution is a hard rule: it behaves like nodeSelector in that a node failing the rule is filtered out and an unsatisfiable rule leaves the Pod Pending. preferredDuringSchedulingIgnoredDuringExecution is a soft rule: it only influences the scoring phase, each preference carries a weight from 1 to 100, and if no node satisfies it (for example, the preferred nodes are full) the Pod is scheduled anyway on the best other feasible node.

The power comes from match expressions. Instead of exact equality only, affinity supports operators: In (the label's value is in a given set), NotIn, Exists (the label key is present, any value), DoesNotExist, plus Gt and Lt for numeric comparison. There is no separate anti-affinity field for nodes; node anti-affinity is simply an affinity rule written with NotIn or DoesNotExist, keeping Pods away from nodes with certain labels.

How expressions combine is a precise rule. A required rule holds a list of nodeSelectorTerms; the expressions inside one term are ANDed, while separate terms are ORed (a node qualifies if it satisfies any one term). So "zone a or zone b" is a single expression zone In [a, b], and putting two unrelated expressions in one term means both must hold. Required and preferred rules also combine: a Pod can require a set of zones and, inside that set, prefer SSD nodes without depending on them.

The suffix IgnoredDuringExecution is itself exam material: it means the rule is evaluated only when the Pod is scheduled. If a node's labels change afterward so that a running Pod no longer satisfies its required affinity, Kubernetes does not evict the Pod; it keeps running where it is. Affinity constrains placement, not continued residence.

A hard zone rule plus a soft disk preference looks like this:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values: [a, b]
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 50
      preference:
        matchExpressions:
        - key: disktype
          operator: In
          values: [ssd]

Node affinity places Pods relative to node labels. Pod affinity and pod anti-affinity place Pods relative to other Pods: schedule me near Pods matching this label selector, or keep me away from them. Both come in the same required (hard) and preferred (soft) flavors as node affinity, with the same IgnoredDuringExecution behavior.

The distinctive ingredient is the topologyKey, which defines what near means. It names a node label, and all nodes sharing the same value for that label count as one topology domain. With topologyKey: kubernetes.io/hostname the domain is a single node, so required anti-affinity means at most one matching Pod per node. With topologyKey: topology.kubernetes.io/zone the whole zone is one domain, no matter how many nodes it has, so required anti-affinity allows at most one matching Pod per zone: in a two-zone cluster a third replica stays Pending even if every node has spare capacity.

The canonical use cases are worth recognizing on sight. Pod affinity co-locates workloads that benefit from proximity, such as placing a cache Pod on the same node (hostname key) or in the same zone (zone key) as the web Pods that read from it. Pod anti-affinity spreads replicas of the same application across nodes or zones so that the loss of one node or one zone cannot take down every replica at once; this is a standard high-availability pattern for databases and other replicated services.

Preferred (soft) anti-affinity never blocks scheduling and keeps nudging each new replica toward the domains holding fewer matching Pods, because its weight counts against a domain once per matching Pod already there. But it is only one weighted score among several plugins, so it can be outweighed (for example by resource balance or image locality), and it sets no limit on imbalance. When you need replicas balanced to a stated tolerance, with more than one per domain allowed, use topology spread constraints (next section), which state the allowed imbalance explicitly as maxSkew.

One practical note the exam may echo: inter-Pod affinity is significantly more expensive for the scheduler to compute than node affinity, because it must consider the Pods already running across the cluster, so it is not recommended for clusters larger than several hundred nodes.

Topology spread constraints (spec.topologySpreadConstraints) tell the scheduler to keep matching Pods evenly distributed across topology domains such as zones or nodes, while still allowing several Pods per domain. They are the built-in answer to "balance my replicas across zones".

Each constraint has four core fields:

• topologyKey: the node label that defines a domain, for example topology.kubernetes.io/zone or kubernetes.io/hostname.
• labelSelector: which Pods are counted, normally the workload's own Pod labels.
• maxSkew: the largest allowed imbalance. Skew for a domain is the number of matching Pods in it minus the number in the least-populated eligible domain.
• whenUnsatisfiable: what to do if no placement keeps skew within maxSkew.

Work through it with numbers. With maxSkew: 1 on zones and current counts of 2, 1 and 1, the minimum is 1. Placing the new Pod in the zone that already has 2 would give 3 there, a skew of 2, which breaks the limit; placing it in either zone with 1 gives a skew of 1, which is allowed.

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: web

Compare the near-twins carefully:

Clusters also apply built-in default spreading as a scoring preference, which is why replicas of one Deployment tend to land on different nodes even with no constraint declared; declare your own constraint when the spread is a requirement.

Taints and tolerations work in the opposite direction from affinity. Affinity is a property of Pods that attracts them to nodes. A taint is a property of a node that repels Pods: once a node is tainted, no Pod may schedule onto it unless the Pod carries a matching toleration. The direction of each mechanism is a favorite exam discriminator:

• nodeSelector and node affinity: set on the Pod, attract or restrict the Pod to certain nodes.
• Taints: set on the node, repel all Pods that do not tolerate them.
• Tolerations: set on the Pod, permit scheduling onto tainted nodes but do not attract or force the Pod there.

A taint has a key, an optional value, and an effect. You add one with kubectl taint and remove it by repeating the taint with a trailing minus:

kubectl taint nodes node1 gpu=true:NoSchedule
kubectl taint nodes node1 gpu=true:NoSchedule-

The three effects are: NoSchedule, new Pods without a toleration are not placed here, but Pods already running stay; PreferNoSchedule, a soft version the scheduler tries to avoid but may still use; and NoExecute, the strictest, which blocks new placement and evicts Pods already running on the node that lack a toleration. A Pod that tolerates a NoExecute taint can set tolerationSeconds to stay bound for that long and then be evicted.

Kubernetes uses taints itself. kubeadm puts node-role.kubernetes.io/control-plane:NoSchedule on control-plane nodes, which is why ordinary workloads never land there even though the node runs a kubelet and has capacity. When a node fails, the node controller adds node.kubernetes.io/not-ready:NoExecute or node.kubernetes.io/unreachable:NoExecute, and Kubernetes gives every Pod default tolerations for those two taints with tolerationSeconds: 300. That is why Pods stay bound to a failed node for about five minutes before they are evicted and replaced elsewhere.

Cordon versus taint. kubectl cordon sets spec.unschedulable on a node, which shows up as the taint node.kubernetes.io/unschedulable:NoSchedule: new Pods without a toleration for it are kept off (DaemonSet Pods tolerate it), and nothing running is evicted. kubectl drain cordons and then evicts the Pods.

Because a toleration only permits and never attracts, dedicating a node pool to special workloads takes both directions: taint the nodes to keep everyone else out, and give the special Pods the toleration plus a hard pull, a nodeSelector or required node affinity on the pool's label, so they can only go there. A preferred node affinity is not enough: it only adds score, so when the dedicated nodes are busy the Pods can still land on general nodes.

Most workload controllers think in counts: a Deployment with replicas: 3 wants three Pods somewhere, and the scheduler is free to place them on any feasible nodes. A DaemonSet thinks in coverage: it runs exactly one copy of a Pod on every eligible node in the cluster. When a new node joins, the DaemonSet controller creates a Pod for it automatically; when a node is removed, its Pod is garbage collected. You never set a replica count on a DaemonSet, because the node count is the replica count.

The classic use cases are per-node infrastructure agents, and the exam expects you to recognize them: log collectors such as Fluentd or Fluent Bit, node monitoring agents such as the Prometheus node exporter, storage daemons, and cluster networking components (CNI plugin agents and kube-proxy commonly run as DaemonSets). Anything that must observe or serve the node itself, rather than a fixed number of application replicas, fits the pattern.

Mechanically, the DaemonSet controller creates one Pod targeted at each eligible node and, in current Kubernetes, the default scheduler binds them using node affinity that the controller injects per Pod. Eligibility still respects the normal rules, and the two mechanisms push in opposite directions. A nodeSelector or node affinity in the DaemonSet's Pod template narrows the set of eligible nodes, for example to only nodes labelled role=edge; nodes that gain or lose the label gain or lose the Pod. Tolerations widen it, letting the DaemonSet also cover tainted nodes such as dedicated pools; a toleration never restricts where the Pods go. DaemonSet Pods are also given tolerations automatically for several node-condition taints, such as not-ready and unreachable, so a monitoring agent keeps running on a node that regular workloads are being evicted from, which is exactly when you need it most.

Pod priority tells the scheduler how important a Pod is relative to others, and preemption lets a higher-priority Pod that cannot be placed make room by evicting lower-priority Pods. Priority is defined by a PriorityClass, a cluster-scoped (non-namespaced) object that maps a name to an integer value; higher means more important. One PriorityClass may set globalDefault: true to apply to Pods that name no class; otherwise such Pods get priority 0. Kubernetes ships two built-in classes, system-cluster-critical and system-node-critical, for essential components.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: Customer-facing services

A Pod opts in with spec.priorityClassName: high-priority, and admission resolves that name into the Pod's numeric priority. Priority then has two effects:

• Queue order. Pending Pods are ordered by priority, so higher-priority Pods are tried first whenever capacity frees up.
• Preemption. If a Pending Pod fits on no node, the scheduler looks for a node where evicting some lower-priority Pods would make it fit. It never evicts Pods of equal or higher priority. It records that node in the Pod's status.nominatedNodeName, and the victims are terminated gracefully, within their termination grace period. The preemptor is scheduled once the room exists; the nomination is not a guarantee, and if another node frees up first the Pod may go there.

A PriorityClass may set preemptionPolicy: Never. Pods using it keep their place at the front of the queue, ahead of lower-priority Pending Pods, but they never evict running Pods; they wait for capacity to free up naturally. This suits important batch work that should go next without disrupting what is running. The default is PreemptLowerPriority.

Preemption is the one built-in way a Pending Pod gets a node without the cluster otherwise changing. With no PriorityClasses in use, every Pod has the same priority, so nothing is preempted and a Pod that fits nowhere simply waits.

Here is the scenario the exam most likes to probe. You deploy a Pod that requests 3 CPUs to a cluster with three worker nodes plus a tainted control-plane node. Each worker node has 4 allocatable CPUs, but existing workloads have already requested 2 CPUs on every node, leaving 2 unrequested per node. Filtering removes all three workers, because no single node has 3 unrequested CPUs, even though the cluster as a whole has 6 free and actual usage may be low. Requests are per node, never pooled across the cluster. The Pod sits in Pending, and the first diagnostic step is:

kubectl describe pod myapp

The Events section at the bottom shows a FailedScheduling event that accounts for every node, for example 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 Insufficient cpu. Read it as a tally: each count is how many nodes failed for that reason. Map each reason to its fix:

Taint-rejected nodes such as a control-plane node are usually not meant for the workload, so the useful fix targets the other nodes. Two distinctions round out the picture. First, Pending is a scheduling problem, not a runtime problem: a Pod in CrashLoopBackOff or ImagePullBackOff was scheduled successfully and failed afterward on the node, so the scheduler is not the suspect. Second, a higher-priority Pending Pod may be placed by preempting lower-priority Pods, as described in the previous section.