Kubernetes Scheduling: Requests, Node Affinity, Taints and Tolerations

The kube-scheduler is the control plane component responsible for placement. It continuously watches the API server for Pods that have no nodeName set, and for each one it runs a scheduling cycle with two phases. First comes filtering: the scheduler eliminates every node that cannot run the Pod at all. A node is filtered out if it lacks enough allocatable CPU or memory to cover the Pod's requests, if its labels do not satisfy the Pod's nodeSelector or required node affinity, if it carries a taint the Pod does not tolerate, or if a required host port is already in use. The nodes that survive are called feasible nodes.

Second comes scoring: the scheduler ranks the feasible nodes to pick the best one. Scoring considers signals such as how much free capacity a node would have left, whether the node already has the container image cached locally, and how well the node satisfies any preferred (soft) affinity rules. The scheduler binds the Pod to the highest-scoring node by writing that node's name into the Pod object.

Two boundaries matter for the exam. If filtering leaves zero feasible nodes, the Pod is not scheduled anywhere: it remains in the Pending state and the scheduler retries as the cluster changes. And the scheduler only decides placement; it never starts containers. Once a Pod is bound, the kubelet on the chosen node notices the assignment, pulls the images through the container runtime, and starts the containers. Scheduler picks, kubelet runs.

Every container in a Pod can declare CPU and memory requests and limits. A request is the amount the container is guaranteed and, critically, the number the scheduler uses: during filtering, a node is feasible only if its remaining allocatable capacity covers the sum of the Pod's requests. The scheduler compares requests against what is already requested on the node, not against live usage, so a node full of idle-but-greedy Pods still counts as full. A limit is different: it plays no part in scheduling and is instead enforced at runtime on the node.

CPU is measured in cores or millicores, where 500m means half a core; memory is measured in bytes with suffixes such as Mi and Gi. The two resources behave differently when a container hits its limit. CPU is compressible: exceeding the CPU limit gets the container throttled, slowing it down but not killing it. Memory is not compressible: exceeding the memory limit gets the container OOMKilled and restarted.

A typical declaration looks like this:

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

Requests and limits also determine a Pod's Quality of Service class: Guaranteed when every container's requests equal its limits, Burstable when requests are set but lower than limits, and BestEffort when neither is set. Under node memory pressure, BestEffort Pods are evicted first.

The most basic way to steer a Pod toward particular nodes is nodeSelector. Nodes carry labels, which are plain key-value pairs; some are added automatically by Kubernetes, such as kubernetes.io/hostname and the topology labels for region and zone, and administrators add their own to describe hardware or roles:

kubectl label nodes worker-2 disktype=ssd

A Pod then declares the labels a node must have:

spec:
  nodeSelector:
    disktype: ssd

The semantics are strict and simple. Every key-value pair in the nodeSelector must match the node's labels exactly, and it is purely a hard filter: a node either qualifies or it does not. There is no way to express a preference, no way to say a label's value should be one of several options, and no way to say a label must be absent. If no node in the cluster carries the required labels, the Pod is not placed on some second-best node; it simply stays Pending until a matching node appears.

For the exam, recognize nodeSelector as the entry-level mechanism: exact label match, hard requirement only. When a question mentions needing expressions like membership in a set of values, or a soft preference that the scheduler should try to honor but may ignore, the answer is node affinity, which exists precisely because nodeSelector cannot express those things.

Node affinity is the expressive successor to nodeSelector. It lives under spec.affinity.nodeAffinity and comes in two flavors whose long names the exam expects you to recognize. requiredDuringSchedulingIgnoredDuringExecution is a hard rule: it behaves like nodeSelector in that a node failing the rule is filtered out and an unsatisfiable rule leaves the Pod Pending. preferredDuringSchedulingIgnoredDuringExecution is a soft rule: it only influences the scoring phase, each preference carries a weight from 1 to 100, and if no node satisfies it the Pod is scheduled anyway on the best available node.

The power comes from match expressions. Instead of exact equality only, affinity supports operators: In (the label's value is in a given set), NotIn, Exists (the label key is present, any value), DoesNotExist, plus Gt and Lt for numeric comparison. There is no separate anti-affinity field for nodes; node anti-affinity is simply an affinity rule written with NotIn or DoesNotExist, keeping Pods away from nodes with certain labels.

The suffix IgnoredDuringExecution is itself exam material: it means the rule is evaluated only when the Pod is scheduled. If a node's labels change afterward so that a running Pod no longer satisfies its required affinity, Kubernetes does not evict the Pod; it keeps running where it is. Affinity constrains placement, not continued residence.

A minimal hard rule looks like this:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values:
          - ssd

Node affinity places Pods relative to node labels. Pod affinity and pod anti-affinity place Pods relative to other Pods: schedule me near Pods matching this label selector, or keep me away from them. Both come in the same required (hard) and preferred (soft) flavors as node affinity, with the same IgnoredDuringExecution behavior.

The distinctive ingredient is the topologyKey, which defines what near means. It names a node label, and all nodes sharing the same value for that label count as one topology domain. With topologyKey: kubernetes.io/hostname the domain is a single node, so anti-affinity means the Pods land on different nodes. With topologyKey: topology.kubernetes.io/zone the domain is an availability zone, so anti-affinity spreads Pods across zones.

The canonical use cases are worth recognizing on sight. Pod affinity co-locates workloads that benefit from proximity, such as placing a cache Pod in the same zone as the web Pods that read from it, reducing latency. Pod anti-affinity spreads replicas of the same application across nodes or zones so that the loss of one node or one zone cannot take down every replica at once; this is a standard high-availability pattern for databases and other replicated services.

One practical note the exam may echo: inter-Pod affinity is significantly more expensive for the scheduler to compute than node affinity, because it must consider the Pods already running across the cluster, so it is recommended for clusters up to a few hundred nodes rather than very large ones.

Taints and tolerations work in the opposite direction from affinity. Affinity is a property of Pods that attracts them to nodes. A taint is a property of a node that repels Pods: once a node is tainted, no Pod may schedule onto it unless the Pod carries a matching toleration. The direction of each mechanism is a favorite exam discriminator:

• nodeSelector and node affinity: set on the Pod, attract or restrict the Pod to certain nodes.
• Taints: set on the node, repel all Pods that do not tolerate them.
• Tolerations: set on the Pod, permit scheduling onto tainted nodes but do not attract or force the Pod there.

A taint has a key, an optional value, and an effect, applied with a command like kubectl taint nodes node1 gpu=true:NoSchedule. The three effects are: NoSchedule, new Pods without a toleration are not placed here, but Pods already running stay; PreferNoSchedule, a soft version the scheduler tries to honor but may violate; and NoExecute, the strictest, which blocks new placement and evicts Pods already running on the node that lack a toleration. A toleration may include tolerationSeconds to delay that eviction.

Kubernetes uses taints itself: control plane nodes typically carry node-role.kubernetes.io/control-plane:NoSchedule, which is why ordinary workloads do not land on them, and the node controller adds NoExecute taints such as node.kubernetes.io/not-ready to unhealthy nodes to drive eviction. Because a toleration only permits and never attracts, dedicating a node pool to special workloads takes both mechanisms: taint the nodes to keep everyone else out, and give the special Pods a toleration plus a node affinity so they actually go there.

Most workload controllers think in counts: a Deployment with replicas: 3 wants three Pods somewhere, and the scheduler is free to place them on any feasible nodes. A DaemonSet thinks in coverage: it runs exactly one copy of a Pod on every eligible node in the cluster. When a new node joins, the DaemonSet controller creates a Pod for it automatically; when a node is removed, its Pod is garbage collected. You never set a replica count on a DaemonSet, because the node count is the replica count.

The classic use cases are per-node infrastructure agents, and the exam expects you to recognize them: log collectors such as Fluentd or Fluent Bit, node monitoring agents such as the Prometheus node exporter, storage daemons, and cluster networking components (CNI plugin agents and kube-proxy commonly run as DaemonSets). Anything that must observe or serve the node itself, rather than a fixed number of application replicas, fits the pattern.

Mechanically, the DaemonSet controller creates one Pod targeted at each eligible node and, in current Kubernetes, the default scheduler binds them using node affinity that the controller injects per Pod. Eligibility still respects the normal rules: a DaemonSet can use a nodeSelector or affinity to cover only a subset of nodes, such as only nodes labeled for logging. DaemonSet Pods are also given tolerations automatically for several node-condition taints, such as not-ready and unreachable, so a monitoring agent keeps running on a node that regular workloads are being evicted from, which is exactly when you need it most.

Here is the scenario the exam most likes to probe. You deploy a Pod that requests 3 CPUs to a three-node cluster. Each node has 4 allocatable CPUs, but existing workloads have already requested 2 CPUs on every node, leaving 2 free per node. Filtering removes all three nodes, because no single node has 3 unrequested CPUs, even though the cluster as a whole has 6 free. Requests are per-node, never pooled across the cluster. The Pod sits in Pending, and the first diagnostic step is:

kubectl describe pod myapp

The Events section at the bottom tells you exactly what filtering rejected, with messages like 0/3 nodes are available: 3 Insufficient cpu. Other tell-tale messages map to the mechanisms in this lesson: node(s) didn't match Pod's node affinity/selector means a nodeSelector or required affinity is unsatisfied, and node(s) had untolerated taint means every remaining node repels the Pod. The fixes follow directly: lower the request, add or enlarge nodes, label a node, relax the affinity, or add a toleration.

Two distinctions round out the picture. First, Pending is a scheduling problem, not a runtime problem: a Pod in CrashLoopBackOff or ImagePullBackOff was scheduled successfully and failed afterward on the node, so the scheduler is not the suspect. Second, Kubernetes supports priority and preemption: a Pending Pod with a higher PriorityClass can cause the scheduler to evict lower-priority Pods from a node to make room, which is the one built-in way a Pending Pod gets a node without the cluster otherwise changing.