SaveMyCert
Log in
5 of 5 free questions left today·for 10 a day
Kubernetes Cluster Component Security

KCSA Control Plane Security: API Server, Controller Manager, Scheduler and etcd

20 min readKCSA · Kubernetes Cluster Component SecurityUpdated

Kubernetes control plane security is the hardening of the four components that run the cluster, the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that only authenticated and authorized requests can change cluster state and no single stolen credential or file hands over the whole cluster. This KCSA lesson covers how the API server processes each request (authentication, authorization, admission), anonymous access and authorization modes, the NodeRestriction admission plugin, TLS between the API server, etcd and kubelets, the keys that sign service account tokens and certificates, securing the controller manager and scheduler endpoints, and isolating, encrypting and backing up etcd.

What you’ll learn
  • Explain the order of authentication, authorization and admission control in the kube-apiserver and which stage rejects which kind of request
  • Describe how anonymous requests are identified, why AlwaysAllow is unsafe and why Node,RBAC is the production authorization pair
  • Use the NodeRestriction admission plugin and the node-restriction.kubernetes.io/ label prefix to limit what a compromised kubelet can change
  • Secure the connections from the API server to etcd and to kubelets with TLS that authenticates both sides
  • Protect and rotate the service account signing key and protect the cluster CA key held by the controller manager
  • Harden the controller manager and scheduler with per-controller credentials, loopback binding and profiling turned off
  • Isolate etcd, encrypt Secrets at rest with an EncryptionConfiguration and protect etcd backups and their keys

What Kubernetes control plane security covers

Control plane security is the protection of the four components that run a Kubernetes cluster: the kube-apiserver, the kube-controller-manager, the kube-scheduler and etcd. Together they decide what the cluster is allowed to do, so an attacker who controls any one of them controls far more than a single workload.

ComponentWhat it doesWhat an attacker gains by compromising itMain controls
kube-apiserverThe front door: every client, node and controller reads and writes cluster state through itWhatever the stolen identity is authorized for; a misconfigured server can expose the whole clusterAuthentication, authorization (Node,RBAC), admission plugins, TLS-only access, network restrictions
kube-controller-managerRuns the controllers that reconcile state, and signs approved certificate requestsBroad write access to cluster objects, and (through the cluster CA key it holds) the power to mint superuser certificatesPer-controller service accounts, protected key files, loopback-only endpoints, profiling off
kube-schedulerPicks a node for each new PodControl over where Pods run, including steering sensitive Pods onto a node the attacker already ownsLeast-privilege identity, loopback-only endpoint, profiling off
etcdThe key-value store holding every cluster object, including SecretsEffectively root on the cluster: direct writes skip every API server checkMutual TLS, network isolation to the API server only, encryption at rest, protected backups

The rest of this lesson takes each component in turn. Node components (kubelet, container runtime, kube-proxy and Pods) are covered in the node security lesson, and container networking, client access and storage in the lesson after it. How users authenticate (client certificates, OIDC tokens), Secrets as objects, audit logging and Pod Security Admission are taught in the Kubernetes Security Fundamentals domain.

How the API server processes a request

Every request to the kube-apiserver passes through three stages in a fixed order: authentication, then authorization, then admission control, and only a request that passes all three is stored in etcd. The order follows from what each stage needs to know.

  1. Authentication answers who is calling. Authenticators (client certificates, bearer tokens such as service account tokens, OIDC tokens, authenticating proxies, webhooks) turn the request into a username and a set of groups. A request that fails here is rejected with 401 Unauthorized.
  2. Authorization answers may this identity perform this verb on this resource. It sees the user, groups, verb, resource, namespace and name, but not the body of the object. A denial is 403 Forbidden.
  3. Admission control sees the actual object being created or changed. Mutating admission runs first and can modify it (add defaults, inject settings); validating admission runs next and can reject it for its content. A rejection here is usually reported as 403 Forbidden too (built-in plugins such as PodSecurity do this), although an admission webhook can choose its own status code and message.

The split tells you where to look when a request fails. kubectl auth can-i asks only the authorizers (through a SelfSubjectAccessReview), so it reports whether an identity holds a verb on a resource; it says nothing about whether a particular object will be accepted. Any rule that depends on what an object contains, such as a Deployment whose Pod template sets hostNetwork: true in a namespace that enforces the baseline Pod Security Standard, can only be enforced by admission, because authorization never looks inside the spec. A 401 points at credentials, and a denial on the verb points at RBAC; a denial that depends on a field's value points at admission.

Anonymous requests and authorization modes

A request with no credentials is not trusted by default: when anonymous access is enabled, the API server labels it as the user system:anonymous in the group system:unauthenticated, and the authorizers then decide what that identity may do. Anonymous callers are never in system:authenticated, which is the group every successfully authenticated user receives.

Anonymous access therefore becomes dangerous only through a binding that names one of those two subjects. With RBAC, the default roles give unauthenticated callers only health and version information (the system:public-info-viewer role), so any wider anonymous access was granted by someone, and the place to audit is the RoleBindings and ClusterRoleBindings whose subjects are system:anonymous or system:unauthenticated. A binding to system:authenticated is a different risk: it reaches every signed-in user and service account, but never a caller with no credentials. To reduce the exposure further, turn anonymous access off (--anonymous-auth=false) or, on recent releases, limit it to the health endpoints through the API server's authentication configuration file.

The authorizers in force are chosen with --authorization-mode (or the newer structured authorization configuration file). They are consulted in order, and the first one to allow or deny decides.

ModeWhat it doesUse
AlwaysAllowPermits every request that gets past authentication, so any valid credential acts as a cluster administrator. It does not change authentication or admission.Testing only; never production
AlwaysDenyRejects everythingTesting only
NodeAuthorizes kubelets (group system:nodes) to read and write only what their own node needs: their Node object, Pods bound to them, and the Secrets, ConfigMaps and volumes those Pods useProduction, together with RBAC
RBACGrants permissions from Roles and ClusterRoles bound to users, groups and service accountsProduction
WebhookAsks an external serviceWhen an external policy decision point is needed
ABACStatic policy file; hard to manageLegacy

Node,RBAC is the standard production pair, and it is what kubeadm configures. Removing Node to "harden" the server breaks kubelets: Kubernetes deliberately does not bind system:nodes to any RBAC role, because the Node authorizer handles them. RBAC grants by resource type and namespace, so it has no way to say "only the Secrets used by Pods bound to this node"; that per-node scoping is exactly what the Node authorizer adds.

Securing the API server endpoint: TLS and network reachability

Clients reach the kube-apiserver only over HTTPS on its secure port (6443 by default), and every request on that port is authenticated and authorized; being on the same host or the cluster network earns no implicit trust. Older releases also had an insecure HTTP port, bound to localhost, that skipped authentication and authorization entirely. It has been removed, so a current API server has no unauthenticated listener.

TLS protects the connection and proves the server's identity; it does not decide who may connect. Reducing who can reach the endpoint at all is a network control, and it complements, rather than replaces, authentication and RBAC:

  • Private endpoint. The strongest option: the API server is reachable only from inside the cluster's private network (VPC), so administrators need a VPN, a bastion or a connected network.
  • Authorized source ranges on a public endpoint. The endpoint stays public, but the provider accepts connections only from listed CIDR ranges and drops the rest at the network edge. Managed services expose this as an "authorized networks" or "public access CIDRs" setting; it filters by source address only, so clients authenticate exactly as before.
  • Firewalls on self-managed control plane hosts, allowing the API port only from the networks that need it.

Two controls are often confused with these. A Kubernetes NetworkPolicy governs traffic between Pods; it does not filter connections to the API server endpoint from outside the cluster. Disabling anonymous requests stops unauthenticated callers from being authorized, but the endpoint stays reachable by anyone.

NodeRestriction: limiting what a kubelet can change

The NodeRestriction admission plugin limits each kubelet to modifying its own Node object and the Pods bound to it, and stops kubelets from setting labels that administrators reserve for security decisions. It is enabled with --enable-admission-plugins=NodeRestriction and works together with the Node authorizer; kubeadm enables both by default.

The two layers do different jobs. Authorization works on verbs and resources, so the Node authorizer can say "kubelets may update Node objects", but not "only their own". Admission sees the object, so NodeRestriction narrows the update to the kubelet's own Node (identified by its system:node:<nodeName> username). Without it, a stolen kubelet credential could taint or relabel other nodes and push sensitive workloads onto the compromised one.

Label on a NodeCan the kubelet set it on its own Node?
kubernetes.io/hostname, kubernetes.io/arch, kubernetes.io/osYes
node.kubernetes.io/instance-type, topology.kubernetes.io/region, topology.kubernetes.io/zoneYes
Any label with the kubelet.kubernetes.io/ or node.kubernetes.io/ prefixYes
Any label with the node-restriction.kubernetes.io/ prefixNo: reserved for administrators

That reserved prefix is the one to use when a label marks nodes as trusted for sensitive workloads (for example node-restriction.kubernetes.io/pci-zone=true on the nodes cleared for card-payment workloads, matched by those Pods' node affinity). A kubelet that is compromised cannot add it to itself, so it cannot attract those Pods. Labels with the allowed prefixes look official but give no such guarantee.

TLS from the API server to etcd and to kubelets

The API server is a client of etcd and of every kubelet, and both connections must be encrypted and must authenticate the other side: mutual TLS for etcd, and CA verification of kubelet serving certificates for kubelets.

API server to etcd

The Kubernetes documentation recommends mutual TLS. The API server presents a client certificate (--etcd-certfile and --etcd-keyfile) and verifies etcd's server certificate against --etcd-cafile; etcd verifies the client certificate against its trusted CA (--client-cert-auth=true with --trusted-ca-file). Weaker setups each drop a half: plain TCP trusted by subnet authenticates nobody, one-way TLS authenticates only etcd, and accepting any client certificate authenticates no client. Peer traffic between etcd members (port 2380) should use TLS as well.

API server to kubelets

The API server connects to the kubelet API to serve kubectl exec, kubectl logs and port-forwarding. It authenticates to the kubelet with a client certificate (--kubelet-client-certificate and --kubelet-client-key). The reverse check is the one usually missing: unless --kubelet-certificate-authority is set, the API server does not verify the kubelet's serving certificate, so an attacker on the node network could impersonate a kubelet and capture exec sessions or logs.

The flag can only verify certificates its CA actually signed, and a kubelet's default serving certificate is self-signed. Kubelets therefore need CA-signed serving certificates before the API server can check them:

  1. Configure kubelets with serverTLSBootstrap: true so they request serving certificates through CertificateSigningRequests (signer kubernetes.io/kubelet-serving).
  2. Approve those CSRs (they are not approved automatically) so the controller manager signs them with the cluster CA.
  3. Set --kubelet-certificate-authority on the API server to that CA.

Keep the two directions apart: the API server's client certificate and the kubelet's own authentication settings protect the kubelet from its callers, while --kubelet-certificate-authority protects the API server from a fake kubelet. Kubelet-side settings (its own authentication and authorization, ports) belong to the node security lesson.

Service account signing keys and the cluster CA key

Two private keys on the control plane let their holder create trusted identities: the service account signing key mints tokens for any service account, and the cluster CA key signs client certificates for any user or group. Both must be readable only by the component that uses them.

KeyWhere it is configuredWhat a thief can do
Service account signing key (private)--service-account-signing-key-file on kube-apiserverForge a token for any service account and act with its RBAC permissions
Service account verification keys (public)--service-account-key-file on kube-apiserver (repeatable)Nothing on its own: public keys only verify tokens, never mint them, and can be published (for example through the OIDC discovery endpoint) so external systems can validate tokens
Cluster CA key--cluster-signing-key-file (with --cluster-signing-cert-file) on kube-controller-managerSign a client certificate for any user or group, including O=system:masters, a superuser that RBAC cannot restrict
Encryption-at-rest keysThe EncryptionConfiguration file, or an external KMSDecrypt Secret data copied from etcd or its backups

Service account tokens are signed JWTs, and the API server accepts any token whose signature verifies against a trusted public key. --service-account-key-file normally holds public keys; it also accepts a private key file, from which only the public half is used. The API server's own HTTPS certificate and key (--tls-cert-file, --tls-private-key-file) are a separate pair used only to serve TLS. The controller manager's --service-account-private-key-file signs the older Secret-based service account tokens.

Rotating the signing key without breaking workloads

Running Pods hold tokens signed with the current key, so rotation must overlap:

  1. Add the new public key to the verification keys on every API server, alongside the old one.
  2. Switch --service-account-signing-key-file to the new private key; new tokens use it, and old tokens still verify.
  3. When every token signed with the old key has expired (projected tokens are short-lived and refreshed by the kubelet), remove the old public key so it can no longer be abused. Legacy Secret-based service account tokens never expire, so any that were signed with the old key must be deleted or reissued first, or they stop working when the key goes.

The principle is that every API server must trust a key before anything is signed with it, and must keep trusting it until nothing signed with it is still in use.

Who signs approved certificate requests

Approval and signing are separate steps. The API server stores a CertificateSigningRequest and authorizes whoever approves it; the kube-controller-manager's csrsigning controllers then sign it with the cluster CA key (kubeadm points --cluster-signing-* at the cluster CA). That is why the controller manager's host and key files need the same protection as the CA itself.

Securing the controller manager and the scheduler

Secure the kube-controller-manager and kube-scheduler by giving them least-privilege identities and by removing network exposure of their own HTTPS endpoints: bind them to the loopback address and turn profiling off.

Per-controller credentials

By default the controller manager can run all of its controllers with its own single, powerful credential, so a flaw in one controller carries the permissions of all of them. --use-service-account-credentials=true makes each controller use its own service account in kube-system, bound to a matching system:controller:* role with only the permissions that controller needs. kubeadm sets it.

Their own endpoints

Both components serve health checks, metrics and debugging over HTTPS: the controller manager on port 10257 and the scheduler on port 10259. Two defaults matter for security:

  • --bind-address defaults to all interfaces. Setting --bind-address=127.0.0.1 keeps the endpoints usable for local health probes on the control plane host while removing them from the network.
  • --profiling defaults to true, which exposes /debug/pprof data about the process. The CIS Kubernetes Benchmark recommends --profiling=false.

What a compromised scheduler can do

The scheduler watches the API server for Pods with no nodeName and posts a Binding for each one through the API server, which authenticates and authorizes it as system:kube-scheduler before storing it. With the scheduler's credentials an attacker can bind new Pods to nodes of their choice, such as a node already compromised, pulling sensitive workloads and their mounted Secrets onto it, or crowd workloads onto chosen nodes. The scheduler does not talk to etcd or kubelets directly, does not hold the cluster CA key and cannot run commands in containers; those powers belong to other components.

Securing etcd: access and isolation

etcd stores every cluster object, so the Kubernetes documentation treats write access to etcd as equivalent to root on the entire cluster, and read access as enough to escalate quickly; only the API server should be able to reach it. Pods, Deployments, RBAC bindings and Secrets all live in etcd. A client that writes there directly bypasses authentication, authorization, admission control and audit logging: it could add a cluster-admin binding that no RBAC check or admission policy ever sees.

Controls that follow the guidance:

  • Network isolation. Allow the etcd client port (2379 by default) only from the kube-apiserver hosts, and the peer port (2380) only between etcd members. Worker nodes, the Pod network and administrators' workstations have no business connecting, so a compromised node or Pod cannot even try. Running etcd on dedicated hosts behind a firewall is common for self-managed clusters.
  • Mutual TLS with client certificates issued to the API server, as described above.
  • No extra etcd clients. Granting another component read or write access to the full keyspace is equivalent to granting it cluster-admin. If something other than the API server truly needs etcd, the guidance is a separate etcd instance or etcd ACLs limited to a subset of keys.

A tool that wants to read cluster state, such as a monitoring system, should read through the API server with a ServiceAccount bound to a narrow read-only role. That keeps authorization and auditing in the path. Any route that hands it etcd data, whatever the certificate, host or file format, gives it all of cluster state, including every Secret, with no authorization or audit in the path.

Encrypting Secrets at rest and protecting etcd backups

Encryption at rest makes the API server encrypt chosen resources, usually Secrets, before writing them to etcd; without it, Secret values sit unencrypted in etcd and in every backup, so anyone with a copy can read them. (Base64 is only how Secret data is written in the API's YAML and JSON; it was never protection.) It is turned on with --encryption-provider-config, pointing at an EncryptionConfiguration file. TLS, wherever it is configured, protects data in transit; encryption at rest is the control over what is stored on disk.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets", "configmaps"]
    providers:
      - aesgcm:
          keys:
            - name: key-2026-11
              secret: <base64-encoded 32-byte key>
      - identity: {}

How the provider list works:

  • The first provider encrypts every new write.
  • On reads, the API server tries the providers in order until one can decrypt the data. identity means "no encryption", so keeping it last lets the server still read data stored before encryption was enabled.
  • Existing data is not changed until it is written again. After enabling encryption (and restarting every API server), rewrite every Secret through the API, for example kubectl get secrets --all-namespaces -o json | kubectl replace -f -. Only then is it safe to remove identity; removing it first makes the old plain-text Secrets unreadable.
  • Putting identity first turns encryption off for new writes.
ProviderWhere the key livesNotes
identityNo keyNo encryption
aescbc, aesgcm, secretboxIn the EncryptionConfiguration file on each control plane nodeProtects against copies of etcd data alone, but anyone with the file can decrypt; aescbc is not recommended, aesgcm needs frequent key rotation
kms v2A key encryption key that stays in an external KMS; data keys are wrapped by itThe recommended choice; stable since Kubernetes 1.29. KMS v1 is deprecated.

Backups

An etcd snapshot (etcdctl snapshot save) is a full copy of cluster state, so it contains every Secret. Without encryption at rest, a snapshot on a shared file server is a plain-text dump of all of them. The guidance is to encrypt backups with a well-reviewed solution, restrict who can read them, and use encryption at rest so Secrets inside snapshots stay encrypted.

Keep the decryption key away from the data. Encryption at rest protects a backup only if the key is not in the same place as the ciphertext. With a local provider (aescbc, aesgcm, secretbox) the key is a line in the EncryptionConfiguration file, so any backup set that captures the control plane's configuration directory captures the key as well. With KMS v2 the key encryption key never leaves the external KMS: a stolen snapshot plus stolen configuration still cannot be decrypted without separate, audited access to the KMS. A restore plan must still be able to reach whatever holds the key, so a key that exists in only one place is its own availability risk.

Tip. KCSA questions on this topic are mostly short definitions and scenarios about which component, flag or setting does what. Expect to order the API server's request stages and work out which stage rejected a request, to name the identity and group of an anonymous caller and trace an exposure back to a binding, and to explain the risks of AlwaysAllow and of removing the Node authorizer. NodeRestriction questions separate it from authorization and from other admission plugins, and test which label prefixes a kubelet may set. TLS questions cover mutual TLS to etcd and the kubelet serving-certificate verification flag, including why it needs CA-signed kubelet certificates first. Key questions distinguish the service account signing key, its public verification keys, the cluster CA key and encryption keys, and the safe order of a signing-key rotation. Controller manager and scheduler items cover per-controller credentials, bind address, profiling and what a stolen scheduler credential allows. etcd items cover why direct access equals cluster root, which hosts may reach its client port, how other tools should read state, how the EncryptionConfiguration provider order works, why existing Secrets must be rewritten, and how backups and their keys are protected. One item may ask you to select two answers.

Key takeaways
  • The API server checks every request in the order authentication, authorization, admission; only admission sees the object's content.
  • Requests without credentials become system:anonymous in system:unauthenticated, never system:authenticated, and get only what is bound to them.
  • AlwaysAllow makes every valid credential a cluster administrator; production runs Node,RBAC, and dropping Node breaks kubelets.
  • NodeRestriction limits a kubelet to its own Node and Pods, and labels under node-restriction.kubernetes.io/ cannot be set by kubelets.
  • Use mutual TLS to etcd, and set --kubelet-certificate-authority only after kubelets have CA-signed serving certificates.
  • The service account signing key mints tokens and the cluster CA key mints superuser certificates; lock both down and rotate the signing key with an overlap.
  • Give the controller manager per-controller service accounts, and bind it and the scheduler to 127.0.0.1 with profiling off.
  • A compromised scheduler decides where Pods run; it reaches cluster state only through the API server.
  • Write access to etcd is root on the cluster: allow its client port only from the API server, and give other tools API access instead.
  • Encrypt Secrets at rest (KMS v2 preferred), rewrite existing Secrets after enabling it, and keep backups encrypted and away from their keys.

Check yourself: 3 questions on this topic

Exam-style KCSA questions on Control plane security: API server, controller manager, scheduler and etcd, answered right here with an explanation after each. Want more? KCSA Kubernetes Cluster Component Security practice questions.

Frequently asked questions

In what order does the Kubernetes API server check a request?

The kube-apiserver authenticates the caller first, then authorizes the requested verb on the resource, then runs admission control (mutating, then validating) on the object itself. Only a request that passes all three is written to etcd. A failed authentication returns 401 Unauthorized; an authorization denial returns 403 Forbidden, and admission rejections are usually 403 as well, though an admission webhook can return its own status code.

What identity does Kubernetes give an anonymous request?

When anonymous access is enabled, the API server treats a request with no client certificate or token as the user system:anonymous in the group system:unauthenticated. It is then authorized like any other identity, so it can do only what RBAC bindings grant to those subjects. Anonymous callers are never members of system:authenticated.

Why should production clusters use --authorization-mode=Node,RBAC?

RBAC grants permissions from roles bound to users, groups and service accounts, and the Node authorizer gives each kubelet access only to its own Node and the Pods, Secrets and ConfigMaps bound to it. Kubernetes does not bind the system:nodes group to an RBAC role, so removing Node breaks kubelets, while AlwaysAllow would let any valid credential do anything.

What does the NodeRestriction admission plugin do?

NodeRestriction limits each kubelet to modifying its own Node object and the Pods bound to its node, and prevents kubelets from setting labels with the node-restriction.kubernetes.io/ prefix. That stops a compromised node from tainting or relabelling other nodes, or from labelling itself as trusted to attract sensitive workloads. It is enabled with --enable-admission-plugins and is on by default in kubeadm clusters.

What can someone do with the Kubernetes service account signing key?

The private key set by --service-account-signing-key-file lets its holder forge a valid token for any service account and act with that account's RBAC permissions. It must be readable only by the API server. The matching public keys in --service-account-key-file only verify tokens and can be shared with external systems.

How do you secure kube-scheduler and kube-controller-manager?

Run the controller manager with --use-service-account-credentials=true so each controller has its own least-privilege service account, protect the cluster CA key it uses to sign certificates, and for both components set --bind-address=127.0.0.1 and --profiling=false so their health, metrics and debug endpoints are not exposed on the network.

Why is access to etcd so dangerous in Kubernetes?

etcd stores every Kubernetes object, including Secrets and RBAC bindings, and a direct write to etcd skips authentication, authorization, admission control and audit logging. Kubernetes documentation therefore treats etcd write access as root on the cluster. Restrict the etcd client port to the API server hosts and require mutual TLS.

Does enabling encryption at rest encrypt existing Kubernetes Secrets?

No. After the EncryptionConfiguration is in place and every API server has restarted, only new writes are encrypted by the first provider in the list. Existing Secrets stay as they were until they are written again, so rewrite them all through the API (for example kubectl get secrets --all-namespaces -o json | kubectl replace -f -) before removing the identity provider.

Source

This lesson covers the "Kubernetes Cluster Component Security" domain of the official KCSA exam guide. Vendors revise their guides — check the source for the current version.

Sign up free to mark lessons complete, bookmark topics and track your exam readiness.

Spotted a mistake in this lesson?