KCSA Control Plane Security: API Server, Controller Manager, Scheduler and etcd
Kubernetes control plane security is the hardening of the four components that run the cluster, the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that only authenticated and authorized requests can change cluster state and no single stolen credential or file hands over the whole cluster. This KCSA lesson covers how the API server processes each request (authentication, authorization, admission), anonymous access and authorization modes, the NodeRestriction admission plugin, TLS between the API server, etcd and kubelets, the keys that sign service account tokens and certificates, securing the controller manager and scheduler endpoints, and isolating, encrypting and backing up etcd.
On this page10 sections
- What Kubernetes control plane security covers
- How the API server processes a request
- Anonymous requests and authorization modes
- Securing the API server endpoint: TLS and network reachability
- NodeRestriction: limiting what a kubelet can change
- TLS from the API server to etcd and to kubelets
- Service account signing keys and the cluster CA key
- Securing the controller manager and the scheduler
- Securing etcd: access and isolation
- Encrypting Secrets at rest and protecting etcd backups
- Explain the order of authentication, authorization and admission control in the kube-apiserver and which stage rejects which kind of request
- Describe how anonymous requests are identified, why AlwaysAllow is unsafe and why Node,RBAC is the production authorization pair
- Use the NodeRestriction admission plugin and the node-restriction.kubernetes.io/ label prefix to limit what a compromised kubelet can change
- Secure the connections from the API server to etcd and to kubelets with TLS that authenticates both sides
- Protect and rotate the service account signing key and protect the cluster CA key held by the controller manager
- Harden the controller manager and scheduler with per-controller credentials, loopback binding and profiling turned off
- Isolate etcd, encrypt Secrets at rest with an EncryptionConfiguration and protect etcd backups and their keys
What Kubernetes control plane security covers
Control plane security is the protection of the four components that run a Kubernetes cluster: the kube-apiserver, the kube-controller-manager, the kube-scheduler and etcd. Together they decide what the cluster is allowed to do, so an attacker who controls any one of them controls far more than a single workload.
| Component | What it does | What an attacker gains by compromising it | Main controls |
|---|---|---|---|
| kube-apiserver | The front door: every client, node and controller reads and writes cluster state through it | Whatever the stolen identity is authorized for; a misconfigured server can expose the whole cluster | Authentication, authorization (Node,RBAC), admission plugins, TLS-only access, network restrictions |
| kube-controller-manager | Runs the controllers that reconcile state, and signs approved certificate requests | Broad write access to cluster objects, and (through the cluster CA key it holds) the power to mint superuser certificates | Per-controller service accounts, protected key files, loopback-only endpoints, profiling off |
| kube-scheduler | Picks a node for each new Pod | Control over where Pods run, including steering sensitive Pods onto a node the attacker already owns | Least-privilege identity, loopback-only endpoint, profiling off |
| etcd | The key-value store holding every cluster object, including Secrets | Effectively root on the cluster: direct writes skip every API server check | Mutual TLS, network isolation to the API server only, encryption at rest, protected backups |
The rest of this lesson takes each component in turn. Node components (kubelet, container runtime, kube-proxy and Pods) are covered in the node security lesson, and container networking, client access and storage in the lesson after it. How users authenticate (client certificates, OIDC tokens), Secrets as objects, audit logging and Pod Security Admission are taught in the Kubernetes Security Fundamentals domain.
How the API server processes a request
Every request to the kube-apiserver passes through three stages in a fixed order: authentication, then authorization, then admission control, and only a request that passes all three is stored in etcd. The order follows from what each stage needs to know.
- Authentication answers who is calling. Authenticators (client certificates, bearer tokens such as service account tokens, OIDC tokens, authenticating proxies, webhooks) turn the request into a username and a set of groups. A request that fails here is rejected with
401 Unauthorized. - Authorization answers may this identity perform this verb on this resource. It sees the user, groups, verb, resource, namespace and name, but not the body of the object. A denial is
403 Forbidden. - Admission control sees the actual object being created or changed. Mutating admission runs first and can modify it (add defaults, inject settings); validating admission runs next and can reject it for its content. A rejection here is usually reported as
403 Forbiddentoo (built-in plugins such as PodSecurity do this), although an admission webhook can choose its own status code and message.
The split tells you where to look when a request fails. kubectl auth can-i asks only the authorizers (through a SelfSubjectAccessReview), so it reports whether an identity holds a verb on a resource; it says nothing about whether a particular object will be accepted. Any rule that depends on what an object contains, such as a Deployment whose Pod template sets hostNetwork: true in a namespace that enforces the baseline Pod Security Standard, can only be enforced by admission, because authorization never looks inside the spec. A 401 points at credentials, and a denial on the verb points at RBAC; a denial that depends on a field's value points at admission.
Securing the API server endpoint: TLS and network reachability
Clients reach the kube-apiserver only over HTTPS on its secure port (6443 by default), and every request on that port is authenticated and authorized; being on the same host or the cluster network earns no implicit trust. Older releases also had an insecure HTTP port, bound to localhost, that skipped authentication and authorization entirely. It has been removed, so a current API server has no unauthenticated listener.
TLS protects the connection and proves the server's identity; it does not decide who may connect. Reducing who can reach the endpoint at all is a network control, and it complements, rather than replaces, authentication and RBAC:
- Private endpoint. The strongest option: the API server is reachable only from inside the cluster's private network (VPC), so administrators need a VPN, a bastion or a connected network.
- Authorized source ranges on a public endpoint. The endpoint stays public, but the provider accepts connections only from listed CIDR ranges and drops the rest at the network edge. Managed services expose this as an "authorized networks" or "public access CIDRs" setting; it filters by source address only, so clients authenticate exactly as before.
- Firewalls on self-managed control plane hosts, allowing the API port only from the networks that need it.
Two controls are often confused with these. A Kubernetes NetworkPolicy governs traffic between Pods; it does not filter connections to the API server endpoint from outside the cluster. Disabling anonymous requests stops unauthenticated callers from being authorized, but the endpoint stays reachable by anyone.
NodeRestriction: limiting what a kubelet can change
The NodeRestriction admission plugin limits each kubelet to modifying its own Node object and the Pods bound to it, and stops kubelets from setting labels that administrators reserve for security decisions. It is enabled with --enable-admission-plugins=NodeRestriction and works together with the Node authorizer; kubeadm enables both by default.
The two layers do different jobs. Authorization works on verbs and resources, so the Node authorizer can say "kubelets may update Node objects", but not "only their own". Admission sees the object, so NodeRestriction narrows the update to the kubelet's own Node (identified by its system:node:<nodeName> username). Without it, a stolen kubelet credential could taint or relabel other nodes and push sensitive workloads onto the compromised one.
| Label on a Node | Can the kubelet set it on its own Node? |
|---|---|
kubernetes.io/hostname, kubernetes.io/arch, kubernetes.io/os | Yes |
node.kubernetes.io/instance-type, topology.kubernetes.io/region, topology.kubernetes.io/zone | Yes |
Any label with the kubelet.kubernetes.io/ or node.kubernetes.io/ prefix | Yes |
Any label with the node-restriction.kubernetes.io/ prefix | No: reserved for administrators |
That reserved prefix is the one to use when a label marks nodes as trusted for sensitive workloads (for example node-restriction.kubernetes.io/pci-zone=true on the nodes cleared for card-payment workloads, matched by those Pods' node affinity). A kubelet that is compromised cannot add it to itself, so it cannot attract those Pods. Labels with the allowed prefixes look official but give no such guarantee.
TLS from the API server to etcd and to kubelets
The API server is a client of etcd and of every kubelet, and both connections must be encrypted and must authenticate the other side: mutual TLS for etcd, and CA verification of kubelet serving certificates for kubelets.
API server to etcd
The Kubernetes documentation recommends mutual TLS. The API server presents a client certificate (--etcd-certfile and --etcd-keyfile) and verifies etcd's server certificate against --etcd-cafile; etcd verifies the client certificate against its trusted CA (--client-cert-auth=true with --trusted-ca-file). Weaker setups each drop a half: plain TCP trusted by subnet authenticates nobody, one-way TLS authenticates only etcd, and accepting any client certificate authenticates no client. Peer traffic between etcd members (port 2380) should use TLS as well.
API server to kubelets
The API server connects to the kubelet API to serve kubectl exec, kubectl logs and port-forwarding. It authenticates to the kubelet with a client certificate (--kubelet-client-certificate and --kubelet-client-key). The reverse check is the one usually missing: unless --kubelet-certificate-authority is set, the API server does not verify the kubelet's serving certificate, so an attacker on the node network could impersonate a kubelet and capture exec sessions or logs.
The flag can only verify certificates its CA actually signed, and a kubelet's default serving certificate is self-signed. Kubelets therefore need CA-signed serving certificates before the API server can check them:
- Configure kubelets with
serverTLSBootstrap: trueso they request serving certificates through CertificateSigningRequests (signerkubernetes.io/kubelet-serving). - Approve those CSRs (they are not approved automatically) so the controller manager signs them with the cluster CA.
- Set
--kubelet-certificate-authorityon the API server to that CA.
Keep the two directions apart: the API server's client certificate and the kubelet's own authentication settings protect the kubelet from its callers, while --kubelet-certificate-authority protects the API server from a fake kubelet. Kubelet-side settings (its own authentication and authorization, ports) belong to the node security lesson.
Service account signing keys and the cluster CA key
Two private keys on the control plane let their holder create trusted identities: the service account signing key mints tokens for any service account, and the cluster CA key signs client certificates for any user or group. Both must be readable only by the component that uses them.
| Key | Where it is configured | What a thief can do |
|---|---|---|
| Service account signing key (private) | --service-account-signing-key-file on kube-apiserver | Forge a token for any service account and act with its RBAC permissions |
| Service account verification keys (public) | --service-account-key-file on kube-apiserver (repeatable) | Nothing on its own: public keys only verify tokens, never mint them, and can be published (for example through the OIDC discovery endpoint) so external systems can validate tokens |
| Cluster CA key | --cluster-signing-key-file (with --cluster-signing-cert-file) on kube-controller-manager | Sign a client certificate for any user or group, including O=system:masters, a superuser that RBAC cannot restrict |
| Encryption-at-rest keys | The EncryptionConfiguration file, or an external KMS | Decrypt Secret data copied from etcd or its backups |
Service account tokens are signed JWTs, and the API server accepts any token whose signature verifies against a trusted public key. --service-account-key-file normally holds public keys; it also accepts a private key file, from which only the public half is used. The API server's own HTTPS certificate and key (--tls-cert-file, --tls-private-key-file) are a separate pair used only to serve TLS. The controller manager's --service-account-private-key-file signs the older Secret-based service account tokens.
Rotating the signing key without breaking workloads
Running Pods hold tokens signed with the current key, so rotation must overlap:
- Add the new public key to the verification keys on every API server, alongside the old one.
- Switch
--service-account-signing-key-fileto the new private key; new tokens use it, and old tokens still verify. - When every token signed with the old key has expired (projected tokens are short-lived and refreshed by the kubelet), remove the old public key so it can no longer be abused. Legacy Secret-based service account tokens never expire, so any that were signed with the old key must be deleted or reissued first, or they stop working when the key goes.
The principle is that every API server must trust a key before anything is signed with it, and must keep trusting it until nothing signed with it is still in use.
Who signs approved certificate requests
Approval and signing are separate steps. The API server stores a CertificateSigningRequest and authorizes whoever approves it; the kube-controller-manager's csrsigning controllers then sign it with the cluster CA key (kubeadm points --cluster-signing-* at the cluster CA). That is why the controller manager's host and key files need the same protection as the CA itself.
Securing the controller manager and the scheduler
Secure the kube-controller-manager and kube-scheduler by giving them least-privilege identities and by removing network exposure of their own HTTPS endpoints: bind them to the loopback address and turn profiling off.
Per-controller credentials
By default the controller manager can run all of its controllers with its own single, powerful credential, so a flaw in one controller carries the permissions of all of them. --use-service-account-credentials=true makes each controller use its own service account in kube-system, bound to a matching system:controller:* role with only the permissions that controller needs. kubeadm sets it.
Their own endpoints
Both components serve health checks, metrics and debugging over HTTPS: the controller manager on port 10257 and the scheduler on port 10259. Two defaults matter for security:
--bind-addressdefaults to all interfaces. Setting--bind-address=127.0.0.1keeps the endpoints usable for local health probes on the control plane host while removing them from the network.--profilingdefaults to true, which exposes/debug/pprofdata about the process. The CIS Kubernetes Benchmark recommends--profiling=false.
What a compromised scheduler can do
The scheduler watches the API server for Pods with no nodeName and posts a Binding for each one through the API server, which authenticates and authorizes it as system:kube-scheduler before storing it. With the scheduler's credentials an attacker can bind new Pods to nodes of their choice, such as a node already compromised, pulling sensitive workloads and their mounted Secrets onto it, or crowd workloads onto chosen nodes. The scheduler does not talk to etcd or kubelets directly, does not hold the cluster CA key and cannot run commands in containers; those powers belong to other components.
Securing etcd: access and isolation
etcd stores every cluster object, so the Kubernetes documentation treats write access to etcd as equivalent to root on the entire cluster, and read access as enough to escalate quickly; only the API server should be able to reach it. Pods, Deployments, RBAC bindings and Secrets all live in etcd. A client that writes there directly bypasses authentication, authorization, admission control and audit logging: it could add a cluster-admin binding that no RBAC check or admission policy ever sees.
Controls that follow the guidance:
- Network isolation. Allow the etcd client port (2379 by default) only from the kube-apiserver hosts, and the peer port (2380) only between etcd members. Worker nodes, the Pod network and administrators' workstations have no business connecting, so a compromised node or Pod cannot even try. Running etcd on dedicated hosts behind a firewall is common for self-managed clusters.
- Mutual TLS with client certificates issued to the API server, as described above.
- No extra etcd clients. Granting another component read or write access to the full keyspace is equivalent to granting it cluster-admin. If something other than the API server truly needs etcd, the guidance is a separate etcd instance or etcd ACLs limited to a subset of keys.
A tool that wants to read cluster state, such as a monitoring system, should read through the API server with a ServiceAccount bound to a narrow read-only role. That keeps authorization and auditing in the path. Any route that hands it etcd data, whatever the certificate, host or file format, gives it all of cluster state, including every Secret, with no authorization or audit in the path.
Encrypting Secrets at rest and protecting etcd backups
Encryption at rest makes the API server encrypt chosen resources, usually Secrets, before writing them to etcd; without it, Secret values sit unencrypted in etcd and in every backup, so anyone with a copy can read them. (Base64 is only how Secret data is written in the API's YAML and JSON; it was never protection.) It is turned on with --encryption-provider-config, pointing at an EncryptionConfiguration file. TLS, wherever it is configured, protects data in transit; encryption at rest is the control over what is stored on disk.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets", "configmaps"]
providers:
- aesgcm:
keys:
- name: key-2026-11
secret: <base64-encoded 32-byte key>
- identity: {}
How the provider list works:
- The first provider encrypts every new write.
- On reads, the API server tries the providers in order until one can decrypt the data.
identitymeans "no encryption", so keeping it last lets the server still read data stored before encryption was enabled. - Existing data is not changed until it is written again. After enabling encryption (and restarting every API server), rewrite every Secret through the API, for example
kubectl get secrets --all-namespaces -o json | kubectl replace -f -. Only then is it safe to removeidentity; removing it first makes the old plain-text Secrets unreadable. - Putting
identityfirst turns encryption off for new writes.
| Provider | Where the key lives | Notes |
|---|---|---|
identity | No key | No encryption |
aescbc, aesgcm, secretbox | In the EncryptionConfiguration file on each control plane node | Protects against copies of etcd data alone, but anyone with the file can decrypt; aescbc is not recommended, aesgcm needs frequent key rotation |
kms v2 | A key encryption key that stays in an external KMS; data keys are wrapped by it | The recommended choice; stable since Kubernetes 1.29. KMS v1 is deprecated. |
Backups
An etcd snapshot (etcdctl snapshot save) is a full copy of cluster state, so it contains every Secret. Without encryption at rest, a snapshot on a shared file server is a plain-text dump of all of them. The guidance is to encrypt backups with a well-reviewed solution, restrict who can read them, and use encryption at rest so Secrets inside snapshots stay encrypted.
Keep the decryption key away from the data. Encryption at rest protects a backup only if the key is not in the same place as the ciphertext. With a local provider (aescbc, aesgcm, secretbox) the key is a line in the EncryptionConfiguration file, so any backup set that captures the control plane's configuration directory captures the key as well. With KMS v2 the key encryption key never leaves the external KMS: a stolen snapshot plus stolen configuration still cannot be decrypted without separate, audited access to the KMS. A restore plan must still be able to reach whatever holds the key, so a key that exists in only one place is its own availability risk.
Tip. KCSA questions on this topic are mostly short definitions and scenarios about which component, flag or setting does what. Expect to order the API server's request stages and work out which stage rejected a request, to name the identity and group of an anonymous caller and trace an exposure back to a binding, and to explain the risks of AlwaysAllow and of removing the Node authorizer. NodeRestriction questions separate it from authorization and from other admission plugins, and test which label prefixes a kubelet may set. TLS questions cover mutual TLS to etcd and the kubelet serving-certificate verification flag, including why it needs CA-signed kubelet certificates first. Key questions distinguish the service account signing key, its public verification keys, the cluster CA key and encryption keys, and the safe order of a signing-key rotation. Controller manager and scheduler items cover per-controller credentials, bind address, profiling and what a stolen scheduler credential allows. etcd items cover why direct access equals cluster root, which hosts may reach its client port, how other tools should read state, how the EncryptionConfiguration provider order works, why existing Secrets must be rewritten, and how backups and their keys are protected. One item may ask you to select two answers.
- The API server checks every request in the order authentication, authorization, admission; only admission sees the object's content.
- Requests without credentials become system:anonymous in system:unauthenticated, never system:authenticated, and get only what is bound to them.
- AlwaysAllow makes every valid credential a cluster administrator; production runs Node,RBAC, and dropping Node breaks kubelets.
- NodeRestriction limits a kubelet to its own Node and Pods, and labels under node-restriction.kubernetes.io/ cannot be set by kubelets.
- Use mutual TLS to etcd, and set --kubelet-certificate-authority only after kubelets have CA-signed serving certificates.
- The service account signing key mints tokens and the cluster CA key mints superuser certificates; lock both down and rotate the signing key with an overlap.
- Give the controller manager per-controller service accounts, and bind it and the scheduler to 127.0.0.1 with profiling off.
- A compromised scheduler decides where Pods run; it reaches cluster state only through the API server.
- Write access to etcd is root on the cluster: allow its client port only from the API server, and give other tools API access instead.
- Encrypt Secrets at rest (KMS v2 preferred), rewrite existing Secrets after enabling it, and keep backups encrypted and away from their keys.
Check yourself: 3 questions on this topic
Exam-style KCSA questions on Control plane security: API server, controller manager, scheduler and etcd, answered right here with an explanation after each. Want more? KCSA Kubernetes Cluster Component Security practice questions.
Frequently asked questions
In what order does the Kubernetes API server check a request?
The kube-apiserver authenticates the caller first, then authorizes the requested verb on the resource, then runs admission control (mutating, then validating) on the object itself. Only a request that passes all three is written to etcd. A failed authentication returns 401 Unauthorized; an authorization denial returns 403 Forbidden, and admission rejections are usually 403 as well, though an admission webhook can return its own status code.
What identity does Kubernetes give an anonymous request?
When anonymous access is enabled, the API server treats a request with no client certificate or token as the user system:anonymous in the group system:unauthenticated. It is then authorized like any other identity, so it can do only what RBAC bindings grant to those subjects. Anonymous callers are never members of system:authenticated.
Why should production clusters use --authorization-mode=Node,RBAC?
RBAC grants permissions from roles bound to users, groups and service accounts, and the Node authorizer gives each kubelet access only to its own Node and the Pods, Secrets and ConfigMaps bound to it. Kubernetes does not bind the system:nodes group to an RBAC role, so removing Node breaks kubelets, while AlwaysAllow would let any valid credential do anything.
What does the NodeRestriction admission plugin do?
NodeRestriction limits each kubelet to modifying its own Node object and the Pods bound to its node, and prevents kubelets from setting labels with the node-restriction.kubernetes.io/ prefix. That stops a compromised node from tainting or relabelling other nodes, or from labelling itself as trusted to attract sensitive workloads. It is enabled with --enable-admission-plugins and is on by default in kubeadm clusters.
What can someone do with the Kubernetes service account signing key?
The private key set by --service-account-signing-key-file lets its holder forge a valid token for any service account and act with that account's RBAC permissions. It must be readable only by the API server. The matching public keys in --service-account-key-file only verify tokens and can be shared with external systems.
How do you secure kube-scheduler and kube-controller-manager?
Run the controller manager with --use-service-account-credentials=true so each controller has its own least-privilege service account, protect the cluster CA key it uses to sign certificates, and for both components set --bind-address=127.0.0.1 and --profiling=false so their health, metrics and debug endpoints are not exposed on the network.
Why is access to etcd so dangerous in Kubernetes?
etcd stores every Kubernetes object, including Secrets and RBAC bindings, and a direct write to etcd skips authentication, authorization, admission control and audit logging. Kubernetes documentation therefore treats etcd write access as root on the cluster. Restrict the etcd client port to the API server hosts and require mutual TLS.
Does enabling encryption at rest encrypt existing Kubernetes Secrets?
No. After the EncryptionConfiguration is in place and every API server has restarted, only new writes are encrypted by the first provider in the list. Existing Secrets stay as they were until they are written again, so rewrite them all through the API (for example kubectl get secrets --all-namespaces -o json | kubectl replace -f -) before removing the identity provider.
Source
This lesson covers the "Kubernetes Cluster Component Security" domain of the official KCSA exam guide. Vendors revise their guides — check the source for the current version.
- KCSA certification page and curriculum — The Linux Foundation
Sign up free to mark lessons complete, bookmark topics and track your exam readiness.