SaveMyCert
Log in
5 of 5 free questions left today·for 10 a day
Kubernetes Security Fundamentals

KCSA Authentication, Secrets and Audit Logging

19 min readKCSA · Kubernetes Security FundamentalsUpdated

Kubernetes authentication is how the API server decides who sent each request, using client certificates, tokens or an OpenID Connect identity provider, before RBAC decides what that identity may do; Secrets hold the cluster's sensitive values, and audit logging records who did what through the API. This lesson covers normal users and service accounts, how certificate subjects, service account names and OIDC claims become users and groups, bound and static tokens, Secret types and the hidden paths to reading them, consuming Secrets safely, encryption at rest and key rotation, and audit stages, levels, policies and backends.

What you’ll learn
  • Explain why Kubernetes has no User objects and how certificates, tokens and OIDC ID tokens supply identity, and tell a 401 from a 403
  • Map a client certificate subject, a service account or an OIDC claim to the user name and groups that RBAC sees, including system:masters and system:authenticated
  • Compare bound TokenRequest tokens, legacy token Secrets and static token files by lifetime, audience and revocation
  • Identify every path to a Secret's value (get, list, watch, Pod creation, cluster-wide grants) and write the narrowest grant for one Secret
  • Choose between Secret volumes, subPath mounts and environment variables, and pick the right Secret type
  • Configure EncryptionConfiguration provider order and rotate a key safely across several API servers
  • Write an audit policy with the right levels, rule order and omitStages, and choose a log or webhook backend mode

How Kubernetes authenticates users and service accounts

Kubernetes authenticates every API request by asking its configured authenticators who sent it, and it knows two kinds of identity: normal users, which are managed outside the cluster, and service accounts, which are real API objects. There is no User or Group resource in the Kubernetes API, so there is no command that creates a user. A person becomes known to the cluster by presenting a credential that the API server trusts: a client certificate signed by the cluster CA, a bearer token, or an ID token from an OpenID Connect identity provider.

Normal usersService accounts
Where they liveOutside the cluster: a certificate authority, a token file, an identity providerIn the API, as namespaced ServiceAccount objects
How they are createdBy issuing a credential (signing a certificate, adding a user in the IdP)By the API (kubectl create serviceaccount); every namespace gets a default one
Typical credentialClient certificate or OIDC ID tokenA signed service account token (a JWT)
Intended forHumans and external systemsProcesses running in Pods, and automation

The API server can run several authenticators at once (client certificates, bearer tokens, OIDC, an authenticating proxy, a webhook token authenticator). The first one to succeed decides the identity and the rest are skipped; the API server does not guarantee the order in which they run. The result is a user name, an optional UID, a list of groups and optional extra fields. That identity is all later stages see.

A request with no credential, when anonymous access is enabled, becomes the user system:anonymous in the group system:unauthenticated. Anonymous access and the API server's authorization mode settings are covered in the control plane security lesson for Domain 2.

Authentication versus authorization: 401, 403 and system:authenticated

Authentication only establishes who is calling; authorization, usually RBAC, then decides whether that identity may perform the request. The two failures look different. If no authenticator accepts the credential (a bad or expired token, an untrusted certificate), the API server answers 401 Unauthorized: it does not know who you are. If the identity is accepted but no authorization rule allows the verb on the resource, it answers 403 Forbidden, and the message names the user that was recognised. Admission control runs only after both stages succeed, and only for requests that create, change or delete objects, so it never explains a 403 on a read such as listing Pods.

Every successfully authenticated identity, human or service account, is also placed in the group system:authenticated. Binding a role to that group therefore grants it to everyone who can log in, including every service account in every namespace. It does not reach anonymous requests, which are in system:unauthenticated instead. Granting a team access through system:authenticated is almost always far wider than intended; bind the team's own group.

RBAC is purely additive: Roles and ClusterRoles only list what is allowed, and there are no deny rules. You remove access by removing or narrowing the bindings that grant it. That matters for the special case in the next section, where no binding is involved at all.

X.509 client certificates: CN, O and system:masters

When a client authenticates with a certificate signed by the cluster CA, the API server takes the user name from the certificate subject's Common Name (CN) and the groups from its Organization (O) fields. A certificate with the subject /CN=amara/O=platform-ops authenticates as the user amara in the group platform-ops. A certificate can carry several O values, giving several groups. Only these two subject fields carry identity; the rest of the subject plays no part in naming the user or its groups.

This mapping is how you grant access to a team with certificates. Put each person's own name in the CN, so audit logs show exactly who made each request, and put a shared group such as web-team in the O field. One RoleBinding to that Group then covers everyone who has, or later receives, a certificate with that O value, and nobody has to edit the binding when people join. Putting the shared name in the CN instead would make everyone the same user and destroy individual accountability.

The group system:masters is the dangerous exception. Members of it are cluster superusers: the API server lets their requests through without consulting RBAC, so no RoleBinding or ClusterRoleBinding is needed, and no change to RBAC can take that access away. Kubernetes also has no way to revoke an individual client certificate; it stays valid until it expires. A leaked or forgotten certificate with O=system:masters is therefore a live master key for its whole lifetime, and the only complete fixes are rotating the cluster CA or waiting for expiry. Issue such certificates only for break-glass use, keep them short-lived, and give people ordinary groups bound to scoped roles instead. (What to do after a client certificate leaks is covered in the Domain 2 lesson on client security.)

Service account identities, bound tokens and static tokens

A service account authenticates as the user system:serviceaccount:<namespace>:<name> and belongs to the groups system:serviceaccounts and system:serviceaccounts:<namespace>. A Pod in the shipping namespace that runs as label-printer therefore appears in audit logs as system:serviceaccount:shipping:label-printer. Note the singular serviceaccount in the user name and the plural serviceaccounts in the group names.

Subject in a bindingWho it covers
kind: ServiceAccount, name and namespaceThat one service account
kind: Group, system:serviceaccounts:reportsEvery service account in the reports namespace, including ones created later
kind: Group, system:serviceaccountsEvery service account in every namespace
kind: Group, system:authenticatedEvery authenticated user and service account

The namespace in a group name selects which accounts are covered. Where the permissions apply is a separate question, decided by whether the grant is a RoleBinding (one namespace) or a ClusterRoleBinding (the whole cluster).

Current clusters issue service account tokens through the TokenRequest API. These tokens are short-lived, carry an audience, and when issued for a Pod are bound to that Pod: the kubelet requests them, mounts them as a projected volume and refreshes them before they expire, and they stop being valid when the Pod is deleted. For automation outside a Pod, kubectl create token <serviceaccount> calls the same API, and --duration sets a lifetime (the server may cap it) while --audience sets who the token is for.

Since Kubernetes 1.24 the cluster no longer creates a long-lived token Secret for each service account automatically, and the ServiceAccount's secrets field no longer lists one. You can still create a Secret of type kubernetes.io/service-account-token with the annotation kubernetes.io/service-account.name, and the control plane fills in a token, but that token never expires and is only invalidated when the Secret or the account is deleted. Use it only when nothing else works.

Token audiences

A bound token's audience says which service it is for, and the receiving service is expected to reject tokens whose audience is not one it accepts. The kube-apiserver accepts only tokens whose audience matches its own configured API audiences (normally the cluster's issuer). A Pod that mounts a projected token with audience: metrics-gateway to authenticate to an external metrics service holds a token that the API server will refuse, even though the cluster signed it. The audience check happens at the receiver, so a stolen copy is only useful against the service it names. This is why issuing a separate, narrowly scoped audience per external service limits the blast radius of a leak.

The static token file

The static token file is the oldest bearer-token method. The API server reads a CSV file passed with --token-auth-file, whose rows hold a token, a user name, a UID and optional groups. It is discouraged because the tokens last indefinitely, and the file is read only at start-up, so adding, changing or revoking a token needs an API server restart. Users from the file are ordinary identities: they still go through RBAC like everyone else, so the risk is long-lived, unrotatable credentials, not a bypass of authorization.

OpenID Connect: how ID tokens are validated

With OpenID Connect (OIDC), a person logs in to the company identity provider, and kubectl sends the resulting ID token, a JWT signed by the provider, as a bearer token on every request. The API server validates that token itself: it checks the signature against the provider's published keys, checks the issuer, the audience (the client ID) and the expiry, and then reads the user name and groups from configured claims. It does not call the provider on each request, and it never sees the user's password. A refresh token, if any, is used by the client (kubectl or a credential plugin) to obtain a new ID token from the provider; the API server never handles it.

Two consequences follow directly:

  • Disabling an account does not cut off tokens already issued. The API server keeps accepting a valid ID token until it expires, because it checks only the signature and the expiry. Blocking the account stops the next refresh. Short ID-token lifetimes are what keep this window small.
  • Prefixes change the names RBAC sees. The API server can add a prefix to the user name and to each group from the token, for example oidc:, so IdP names cannot collide with built-in names such as system:masters. An IdP group oncall then arrives as oidc:oncall; RBAC compares names exactly, so a binding has to name the prefixed group, and a binding to the unprefixed name grants nothing (the user gets 403 Forbidden).

OIDC is configured with API server flags (issuer URL, client ID, username and groups claims, and their prefixes) or, on current versions, a structured authentication configuration file that can trust several issuers. Once the prefix is applied, a group from an ID token is an ordinary RBAC group like any other.

Kubernetes Secrets: encoding, types and who can read them

A Secret is an API object for small pieces of sensitive data, such as passwords, tokens and keys, and its values are only base64-encoded, not encrypted. Base64 is a reversible encoding: anyone who holds the manifest or the API response can decode a value in one step. A Secret manifest in Git is therefore a leaked secret. Keep the real values in a secrets manager or encrypt them before they reach Git. The stringData field is a convenience for writing plain-text values that the API server stores base64-encoded in data; it adds no protection.

The type field tells Kubernetes and tools what the Secret holds and which keys must be present:

TypeUsed forRequired keys
Opaque (default)Arbitrary user dataNone
kubernetes.io/tlsA certificate and private key, for example for an Ingresstls.crt and tls.key
kubernetes.io/dockerconfigjsonRegistry credentials for pulling images.dockerconfigjson
kubernetes.io/basic-authBasic authentication credentialsusername and/or password
kubernetes.io/ssh-authAn SSH private keyssh-privatekey
kubernetes.io/service-account-tokenA legacy long-lived service account tokenAnnotation kubernetes.io/service-account.name; the token is filled in

Who can read a Secret is wider than who has get on it. The Kubernetes good-practice guidance names three paths:

  • list and watch return full objects, values included. There is no RBAC verb that reveals Secret names without their values, so any read access beyond get on named Secrets exposes every value in scope.
  • Creating Pods in a namespace implies reading its Secrets. Anyone who can create a Pod (or a Deployment, Job or other workload) can mount any Secret in that namespace and send its contents out, without any RBAC rule on Secrets. The kubelet fetches the Secret on the Pod's behalf, so the workload, not the user, is what reads it.
  • Cluster-wide grants reach every namespace. A ClusterRole bound with a ClusterRoleBinding applies everywhere, so even resourceNames: ["smtp-password"] matches a Secret of that name in every namespace.

The tightest grant for a controller that reads one Secret is a namespaced Role allowing get on secrets with resourceNames set to that Secret, bound by a RoleBinding in the same namespace. Keep workload-creation rights in that namespace equally narrow, because they are a second route to the same value. Other good practices: keep secret-handling workloads in their own namespaces, mark Secrets immutable when they should not change, and consider an external secret store with the Secrets Store CSI driver or a similar operator.

Consuming Secrets: volumes versus environment variables

A Pod can consume a Secret as files in a mounted volume or as environment variables, and only the volume form picks up changes while the Pod runs.

MethodHow it appears in the containerUpdates after the Secret changes
Secret volume (whole volume)Files in a tmpfs volume on the nodeYes, after the kubelet's next sync, without a restart
Secret volume with subPathOne key mounted at a chosen pathNo; the file stays as it was at container start
env with secretKeyRef, or envFromCopied into the process environment at container startNo; the container must restart

Volumes are also the safer default for other reasons: environment variables are easily exposed through logs, crash dumps, debugging tools and child processes, while a file can be given tight permissions. The kubelet only sends a Secret to nodes running a Pod that uses it and keeps volume contents in memory (tmpfs) rather than on disk. The application still has to re-read the file to notice a rotated value.

Encryption at rest: EncryptionConfiguration and key rotation

By default the API server stores Secrets in etcd unencrypted, so anyone with access to etcd or its backups can read them; encryption at rest is turned on with an EncryptionConfiguration file passed to every kube-apiserver with --encryption-provider-config. The file lists, per resource, an ordered list of providers. The rule that decides everything is order:

  • The first provider in the list encrypts all new writes.
  • For reads, the providers are tried in order until one can decrypt the stored data.
  • identity means no encryption. If it is listed first, new Secrets are written in plain text no matter what follows it. It belongs last, where it only lets the server read data written before encryption was enabled.

The providers include aescbc, aesgcm and secretbox, which keep the key in the configuration file on each control plane node, and kms (version 2), which uses an external key management service through envelope encryption and is the recommended choice because the key never sits in the file. The provider's key is kept in that file or in the KMS, never in a Kubernetes Secret. Turning encryption on only affects writes from then on; rewriting existing Secrets so they are encrypted is covered in the Domain 2 lesson on control plane and etcd security.

Rotating a key on several API servers needs two rounds of restarts. Every server must be able to read the new key before any server starts writing with it, or a server that does not know the key yet fails to read Secrets the others have written. The safe order is:

  1. Add the new key as the second key of the provider on every API server and restart them all. Each server can now decrypt with either key but still writes with the old one.
  2. Make the new key the first key on every server and restart them all. New writes now use it.
  3. Rewrite every Secret (for example kubectl get secrets --all-namespaces -o json | kubectl replace -f -) so all stored data uses the new key.
  4. Remove the old key from every server and restart.

Audit logging: stages, levels and policy rules

Kubernetes auditing records a chronological set of events for requests to the API server, showing who did what, to which resource, when and with what result, and it only records anything once an audit policy is supplied. Starting kube-apiserver with --audit-log-path but no --audit-policy-file produces no events, because there is no default policy.

Each request can generate an event at up to four stages:

StageWhen it is emitted
RequestReceivedAs soon as the request arrives, before it is processed
ResponseStartedWhen the response headers are sent but the body is not; only for long-running requests such as watch
ResponseCompleteWhen the response body is complete; carries the response status code
PanicWhen a panic occurred while handling the request

Each policy rule sets a level, which controls how much of the request is recorded:

LevelWhat is recorded
NoneNothing for matching requests
MetadataUser, groups, verb, resource, namespace, source IPs, user agent, timestamps and response status; no bodies
RequestMetadata plus the request body, no response body
RequestResponseMetadata plus request and response bodies

Body levels apply only to resource requests; non-resource requests such as /healthz are logged at metadata detail at most. omitStages is a separate dial: it drops whole stages, which changes how many events each request produces, but not what each event contains. Each stage is emitted at a different point in the request, so what an event can contain depends on its stage: only events from ResponseComplete onwards know how the request ended, because the status code does not exist when the request arrives. Level and stages are independent: the level decides how much detail goes into every event that is emitted, and omitStages decides which events are emitted at all.

Rules are evaluated in order, and the first rule that matches a request sets its level. A rule matches only if all of its conditions (users, groups, verbs, resources, namespaces) match. Suppose a policy's first rule logs every request from the group ci-bots at Request, its second logs ConfigMaps at Metadata and its third logs Deployments at RequestResponse. An update to a ConfigMap by a member of ci-bots matches the first rule, so its request body, the new ConfigMap data, is logged even though the second rule was written for exactly that resource. Order, not specificity, decides: a narrow rule placed after a broad one never sees the requests the broad one already claimed.

That is why the upstream example policy, and good practice, puts rules for secrets, configmaps and tokenreviews early, at the Metadata level. Metadata keeps the record of who read or changed each Secret, while Request or RequestResponse would copy Secret values (and tokens) into a log that is usually far more widely readable than the Secrets themselves. The aim is to record the access while leaving the value out, and Metadata is the level that does both.

Audit backends: log files and webhooks

The API server writes audit events to a log backend, a webhook backend, or both.

  • Log backend: writes JSON lines to the file named by --audit-log-path (- means standard output), with rotation controlled by --audit-log-maxage, --audit-log-maxbackup and --audit-log-maxsize. On a kubeadm-style control plane, the policy file and log directory must be mounted into the API server's static Pod.
  • Webhook backend: sends events to an external HTTP API described by a kubeconfig-format file given with --audit-webhook-config-file. Shipping events off the node keeps them out of reach of someone who compromises that node.

Both backends support the same three modes, set with --audit-log-mode for the log backend (default blocking) and --audit-webhook-mode for the webhook (default batch):

ModeBehaviour
batchBuffers events and sends them asynchronously in batches; requests are not delayed, but events can be dropped if the buffer fills
blockingBlocks the API server's response while each event is processed; a failure is logged but the request continues
blocking-strictLike blocking, but a failure to record the event at the RequestReceived stage fails the whole API request

Only blocking-strict refuses a request whose RequestReceived event cannot be recorded, at the cost of availability: if the backend cannot record, the API stops serving. It does not make the log complete in every case: a failure at a later stage does not fail the request, and requests that a rule sets to None or whose RequestReceived stage is omitted produce no such event in the first place. In the other two modes a recording failure never stops a request; their tuning options only change how often events are lost.

Tip. KCSA questions on this topic are short definitions and practical scenarios. Expect to name the user or group a certificate, service account or OIDC token produces, read a 401 or 403 correctly, and judge what a binding to a built-in group really covers, including system:masters. Token questions compare lifetimes, audiences and how each kind of token is revoked. Secret questions cover base64 versus encryption, Secret types and their required keys, the indirect ways to read a Secret, the narrowest grant for one Secret, and which consumption method picks up a rotated value. Others give an EncryptionConfiguration or a key rotation plan and ask what gets written, or give an audit policy and ask what a request logs, which level, stage or omitStages change fits a requirement, and which backend mode fits a strict recording rule. Distractors are usually real settings that belong to a different stage, level, field or mechanism.

Key takeaways
  • Kubernetes has no User objects: people are identified by client certificates, tokens or an OIDC provider, while service accounts are API objects.
  • 401 means the credential was not accepted; 403 means the identity was accepted but not authorized. Every authenticated identity is in system:authenticated.
  • A client certificate's CN is the user name and each O value is a group; system:masters bypasses RBAC and a certificate cannot be revoked, only expire.
  • Service accounts authenticate as system:serviceaccount:<ns>:<name> and are in system:serviceaccounts and system:serviceaccounts:<ns>.
  • Prefer TokenRequest tokens (kubectl create token, projected volumes): time-limited, audience-bound and tied to the Pod. Static token files and legacy token Secrets never expire.
  • The API server validates OIDC ID tokens itself, so a disabled user's existing token works until it expires, and claim prefixes such as oidc: change the group names RBAC sees.
  • Secret values are base64-encoded, not encrypted; list, watch and the right to create Pods all expose them. Scope access with a namespaced Role and resourceNames.
  • Secret volumes update in running Pods; subPath mounts and environment variables do not.
  • In an EncryptionConfiguration the first provider writes and identity means plain text; rotate keys by adding the new key second everywhere, then first everywhere.
  • Audit logging needs a policy file; the first matching rule sets the level, so log Secrets at Metadata early, and only blocking-strict fails a request whose RequestReceived event cannot be recorded.

Check yourself: 3 questions on this topic

Exam-style KCSA questions on Authentication, Secrets and audit logging, answered right here with an explanation after each. Want more? KCSA Kubernetes Security Fundamentals practice questions.

Frequently asked questions

How do you create a user in Kubernetes?

You cannot create a user object in Kubernetes, because normal users are not API resources. A person becomes a user by presenting a credential the API server trusts: a client certificate signed by the cluster CA (the CN becomes the user name and the O fields become groups), a bearer token, or an OpenID Connect ID token from the company identity provider. Service accounts are different: they are real namespaced objects created through the API for workloads.

What is the difference between 401 Unauthorized and 403 Forbidden in Kubernetes?

A 401 Unauthorized response means authentication failed: the Kubernetes API server could not establish who sent the request, for example because a token is invalid or a certificate is not trusted. A 403 Forbidden response means authentication succeeded and the user is known, but authorization (normally RBAC) has no rule allowing that verb on that resource. The 403 message usually names the recognised user.

Why is a certificate with O=system:masters so dangerous?

In Kubernetes, the system:masters group is a built-in superuser group whose requests are not checked against RBAC, so a client certificate with O=system:masters has full cluster access without any RoleBinding, and no RBAC change can remove it. Kubernetes cannot revoke individual client certificates, so such a certificate remains a working master key until it expires or the cluster CA is replaced.

Are Kubernetes Secrets encrypted?

Not by default. Kubernetes Secret values are base64-encoded, which anyone can decode, and the API server stores them in etcd unencrypted unless encryption at rest is configured with an EncryptionConfiguration, ideally using a KMS provider. Even with encryption at rest, anyone allowed to get, list or watch Secrets, or to create Pods in the namespace, can read the values through the API.

Do running Pods see an updated Secret?

A Kubernetes Pod that mounts a Secret as a volume sees updated values after the kubelet's next sync, without a restart, provided the volume is not mounted with subPath. Values consumed as environment variables, through secretKeyRef or envFrom, and files mounted with subPath are fixed when the container starts, so the container must be restarted to pick up a change.

What do the Kubernetes audit levels record?

Kubernetes audit policies use four levels. None records nothing. Metadata records who made the request, the verb, resource, namespace, source, timestamps and response status, but no bodies. Request adds the request body, and RequestResponse adds both request and response bodies. Sensitive resources such as Secrets and ConfigMaps are usually logged at Metadata so values are not copied into logs.

Why does a disabled OIDC user still have access to Kubernetes for a while?

The Kubernetes API server validates an OpenID Connect ID token on its own, by checking the provider's signature, issuer, audience and expiry, and does not contact the identity provider on each request. Disabling the account stops the user refreshing the token, but an ID token already issued keeps working until it expires, which is why short ID-token lifetimes matter.

What is the difference between the audit webhook modes blocking and blocking-strict?

Both Kubernetes audit webhook modes make the API server wait while each audit event is processed. In blocking mode, a failure to deliver an event is logged and the API request still proceeds. In blocking-strict mode, a failure to record the event at the RequestReceived stage fails the API request itself, so no request proceeds without its RequestReceived event being recorded, at the cost of availability. A failure at a later stage still lets the request proceed. The default batch mode sends events asynchronously and can drop them.

Source

This lesson covers the "Kubernetes Security Fundamentals" domain of the official KCSA exam guide. Vendors revise their guides — check the source for the current version.

Sign up free to mark lessons complete, bookmark topics and track your exam readiness.

Spotted a mistake in this lesson?