Kubernetes Administration: kubectl, RBAC, Namespaces, and Resource Quotas

kubectl is the command-line client for the Kubernetes API. Every command you type - kubectl get pods, kubectl apply, kubectl delete - is translated into an HTTPS request to the kube-apiserver. kubectl never connects to nodes, the kubelet, or etcd directly; the API server authenticates the request, checks authorization (RBAC), validates it, and acts on the cluster state. This single-entry-point design is why access control, auditing, and validation can all live in one place.

kubectl finds the cluster through a kubeconfig file, by default at ~/.kube/config. A kubeconfig holds three kinds of entries: clusters (API server addresses and certificate data), users (credentials), and contexts, which pair a cluster with a user and optionally a default namespace. Switching contexts with kubectl config use-context is how one workstation manages several clusters. Two details matter: the command writes current-context into the kubeconfig file itself, so every terminal using that file switches too (a per-shell choice needs the --context flag or a separate file via KUBECONFIG); and because a context bundles cluster, user, and default namespace, switching it changes all three at once.

The everyday verbs are worth recognizing on sight: kubectl get lists objects, kubectl describe shows detailed state plus recent events (the first stop when debugging), kubectl logs prints a container's output, kubectl exec runs a command inside a running container, and kubectl explain documents the fields of any resource type. Flags follow a consistent pattern: -n selects a namespace, -A spans all namespaces, -l filters by label selector, and -o yaml or -o wide change the output format. Knowing what each verb returns is the skill to build: events live in describe, container output in logs, field documentation in explain, and permission answers in auth can-i.

logs and exec take a longer path. The API server still authenticates and authorizes them, but it does not hold container output, so it opens a connection to the kubelet API on the Pod's node (TCP port 10250 by default) and relays the stream back. That explains two distinct failures. If the control plane cannot reach one node's kubelet (a firewall or security group blocking 10250), kubectl get pods still works - that data comes from the API server - but kubectl logs and exec for Pods on that node time out with a dial error naming the node's IP and port 10250. If instead the user lacks permission, the API server answers immediately with Forbidden, for every node alike. The error class tells you whether to look at the network path or at RBAC.

You have met the components as architecture; an administrator meets them as failure domains. Each component owns one job, so each has a characteristic symptom when it is unhealthy, and exam questions often describe the symptom and ask for the component.

Localising a failure means following the chain of handoffs. Creating a Deployment is a write to the API server; the Deployment controller (in kube-controller-manager) then creates a ReplicaSet, and the ReplicaSet controller creates Pods; the scheduler binds each Pod to a node; that node's kubelet starts the containers; and each node's kube-proxy programs Service routing. So the place the chain stops names the broken component. A Deployment that exists with no ReplicaSet or Pods at all points at the controller manager - nothing created them. Pods that exist but stay Pending with no node assigned point at the scheduler (or at a genuine lack of fitting nodes, which kubectl describe pod events reveal). Pods that are bound but never start point at that node's kubelet or container runtime. And when a Service's ClusterIP fails only from one node while Pod IPs answer directly, the Pods and the Service object are fine - the routing rules on that node are not, which points at that node's kube-proxy.

etcd and quorum. Cluster state lives in etcd, so backing up etcd (a snapshot) is the canonical way to back up a cluster. etcd uses the Raft consensus protocol and accepts writes only while a majority of members - the quorum, floor(n/2)+1 - can agree. That arithmetic is the real reason for odd member counts:

A fourth member adds no fault tolerance over three (both survive one loss) while adding a machine that must take part in every write, so even counts cost more for nothing. Odd counts are not about breaking tied leader elections - Raft resolves split votes with randomized election timeouts. When quorum is lost, etcd stops accepting writes, so the API server can no longer change cluster state.

Finally, in managed clusters (EKS, AKS, GKE) the provider runs the control plane, so an administrator's hands-on scope narrows to worker nodes, workloads, and the API - one reason the exam favors conceptual questions over operational ones.

A quick health picture comes from commands like kubectl get nodes, kubectl get pods -n kube-system, and kubectl describe node NODE_NAME - reading conditions and events rather than logging into machines.

There are two styles of driving the API, and KCNA expects you to tell them apart. Imperative commands state an action: kubectl create deployment web --image=nginx, kubectl run tmp --image=busybox, kubectl scale deployment web --replicas=5, kubectl expose deployment web --port=80. They are quick for experiments, but the resulting configuration exists only in the live cluster - there is no file to review, version, or reapply.

Declarative management states a destination: you keep manifests in files and run kubectl apply -f manifest.yaml (or -f a whole directory). Apply creates the object if it is missing and updates it if it drifted, so the same command is safe to run repeatedly - it is idempotent. Because the manifests are plain files, they can live in Git, go through code review, and be applied by automation; this is the foundation of the GitOps pattern of cluster management.

When manifests in a repository are the source of truth, kubectl apply is what keeps the cluster matching them, and it succeeds where kubectl create fails with an already-exists error because it computes and patches the difference instead of insisting on creating something new.

Mixing the two styles causes drift. Client-side kubectl apply records the configuration it applied in a kubectl.kubernetes.io/last-applied-configuration annotation on the object (kubectl apply view-last-applied prints it), and compares it with the manifest and the live object on the next run. An imperative change such as kubectl scale --replicas=8 or kubectl edit alters only the live object. If the manifest sets that field - say replicas: 3 - the next kubectl apply sets it back to 3, silently undoing the manual change. The durable fix is to change the manifest, or to remove the field from the manifest when something else (such as a HorizontalPodAutoscaler) should own it. kubectl diff -f previews what an apply would change.

Imperative commands still have one declarative trick worth knowing: --dry-run=client -o yaml prints the manifest an imperative command would create, which is a fast way to scaffold a file you then manage with apply.

Role-Based Access Control (RBAC) is how Kubernetes decides who may do what. It is built from exactly four API objects, in two pairs. A Role is a named list of permissions that applies within a single namespace. A ClusterRole is the same idea without the namespace boundary: it can grant access across all namespaces, or to cluster-scoped resources such as nodes and PersistentVolumes that no namespaced Role could ever cover. Neither object names any people - roles define what can be done, not who can do it.

Bindings supply the who. A RoleBinding grants the permissions of a role to a set of subjects within one namespace; a ClusterRoleBinding grants a ClusterRole's permissions across the entire cluster. One combination surprises people and is beloved by exam writers: a RoleBinding may reference a ClusterRole, which grants that ClusterRole's permissions only inside the binding's namespace. This lets administrators define a permission set once (say, a generic developer ClusterRole) and bind it per namespace, instead of copying identical Roles everywhere. The reverse limit matters just as much: because a RoleBinding is namespaced, a ClusterRole bound through RoleBindings only ever reaches namespaced resources - however many namespaces receive a RoleBinding, it never grants access to cluster-scoped resources such as Nodes or PersistentVolumes. Those need a ClusterRoleBinding.

Every binding has the same anatomy: a roleRef that names exactly one Role or ClusterRole (and cannot be changed after creation - you delete and recreate the binding), and a subjects list of User, Group, or ServiceAccount entries. Roles and ClusterRoles only ever appear in roleRef; they are never subjects.

Two properties define RBAC's character. It is additive and allow-only: there are no deny rules, permissions accumulate across bindings, and anything not explicitly granted is forbidden. And it defaults closed, which makes least privilege the natural practice: start from nothing and grant only the verbs and resources a subject genuinely needs.

No deny rules has a practical consequence: you cannot carve an exception out of a broad grant. If a team holds a ClusterRole through a ClusterRoleBinding (all namespaces) and must lose access to one namespace, nothing can be added to subtract it. The fix is to remove the ClusterRoleBinding and grant the same ClusterRole through a RoleBinding in each namespace the team should keep - and to automate creating that binding for new namespaces, since a per-namespace grant no longer covers them automatically.

Every RBAC rule answers three questions: who (the subject), may do what (the verbs), to which objects (the resources). Subjects come in three kinds: Users and Groups, which represent humans and are not Kubernetes API objects (they come from certificates or an external identity provider - there is no kubectl create user), and ServiceAccounts, which are API objects and represent workloads. Every namespace has a default ServiceAccount, every Pod runs as some ServiceAccount, and a Pod that needs to call the API should get its own ServiceAccount bound to a narrowly scoped Role.

Verbs are the actions the API server recognizes: get, list, and watch for reading; create, update, patch, and delete for writing. Resources are the object types the verbs act on - pods, deployments, services, secrets - and every rule names the API group they belong to. apiGroups: [""] (an empty string) is the core group: Pods, Services, ConfigMaps, Secrets, ServiceAccounts, Namespaces, and Nodes. Deployments, ReplicaSets, StatefulSets, and DaemonSets live in the apps group, Jobs and CronJobs in batch, and Roles and bindings in rbac.authorization.k8s.io. A rule listing deployments under the core group matches nothing, because no core resource has that name. A Role that lets its holder read Pods in the dev namespace looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

Subresources are authorized separately. Some operations act on a sub-path of an object, written resource/subresource in a rule. kubectl logs is a get on pods/log; kubectl exec needs create on pods/exec (and kubectl port-forward needs create on pods/portforward). A rule on pods does not cover any of them, so the pod-reader Role above lets its holder list Pods but answers Forbidden to kubectl logs until pods/log is added to its resources. Scaling a Deployment likewise touches deployments/scale.

A RoleBinding then attaches the Role to a subject - a user, a group, or a ServiceAccount. Inside the API, a ServiceAccount authenticates as the username system:serviceaccount:<namespace>:<name> and is also a member of the groups system:serviceaccounts (every ServiceAccount) and system:serviceaccounts:<namespace> (every ServiceAccount in that namespace), so bindings to those groups apply to it too.

To check the outcome, administrators use kubectl auth can-i, for example kubectl auth can-i list pods -n dev. With --as it impersonates another identity, and the exact string matters. --as=system:serviceaccount:build:ci-bot tests the ServiceAccount ci-bot from the build namespace, including its group memberships; -n prod separately sets the namespace whose permissions are being tested, so the two namespaces can differ. --as=ci-bot without the prefix impersonates an ordinary user named ci-bot, so the ServiceAccount's bindings and groups are never evaluated and the answer describes a different identity. Remember the human/workload split: ServiceAccounts for Pods, users and groups for people.

For an administrator, a namespace is the unit of tenancy: the boundary at which access is granted, quotas are enforced, and blast radius is contained. Object names must be unique only within their namespace, so multiple teams can use the same conventional names side by side. Policy objects attach at the same seam - a RoleBinding grants access within its namespace, and a ResourceQuota caps consumption within its namespace - so putting each team or environment in its own namespace gives each a private naming scope with tailored permissions and limits.

Not everything lives inside a namespace. Namespaced resources include Pods, Deployments, Services, ConfigMaps, Secrets, Roles, and RoleBindings. Cluster-scoped resources include Nodes, PersistentVolumes, ClusterRoles, ClusterRoleBindings, and namespaces themselves. You can ask the API which is which: kubectl api-resources --namespaced=true and --namespaced=false. This split is exactly why ClusterRoles exist - no namespaced Role could grant access to a Node.

Day to day, you select a namespace per command with -n, span all of them with -A, or set a default in your kubeconfig context with kubectl config set-context --current --namespace=dev. Deleting a namespace deletes every namespaced object in it - Pods, Deployments, Secrets, ServiceAccounts, Roles, RoleBindings - which makes namespaces convenient for disposable environments and dangerous to delete casually. Cluster-scoped objects are not in the namespace, so they survive even when they refer to it: a ClusterRoleBinding whose subject is a ServiceAccount in the deleted namespace stays behind and must be cleaned up separately (and would grant its access again to a same-named ServiceAccount if the namespace were recreated).

Know the limits of the boundary. Namespaces isolate names, access, and quota accounting - they do not isolate the network. With no NetworkPolicy in place, any Pod can reach any Pod in any namespace directly by its IP (or through a Service's cross-namespace DNS name); restricting that requires NetworkPolicies. Nor do namespaces isolate nodes: Pods from every namespace are scheduled onto the same machines unless scheduling rules (node selectors, affinity, taints and tolerations) keep them apart, and they compete for the same node capacity - which is precisely the problem quotas address next.

A ResourceQuota caps the total consumption of a namespace. It can limit aggregate compute - the sum of CPU and memory requests and limits across all Pods - and object counts, such as the number of Pods, Services, or PersistentVolumeClaims. When creating an object would push the namespace past its quota, the API server rejects the request at admission time with a Forbidden error naming the exceeded quota - the object is never created, so it is never queued or left Pending waiting for room. Because enforcement happens only at admission, quota changes are not retroactive: lowering a quota below current usage leaves existing Pods running and simply rejects new creations until usage falls back under the limit. A LimitRange works at the opposite altitude: it constrains individual containers or Pods in the namespace, setting default requests and limits for containers that do not declare their own, plus minimums and maximums for those that do.

The two are designed to work together, and a classic scenario shows why. An administrator gives the team-a namespace a ResourceQuota of 10 CPU and 20Gi of memory in total requests. Quotas on compute have a sharp edge: once a quota tracks a compute field, every new Pod must set that exact field, or the API server rejects it. A quota on requests.cpu requires every container to declare a CPU request; a quota on limits.memory requires a memory limit. A LimitRange with defaults is the usual way to fill those fields in automatically. A developer's minimal manifest with no resources section suddenly fails to create. The fix is a LimitRange in the same namespace that injects default requests and limits into any container that omits them - now plain manifests work again, every Pod is accounted against the quota, and no single container can request more than the LimitRange maximum.

Keep the division of labor straight, because the exam tests it directly: quota is the namespace's total budget, LimitRange is the per-container guardrail and default-setter. Both are namespaced objects enforced by the API server at admission, not at runtime - they shape what can be created, while the scheduler and kubelet deal with what actually runs. And remember what requests and limits themselves mean: a request is the amount the scheduler reserves when placing a Pod; a limit is the ceiling enforced on the running container.