Managing Cloud Storage and Databases: Lifecycle, Backups, and CMEK

You secure Cloud Storage objects with IAM, and on almost every bucket you should enable uniform bucket-level access (UBLA), which disables object and bucket ACLs so IAM policies alone govern access. That gives one consistent, auditable permission model. The legacy alternative, fine-grained access, lets each object carry its own ACL alongside IAM; it is needed only for rare per-object permission schemes and is a common source of accidental exposure. Grant access with roles such as roles/storage.objectViewer (read objects), roles/storage.objectCreator (upload only), roles/storage.objectAdmin (full object control) and roles/storage.admin (buckets and objects), for example gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME --member=user:ana@example.com --role=roles/storage.objectViewer. The gcloud storage command group replaces the older gsutil workflows.

What changes when you switch a bucket to uniform access

• ACL requests fail. While UBLA is on, any request that reads or sets an ACL (including uploads that pass a predefined ACL) is rejected with HTTP 400. Applications that set ACLs must be changed to rely on IAM.
• Access granted only through ACLs disappears. Anyone who could read an object only because of its ACL loses access; grant them an IAM role on the bucket (or use IAM Conditions) instead.
• There is a 90-day window back. You can disable UBLA only within 90 days of enabling it. Cloud Storage keeps the old ACLs and restores them if you switch back in time; objects uploaded while UBLA was on receive the bucket's default object ACL. After 90 days the change is permanent.
• The organization policy affects new buckets only. The storage.uniformBucketLevelAccess constraint forces UBLA on buckets created after it is set; it does not convert existing buckets, which you must update individually (gcloud storage buckets update gs://BUCKET_NAME --uniform-bucket-level-access). While the constraint is enforced, buckets that already have UBLA also cannot disable it, so the 90-day window only helps where the constraint does not apply.

Public access prevention is a separate control: it blocks any grant to allUsers or allAuthenticatedUsers, through IAM or ACLs, but it does not disable ACLs and does not replace UBLA. Use both when a bucket must never be public and must be governed by IAM only.

To give someone without a Google identity or IAM grant temporary access, sign the request with a service account's credentials instead of changing the bucket policy. Two mechanisms exist, and the difference is what the holder may do.

A V4 signed URL is valid for at most 7 days, for example gcloud storage sign-url gs://BUCKET_NAME/report.pdf --duration=1h --private-key-file=key.json (or sign through the IAM Credentials API to avoid downloading a key). A signature only delegates access the signer already has: the signing service account must itself be allowed to read (or write) the object, otherwise the URL returns an access error. Neither mechanism needs the bucket to be public, and making a bucket public to share one file is never the right fix.

Cloud Storage has three features that keep data around after a delete or overwrite, and they solve different problems: versioning keeps old generations indefinitely, soft delete keeps deleted data for a fixed window, and retention policies and holds stop the change from happening at all.

Soft delete is on by default for new buckets, so even a bucket without versioning can recover an accidental delete for a week. It bills the retained data at the object's storage class rate for the retention window. Versioning plus lifecycle rules suits keeping a set number of earlier copies; soft delete suits a safety net against mistakes; a retention policy (or event-based/temporary holds) suits compliance rules that must stop even administrators from deleting data.

A lifecycle policy automates cost control: each rule pairs one action with one or more conditions, and Cloud Storage applies the action when every condition in the rule is true. The main actions are SetStorageClass (move the object to a colder class only: Standard to Nearline, Coldline or Archive; Nearline to Coldline or Archive; Coldline to Archive) and Delete (remove the object, or a noncurrent version); AbortIncompleteMultipartUpload cleans up abandoned uploads.

Counting versions and days correctly

• numNewerVersions includes the live version. To keep the live object plus K noncurrent versions, delete when numNewerVersions is K + 1. With N = 1 every noncurrent version is deleted as soon as it exists, because the live version is already newer.
• age counts from creation, not from replacement. A rule of isLive: false plus age: 14 deletes an old version the moment it is replaced, because it was created long ago. To keep each replaced version for 14 days after it was replaced, use daysSinceNoncurrentTime: 14.

Policies are JSON applied to the bucket: gcloud storage buckets update gs://BUCKET_NAME --lifecycle-file=lifecycle.json. A typical policy for versioned log data moves objects to Nearline at 30 days, Coldline at 90 and Archive at 365, and deletes noncurrent versions with numNewerVersions: 5 (live plus four earlier copies). Changes can take up to 24 hours to take effect.

Storage class costs that lifecycle rules trigger

Nearline, Coldline and Archive carry minimum storage durations of 30, 90 and 365 days and charge retrieval fees per GiB read. The minimum duration is measured from the object's original creation time, including time spent in an earlier class: an object created 40 days ago and moved to Coldline can be deleted 50 days later without an early-deletion charge. Deleting or replacing it sooner bills the remainder. A rule that pushes frequently read data into Archive therefore raises cost. When access patterns are unknown or shift, enable Autoclass, which moves each object between classes based on its access; an Autoclass bucket cannot also have SetStorageClass lifecycle rules (Delete rules are still allowed).

Each data service has a native query path, and you choose by data model and service. The table summarizes the tools; the notes below cover the behaviour that trips people up.

Cloud SQL and AlloyDB

gcloud sql connect INSTANCE_NAME --user=postgres allowlists your current IP address on the instance for a short time (about five minutes) and opens the engine's shell, which makes it convenient for ad hoc queries from Cloud Shell but not for permanent connections. For applications, the Cloud SQL Auth Proxy opens IAM-authorized, encrypted connections without authorized networks or a fixed client IP. Its prerequisites are the Cloud SQL Admin API enabled in the project and an identity holding roles/cloudsql.client (Cloud SQL Client). Without the proxy, a public-IP connection needs the client's address in authorized networks; client SSL certificates encrypt and authenticate the connection but do not replace that allowlist. AlloyDB follows the same pattern with its own Auth Proxy and standard PostgreSQL clients.

BigQuery with bq

The bq query command uses legacy SQL by default unless a default dialect is configured (in .bigqueryrc, or with the project or organization default_sql_dialect_option setting). Pass --use_legacy_sql=false (or --nouse_legacy_sql) to run GoogleSQL: bq query --use_legacy_sql=false 'SELECT name, COUNT(*) FROM dataset.table GROUP BY name'. Results print in the terminal. Adding --dry_run validates the query and reports the bytes it would process without running it, returning no rows and incurring no query charge.

Spanner, Bigtable and Firestore

For Spanner, gcloud spanner databases execute-sql DATABASE_NAME --instance=INSTANCE_NAME --sql='SELECT SingerId, FirstName FROM Singers WHERE LastName = "Rivera"' runs SQL from Cloud Shell and prints the result. Bigtable is a wide-column store addressed by row key. The cbt CLI covers everyday reads: cbt lookup TABLE ROW_KEY fetches one row, cbt read TABLE prefix=PREFIX (or start=/end=) scans a range, cbt count TABLE counts rows, and cbt ls TABLE lists a table's column families. Bigtable also supports GoogleSQL queries through Bigtable Studio and the client libraries, and you can still query it from BigQuery with an external table. Firestore is queried as documents in collections through client libraries or the console, not with table SQL.

You estimate storage cost from three factors — volume stored, storage class or tier, and access pattern — and the Google Cloud Pricing Calculator turns planned usage into a number before any resource exists. For Cloud Storage the estimate is GiB-months stored per class, plus operation charges, plus retrieval fees for cold classes, plus network egress, plus early-deletion charges where objects leave before their minimum duration. Archive is the cheapest at rest but expensive to read, so the right class depends on how often data is accessed, not just how much there is.

BigQuery separates storage from compute. Storage is billed for data at rest, and any table or partition that is not modified for 90 consecutive days automatically moves to long-term storage at about half the active rate. Querying a table does not reset that timer; loading, updating or streaming data into it does. Long-term pricing applies under both the logical and the physical (compressed) storage billing models, and there is no performance difference. Compute is either on-demand, billed per TiB scanned by each query, or capacity-based, billed for reserved slots. On-demand is why partitioning, clustering and dry runs matter: a query pruned to one partition scans and bills a fraction of the table.

Cloud SQL and AlloyDB bill for provisioned vCPUs, memory and storage over time plus backup storage, so their estimates look like VM sizing; Spanner bills compute capacity plus storage and backups; Bigtable bills nodes plus storage. After launch, use actual-spend tools instead of estimates: billing reports break spend down by service and SKU, budgets alert on thresholds, and exporting billing data to BigQuery lets you query it. Use the Pricing Calculator for planned designs and billing reports, budgets or the billing export for what you are actually spending.

Each managed database has its own backup mechanism, and the main question is whether it restores to a backup's state or to an exact moment, and whether the restore overwrites something or creates a new resource.

Cloud SQL

Enable automated backups (daily, in a window you choose) with gcloud sql instances patch INSTANCE_NAME --backup-start-time=03:00, and take on-demand backups with gcloud sql backups create --instance=INSTANCE_NAME. Restoring a backup overwrites the target instance's data. The target may be the original instance or a different one: gcloud sql backups restore BACKUP_ID --restore-instance=TARGET_INSTANCE --backup-instance=SOURCE_INSTANCE overwrites TARGET_INSTANCE with a backup taken from SOURCE_INSTANCE and leaves the source untouched. Point-in-time recovery (PITR) requires automated backups plus transaction log retention (binary logging on MySQL, write-ahead log archiving on PostgreSQL), and always creates a new instance at the chosen timestamp: gcloud sql instances clone prod-db prod-db-recovered --point-in-time='2026-10-05T10:39:00Z'. Read replicas and the high-availability standby are not recovery tools: they mirror the primary's current data, including an accidental DELETE or corruption.

Spanner, Firestore, AlloyDB and Bigtable

Commands worth recognising: gcloud spanner backups create BACKUP --instance=INSTANCE --database=DATABASE --retention-period=30d (add --version-time to capture an earlier moment), gcloud firestore backups schedules create --database='(default)' --recurrence=weekly --day-of-week=SUN --retention=10w, and gcloud firestore databases restore --source-backup=BACKUP_NAME --destination-database=NEW_DB. Managed backups differ from exports: a Firestore export to Cloud Storage or a Spanner export via Dataflow is portable data for migration or long-term archive, while native backups are faster to restore but live inside the service.

You check Dataflow job status on the console's Dataflow Jobs page, where each job shows its state (Running, Succeeded, Failed, Cancelled, Drained), the pipeline graph, per-step metrics, autoscaling history and worker logs. From the CLI, gcloud dataflow jobs list --region=REGION lists jobs and gcloud dataflow jobs describe JOB_ID --region=REGION shows one job. To stop a streaming job, drain finishes processing in-flight data then stops, while cancel stops immediately and may drop buffered records. Diagnose failures from the step-level logs and error counters.

In BigQuery, every query, load, export and copy runs as a job. The console's personal and project Job history shows status, duration and bytes processed. From the CLI, bq ls -j --max_results=20 lists recent jobs (-a includes all users' jobs if you have permission) and bq show -j JOB_ID prints one job's status and error detail. For analysis across many jobs, query the INFORMATION_SCHEMA job views with SQL. They are region-qualified, not dataset-qualified, and you query the region where the jobs ran, for example `region-europe-west2`.INFORMATION_SCHEMA.JOBS.

The views expose columns such as user_email, query, state, error_result and creation_time, so ordinary SQL filters and aggregates answer most job-history questions. Operationally, connect job monitoring to alerting: Dataflow metrics flow into Cloud Monitoring for alerting policies on failures or system lag, and BigQuery job errors appear in Cloud Logging where log-based alerts apply.

Database Center is a single console view of your whole database fleet — Cloud SQL, AlloyDB, Spanner, Bigtable, Firestore, Memorystore and more — across projects, so you stop checking instance pages one by one. It inventories every database resource and adds health signals: availability risks such as instances without high availability or automated backups, security findings such as broad network exposure, performance issues, and cost recommendations, largely from Active Assist recommenders. It also offers a Gemini-powered chat for natural-language questions about the fleet.

The workflow is triage: review the fleet summary of issues by category and severity, drill into an affected resource, and act on the recommendation. Position it against its neighbours: Database Center is fleet-wide inventory, health and recommendations; Cloud Monitoring is metrics, dashboards and alerting for any resource; Query Insights (Cloud SQL and AlloyDB) finds which queries consume CPU or run slowly inside one instance; Cloud Asset Inventory lists and searches resources and their metadata of every type, without database health analysis. Finding every database missing backups is a fleet question; diagnosing slow or expensive queries inside one instance is Query Insights work.

All data in Google Cloud is encrypted at rest by default with Google-managed keys; CMEK changes who controls the key, not whether encryption happens. With CMEK, data is protected by a key you create and manage in Cloud KMS: you control rotation, you can disable or destroy it, and its use is audited. Any protection level works — software, HSM (Cloud HSM) or external (Cloud EKM) — as long as the service supports it. Customer-supplied encryption keys (CSEK), where you send raw key material with each request, are a rarer, heavier alternative available for Cloud Storage and Compute Engine.

Setup has three steps and one critical grant. First, create a key ring and key: gcloud kms keyrings create my-ring --location=us-central1, then gcloud kms keys create my-key --keyring=my-ring --location=us-central1 --purpose=encryption. The key can live in a different project from the data (a central key project is common), but its location must be compatible with the resource it protects. Second, grant the data service's service agent roles/cloudkms.cryptoKeyEncrypterDecrypter on the key. Each service has its own per-project agent, for example service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com for Cloud Storage and bq-PROJECT_NUMBER@bigquery-encryption.iam.gserviceaccount.com for BigQuery; the grant goes to the agent of the project that owns the data. Third, attach the key: set a bucket's default key with gcloud storage buckets update gs://BUCKET_NAME --default-encryption-key=KEY_RESOURCE_NAME, or pass the key when creating a Cloud SQL instance, BigQuery dataset or Spanner database. For many services, CMEK can only be chosen at creation.

So keep old key versions enabled for as long as data encrypted with them exists. When storage "suddenly became inaccessible" after a security cleanup, the usual cause is a disabled or destroyed key version or a removed service agent grant.