AZ-305 Semi-Structured and Unstructured Storage Design: Cosmos DB, Blob Storage, Azure Files and Data Protection

Designing storage for semi-structured and unstructured data means choosing where JSON documents, key-value entities, files and binary objects live in Azure, and then configuring that store so it meets the workload's performance, cost, durability and protection requirements. The official task has four skills: recommend a solution for storing semi-structured data, recommend a solution for storing unstructured data, recommend a data storage solution that balances features, performance and costs, and recommend a data solution for protection and durability.

Semi-structured data has a shape but no fixed schema: product documents whose attributes vary by category, device readings, user profiles. Unstructured data is anything stored as an opaque object or file: images, video, scanned forms, logs, backups and data-lake files. In Azure, a small set of services covers almost every case.

Choosing the service is only half the design. The rest is configuration: Cosmos DB capacity mode, partition key and consistency level; storage account type, access tier, lifecycle rules and redundancy; and the recovery, immutability and encryption features that protect the data. As everywhere in AZ-305, the right recommendation meets every stated requirement at the least cost or effort. Relational stores belong to the sibling topic on relational data, and moving or analysing data belongs to the data integration topic. Backups with Azure Backup and regional failover are covered in the business continuity domain.

Use Azure Cosmos DB for NoSQL when documents vary in shape and are queried on many properties; use Azure Table storage when entities are only ever read by their keys and cost matters most. Cosmos DB for NoSQL stores schema-agnostic JSON items and, by default, indexes every property, so a new attribute becomes queryable without schema or index work. Adding regions to the account replicates the data so each region serves local reads with single-digit millisecond latency.

Azure Table storage lives in a storage account and indexes only PartitionKey and RowKey. A query that filters on any other property scans entities, so it is slow at scale. It has no secondary indexes and no multi-region writes; its only geo option is the storage account's redundancy. In return it is billed only for capacity and transactions, which makes it the cheapest choice for key-and-range access such as device ID plus a time-based row key.

Choosing a Cosmos DB API

Cosmos DB exposes several APIs over the same engine. Choose the API from the application that already exists, because the API decides which SDK or driver connects:

• API for NoSQL is the native API and the default for new applications. Moving an app to it from another data model means changing its data access code.
• API for Table accepts apps written for Azure Table storage with the Azure Tables SDK. Usually only the connection string changes, and the app gains automatic indexing of all properties, global distribution and latency guarantees. It costs more than Table storage, so it is worth it only when those features are needed.
• API for Apache Cassandra implements the Cassandra wire protocol, so existing CQL drivers connect unchanged. There are no nodes to size, patch or add: capacity is request units per second (RU/s) set at keyspace or table level, and regions are added to the account. Azure Managed Instance for Apache Cassandra is the alternative when you need full Apache Cassandra compatibility and control over configuration; it automates deployment and patching, but you still choose node SKUs and node counts.
• API for MongoDB (request-unit based) gives MongoDB wire-protocol compatibility on Cosmos DB. Microsoft now directs vCore-based MongoDB workloads to Azure DocumentDB, the renamed Cosmos DB for MongoDB vCore, so check current guidance before recommending either for a MongoDB migration.

Choose serverless for intermittent, low-traffic workloads in one region, autoscale for variable or spiky traffic, and manual provisioned throughput for steady, predictable traffic. Every Cosmos DB operation costs request units (RUs), and the capacity mode decides how you pay for them.

The deciding facts are usually two. First, serverless can't span regions: a workload that must read from a second region needs provisioned throughput, however bursty it is. Second, when traffic swings between a busy peak and long idle periods, autoscale with its maximum set to the known peak avoids throttling and costs less than a fixed peak setting; a fixed setting at the average throttles at peak. Reserved capacity discounts provisioned throughput in return for a one- or three-year commitment, but it still bills capacity every hour, so it doesn't help a workload that is mostly idle.

A good partition key has many distinct values, spreads writes evenly over time, and matches the filter most queries use. Cosmos DB groups items with the same partition key value into a logical partition and spreads logical partitions across physical partitions. Provisioned throughput is divided between physical partitions, so if most requests target one key value, that partition is throttled with HTTP 429 even though the container as a whole has spare RU/s. That is a hot partition.

• Time-based keys are classic hot spots. A key such as the order date sends all of today's writes to one value; shrinking it to the hour only moves the hot spot into a smaller window.
• Low-cardinality keys don't scale. Status, region or category values are too few to spread load.
• Match the dominant query. If most reads fetch everything for one customer, a customer key makes those reads efficient single-partition queries. Queries without the partition key fan out across partitions and cost more RUs.
• Keep items that change together in one logical partition. Transactional batch operations and stored procedures are scoped to a single logical partition, so items that must be updated atomically (an order header and its lines, for example) must share a partition key value. Pick a key that does that and still has high cardinality: the key should group the items written together while still having many distinct values, and a key whose few values carry most of the traffic or data creates hot partitions.

Each logical partition has a fixed storage limit (20 GB). In multitenant designs keyed on tenant ID, a few large tenants can hit it while thousands of small ones are fine. Hierarchical partition keys (up to three levels, such as tenant ID then user ID) let one tenant's data span several physical partitions, while queries that supply the tenant ID are still routed only to the partitions holding that tenant. A synthetic key with a random suffix also spreads data, but queries for one tenant then have to fan out. Raising throughput never raises the storage limit. A container's partition key can't be changed after creation, so fixing a bad key means a new container and a data migration.

Cosmos DB offers five consistency levels, from strongest to weakest: strong, bounded staleness, session, consistent prefix and eventual; stronger levels give fresher reads at the cost of latency, availability and RUs. The level is set on the account and can be relaxed per request.

Read the requirement for its scope. "A user must see their own change immediately, others may lag" is session, and it keeps reads cheap. "No read may ever be stale, anywhere" is strong, which also forces a single write region. A guaranteed maximum lag, measured in time or in versions, with reads in order and a single write region, is bounded staleness. Session guarantees belong to one client session, so they don't give other clients any bound on how far behind they read.

Use Blob Storage for objects accessed by REST, Azure Files for managed shares that applications mount over SMB or NFS, Azure NetApp Files when a shared file system needs sub-millisecond latency, and shared managed disks only for clustered applications that manage their own block-level access.

Azure Files tiers and protocols

Azure Files has SSD (premium) shares, in a FileStorage account, and HDD (standard) shares (pay-as-you-go in a general-purpose v2 account, or provisioned v2 in a FileStorage account). NFS shares are available only on SSD file shares; HDD shares are SMB-only. NFS shares support LRS and ZRS and must be reached through a private endpoint or service endpoint. SSD shares give low single-digit millisecond latency; when a workload needs sub-millisecond latency, look at Azure NetApp Files instead.

Mounting blob containers on Linux

Two ways let Linux machines mount blob containers as directories while the data stays in Blob Storage and other services keep using the REST API. BlobFuse2 is a FUSE driver installed on each node; it translates file operations into blob calls and has weaker POSIX semantics than a real file system. NFS 3.0 for Blob Storage requires an account created with a hierarchical namespace and NFS 3.0 enabled (it can't be turned on later), and accepts traffic only from allowed virtual networks. Azure File Sync and NFS 4.1 belong to Azure Files, and SFTP for Blob Storage is a transfer endpoint, not a mount.

Data Lake Storage and the hierarchical namespace

Enabling the hierarchical namespace (HNS) turns a storage account into Azure Data Lake Storage. Directories become real objects, so renaming or deleting a directory with millions of files is one atomic metadata operation, which suits Spark and other job-commit patterns; with a flat namespace, folders are just name prefixes and a rename touches every blob. HNS also adds POSIX-style ACLs on directories and files alongside Azure RBAC. It is a trade-off, because some Blob Storage features aren't supported on HNS accounts:

Microsoft's feature support table for Blob Storage is the reference; check it for versioning and change feed too. When one workload needs a lake and another needs those features, the usual design is two accounts: an HNS account for analytics and a flat-namespace account for the archive.

Match the account type to the performance need first, then use access tiers to match each blob's storage cost to how often it is read.

Standard and premium accounts

• Standard general-purpose v2 is the default for blobs, files, queues and tables. It supports every access tier and every redundancy option.
• Premium block blob accounts store data on SSDs for consistently low latency and high transaction rates, which suits IoT telemetry, many small objects and interactive workloads. They cost more per GB but less per transaction. They don't support access tiers and offer LRS and ZRS only, so geo-redundancy needs a standard account.
• Premium file shares (FileStorage accounts) are the SSD option for Azure Files, required for NFS shares.

Blob access tiers

Each step down lowers the storage price and raises the read and transaction price. Moving or deleting a blob before its minimum retention incurs an early deletion charge for the remaining days. Two rules decide many designs. Data that needs immediate (online) access rules out archive, because an archived blob can't be read until it is rehydrated to an online tier; cold is then the cheapest choice for rare reads. And archive is supported only in LRS, GRS and RA-GRS accounts, not in ZRS, GZRS or RA-GZRS. So a design that combines zone redundancy with archiving needs the archived data in a separate account whose redundancy supports archive.

Use lifecycle management rules when you know the age or access pattern that should move or delete data, and smart tier when access is unpredictable and you want Azure to move objects between online tiers with no rules to maintain.

Lifecycle management

A lifecycle policy is a set of rules, each with a filter (blob type, name prefix, blob index tags) and actions with conditions. Azure evaluates the policy about once a day.

• Conditions are days since creation (daysAfterCreationGreaterThan), since last modification (daysAfterModificationGreaterThan) or since last access (daysAfterLastAccessTimeGreaterThan). The last-access condition works only after last access time tracking is enabled on the account.
• Actions tier blobs to cool, cold or archive, or delete them. With a last-access rule, enableAutoTierToHotFromCool moves a blob back to hot when it is read again.
• Separate action sets exist for base blobs, snapshots and previous versions. A version action can delete previous versions a set number of days after they were created, which controls the cost of blob versioning without touching current blobs.

Modification-based rules tier on writes, not reads, so they are the wrong tool when "unused" means "not read". Azure Storage Actions storage tasks can also apply tier changes across accounts, but they are still rules you define and maintain.

Smart tier

Smart tier is set as the account's default access tier. New data starts in hot; objects that aren't accessed move to cool and, after further inactivity, to cold, and an object moves back to hot as soon as it is accessed. There are no early deletion, tier transition or retrieval fees, only a small monitoring fee per object larger than 128 KiB. Smart tier doesn't use archive and requires a standard general-purpose v2 account configured for ZRS, GZRS or RA-GZRS; it isn't available for LRS, GRS or RA-GRS accounts. For an LRS account that needs read-based tiering, use lifecycle management with access tracking.

Pick redundancy from the failure the data must survive: LRS survives hardware failure in one datacenter, ZRS survives a zone failure, GRS and GZRS add an asynchronous copy in a secondary region, and the RA- variants let applications read that copy without a failover.

Map each requirement to one property. "No data loss if a zone fails" needs synchronous zone replication: ZRS or GZRS. "Keep reading if the primary region is down, without waiting for a failover" needs read access to the secondary: RA-GRS or RA-GZRS, which publish a read-only endpoint with a -secondary suffix. With GRS or GZRS the secondary copy exists but isn't readable until a failover. RA-GZRS is the only option that combines zone redundancy in the primary region with a readable secondary; without the zone requirement, RA-GRS gives the readable secondary at lower cost. Geo-replication is asynchronous, so recent writes can be lost in a regional disaster. Remember the account-type limits too: premium block blob and NFS file shares support only LRS and ZRS. Planning and running failovers is covered in the high availability topic of the business continuity domain.

Protect blob data with layered, built-in features: soft delete and versioning to recover from mistakes, point-in-time restore to roll containers back, immutable storage to meet retention rules, and encryption keys you control where regulations demand it. Azure Resource Manager locks protect the storage account resource from being deleted or changed, but they don't stop anyone deleting containers or blobs through the data plane.

Recovering from deletes and overwrites

Backing up blobs and file shares with Azure Backup belongs to the backup and disaster recovery topic.

Immutable (WORM) storage

Immutable storage keeps data in a write once, read many state. A time-based retention policy starts unlocked, so it can be tested, shortened or deleted; once locked it can't be deleted and its interval can only be extended. The locked state is what has been assessed for regulations such as SEC Rule 17a-4(f), FINRA and CFTC rules. A legal hold keeps data immutable until the hold is cleared, for litigation or investigations whose end date isn't known. Scope matters:

• Container-level WORM applies one policy to every blob in the container, so it works when data can be grouped by retention period. It is supported with a hierarchical namespace.
• Version-level WORM lets policies be set at account, container or individual blob version level, so each blob can have its own retention date. It requires blob versioning (an amended blob becomes a new version while the protected earlier version is kept) and isn't supported on hierarchical namespace accounts.

Encryption at rest

Azure Storage always encrypts data at the service level with 256-bit AES. The design choice is who controls the key and whether a second layer is added:

An account-wide requirement for double encryption on an existing account therefore means creating a new account with infrastructure encryption on and moving the data.