Data backup from Linux servers to Azure with Duplicity - pricing

by Michal Smereczynski

Back in 2015 I wrote post about backing up data from Linux servers to Azure Blob Storage (with encryption). It's a good time to refresh this topic with some updates.

  1. There was client-side only encryption back in 2015.
  2. There was a problem with Duplicity compatibility with new Azure Storage API.
  3. There was only one pricing tier for storage back in 2015 - no storage tiers dedicated to backup solutions.

Ad.1. Now we have "Azure Storage Service Encryption for Data at Rest":

Azure Storage Service Encryption (SSE) for Data at Rest helps you protect and safeguard your data to meet your organizational security and compliance commitments. With this feature, Azure Storage automatically encrypts your data prior to persisting to storage and decrypts prior to retrieval. The encryption, decryption, and key management are totally transparent to users.

Encryption and Decryption Workflow

Here is a brief description of the encryption/decryption workflow:

  • The customer enables encryption on the storage account.
  • When the customer writes new data (PUT Blob, PUT Block, PUT Page, etc.) to Blob storage; every write is encrypted using 256-bit AES encryption, one of the strongest block ciphers available.
  • When the customer needs to access data (GET Blob, etc.), data is automatically decrypted before returning to the user. If encryption is disabled, new writes are no longer encrypted and existing encrypted data remains encrypted until rewritten by the user. While encryption is enabled, writes to Blob storage will be encrypted. The state of data does not change with the user toggling between enabling/disabling encryption for the storage account.
  • All encryption keys are stored, encrypted, and managed by Microsoft.

Ad.2. Now Duplicity is fully compatible with stable Azure Storage API.

If You are using Ubuntu 16.04 LTS, just install Duplicity from distorts repository. If You need fresh version for Your distribution, just get it from Duplicity PPA

Ad.3. There is new pricing tier for backup-oriented solutions. Cool storage.

Cool Blob Storage – low cost storage for cool object data. Example use cases for cool storage include backups, media content, scientific data, compliance and archival data. In general, any data that is seldom accessed is a perfect candidate for cool storage. With the new Blob storage accounts, you will be able to choose between Hot and Cool access tiers to store object data based on its access pattern.

Differences?

Availability: The Hot access tier guarantees high availability of 99.9% while the Cool access tier offers a slightly lower availability of 99%.

How to create encrypted, cool blob storage account compatible with Duplicity?

azure storage account create --sku-name LRS --kind BlobStorage --access-tier Cool --enable-encryption-service Blob -l <region> -g <resource group name> <account name>

Data backup in Azure Blob Storage Pricing

We have two pricing tiers for Blob Storage in Azure - hot and cool. But we also have replication options which will change the pricing:

  • Locally redundant storage (LRS): Locally redundant storage replicates your data three times within a storage scale unit which is hosted in a datacenter in the region in which you created your storage account.
  • Geo-redundant storage (GRS): Geo-redundant storage (GRS) replicates your data to a secondary region that is hundreds of miles away from the primary region. If your storage account has GRS enabled, then your data is durable even in the case of a complete regional outage or a disaster in which the primary region is not recoverable.
  • Read-access geo-redundant storage (RA-GRS): Read-access geo-redundant storage (RA-GRS) maximizes availability for your storage account, by providing read-only access to the data in the secondary location, in addition to the replication across two regions provided by GRS.

Assumptions for backup

We need to have some sample backup assumptions for every option for good comparison. So let's say we have:

  • 1024GB of data to back up
  • 100 000 put, list, create container operations on storage (around 100 000 files)
  • 100 000 other operations (just for "other" situations)
  • Azure will retrieve 1024GB of data at once (in one month) and write it.

Assumptions for recovery

After all that we recover, all the data from backup (at once):

  • 100 000 other operations (get)
  • 1024GB of bandwidth (data out from Azure)

Backup calculations

  1. LRS:

    Capacity (1024GB): €8.64/MO

    Put operations (100000): €0.84/MO

    Other operations (100000): €0.08/MO

    Data write (1024GB): €2.16/MO

    Total: €11.72/MO

  2. GRS:

    Capacity (1024GB): €17.27/MO

    Put operations (100000): €1.69/MO

    Other operations (100000): €0.08/MO

    Data write (1024GB): €4.32/MO

    Data geo-replication (1024GB): €17.27/MO

    Total: €40.63/MO

  3. RA-GRS:

    Capacity (1024GB): €21.59/MO

    Put operations (100000): €1.69/MO

    Other operations (100000): €0.08/MO

    Data write (1024GB): €4.32/MO

    Data geo-replication (1024GB): €17.27/MO

    Total: €44.95/MO

Recovery calculations

  1. LRS:

    Data retrieval (1024GB): €8.64/MO

    Bandwidth (1024GB, Zone 1: North America, Europe): €74.76/MO

  2. GRS:

    Data retrieval (1024GB): €8.64/MO

    Bandwidth (1024GB, Zone 1: North America, Europe): €74.76/MO

  3. RA-GRS:

    Data retrieval (1024GB): €8.64/MO

    Bandwidth (1024GB, Zone 1: North America, Europe): €74.76/MO

Conclusion

As You see, LRS backup on Cool Storage is the most cost-effective. With LRS we have 3 replicas of data for €11.72 in first month and €8.64/MO after initial backup (assuming that we had backed up 1024GB at once in first month and then just store it). Yearly expense of this solution is €106,76. Storing this data for 10y will cost us €1067,60. It's hard to achieve this cost effectiveness with traditional disk arrays in triple redundancy scenario. But...

If we are going to restore all the data from this backup, it will be much more expensive than storing backup itself. The question is how often we will restore whole backup?

P.S. All the prices above are from official Azure services pricing and cost calculator. This means that the prices are for pay-as-you-go subscriptions. If You will contact Microsoft Partner You can buy those services much cheaper (even more than 25%).