Skip to main content

Migrate from Azure Blob Storage to Tigris

Azure Blob Storage is not S3-compatible. Tigris shadow buckets require an S3-compatible source. You cannot point a Tigris shadow bucket at Azure Blob Storage directly.

To migrate from Azure Blob Storage, you must put an S3-compatible shim in front of your Azure Blob account. This guide uses s3proxy, an open-source proxy that exposes an S3 API and forwards requests to Azure Blob Storage. You point the Tigris shadow bucket at the s3proxy endpoint. Tigris then migrates your data with zero downtime.

note

This setup adds a component that you must run and operate. s3proxy sits between Tigris and Azure Blob Storage. For a native S3 source, such as AWS S3, Google Cloud Storage, Cloudflare R2, or MinIO, see the provider-specific guides. Those sources need no shim.

Why an S3 shim is required​

Tigris is S3-compatible. Shadow bucket migration speaks the S3 API to your source bucket. Azure Blob Storage uses a different API and does not accept S3 requests. Tigris cannot read from it directly.

s3proxy solves this. s3proxy presents an S3 endpoint to Tigris. s3proxy translates each S3 request into an Azure Blob Storage request. Tigris sees a standard S3 source. Azure Blob Storage stays unchanged.

Migration approach​

Tigris supports lazy migration with shadow buckets. Tigris does not copy all your data upfront. Instead, Tigris fetches objects from the s3proxy endpoint on demand. It caches them for future access. No downtime is required.

You can also enable write-through mode, which syncs new writes back through s3proxy to your Azure Blob container. Your Azure Blob container stays up to date throughout the migration. You can take as long as you need before you complete the cutover.

Prerequisites​

Before you start, verify that you have:

  • A Tigris account with a bucket created
  • An Azure Storage account and the container you want to migrate from
  • An Entra ID identity with the Storage Blob Data Reader role on that container (recommended), or the storage account access key
  • A host to run s3proxy that Tigris can reach over the public internet
  • s3proxy 4.0.0 or newer, and Java 17 or newer on that host
  • No blobs in the Archive access tier (see the next section)
note

This guide uses the configuration properties of s3proxy 4.0.0. Releases before 4.0.0 use a different Azure backend with different properties and support account keys only.

Rehydrate archived blobs before you migrate​

s3proxy cannot read blobs in the Azure Archive access tier. Archive-tier blobs are offline in Azure. When the migration reaches an archived blob, the tigris buckets migrate command fails with the message an internal error occurred, try again. The message does not name the archived blob as the cause.

Containers with lifecycle management policies commonly contain archived blobs, because those policies move cold objects to the Archive tier automatically. No S3 shim reads archived blobs transparently. You must rehydrate them to the Hot or Cool tier before you migrate.

A retry after a failed drain is expensive. The tigris buckets migrate command walks the full container again from the start on each run. Rehydrate all archived blobs first, then run the drain once.

Rehydration changes blob data. Run the commands in this section with your own Azure identity, not with the read-only migration identity from the prerequisites. The commands use --auth-mode login to authenticate as the identity from az login. That identity must have the Storage Blob Data Contributor role, or a higher role, on the container. The Storage Blob Data Reader role can list blobs but cannot change their tier.

To find the archived blobs in a container:

az storage blob list \
  --account-name <azure-storage-account-name> \
  --container-name <container-name> \
  --auth-mode login \
  --query "[?properties.blobTier=='Archive'].name" \
  --output tsv

To rehydrate one blob, set its tier to Hot or Cool:

az storage blob set-tier \
  --account-name <azure-storage-account-name> \
  --container-name <container-name> \
  --auth-mode login \
  --name <blob-name> \
  --tier Hot

To rehydrate every archived blob, pipe the list into a loop that runs the same command one time for each blob:

az storage blob list \
  --account-name <azure-storage-account-name> \
  --container-name <container-name> \
  --auth-mode login \
  --query "[?properties.blobTier=='Archive'].name" \
  --output tsv |
  while IFS= read -r blob_name; do
    az storage blob set-tier \
      --account-name <azure-storage-account-name> \
      --container-name <container-name> \
      --auth-mode login \
      --name "$blob_name" \
      --tier Hot
  done

The loop reads one blob name from each line and keeps spaces, quotes, and backslashes in the name intact. A blob name that contains a newline character is the one exception. Rehydrate such a blob with the single-blob command.

caution

Rehydration is not immediate. A standard-priority rehydration can take up to 15 hours. Wait until no blob reports Archive before you start the migration.

Step 1: Configure s3proxy for Azure Blob Storage​

s3proxy reads its settings from a properties file. Create a file named s3proxy.conf with the Azure Blob backend.

s3proxy 4.0.0 authenticates to Azure Blob Storage in two ways. Entra ID is the recommended way because you can scope it to one container with a read-only role. The storage account access key is the fallback.

caution

Do not deploy the example credential values. Anyone with these public values can read your Azure Blob objects. The s3proxy endpoint must stay publicly reachable. Generate your own strong, unique S3 credentials for s3proxy.identity and s3proxy.credential. Use the same credential values in both the s3proxy config and the Tigris shadow-bucket command. The s3proxy.identity value must match the --access-key value. The s3proxy.credential value must match the --secret-key value. Replace the example credential values with your own secrets.

Create a service principal with the Storage Blob Data Reader role. Scope the role to the one container that you migrate from:

az ad sp create-for-rbac \
  --name s3proxy-migration \
  --role "Storage Blob Data Reader" \
  --scopes "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<azure-storage-account-name>/blobServices/default/containers/<container-name>"

Leave jclouds.identity and jclouds.credential empty in the properties file. s3proxy then authenticates with DefaultAzureCredential from the Azure SDK:

s3proxy.endpoint=http://0.0.0.0:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
s3proxy.read-only-blobstore=true
jclouds.provider=azureblob
jclouds.identity=
jclouds.credential=
jclouds.endpoint=https://<azure-storage-account-name>.blob.core.windows.net

Give the service principal credentials to s3proxy through environment variables. Set them in the environment that starts s3proxy in Step 2:

export AZURE_CLIENT_ID=<service-principal-app-id>
export AZURE_TENANT_ID=<entra-tenant-id>
export AZURE_CLIENT_SECRET=<service-principal-secret>
note

DefaultAzureCredential also supports managed identities. If s3proxy runs on an Azure VM with a managed identity, assign the Storage Blob Data Reader role to that identity and set no environment variables.

The two credential pairs serve different roles:

  • s3proxy.identity and s3proxy.credential are the S3 access key and secret that Tigris uses to authenticate to s3proxy. You set them to values you choose. You enter these same values in the shadow bucket configuration in Step 3.
  • jclouds.identity and jclouds.credential are your Azure storage account name and account access key. When you leave them empty, s3proxy reads Entra ID credentials from its environment instead.

Set s3proxy.read-only-blobstore=true in every setup. Migration only reads from the source. With this property, s3proxy rejects write and delete requests with a NotImplemented error, and read requests continue to work. This limits the damage if your s3proxy credentials leak.

note

Write-through mode (Step 6) writes back to Azure through s3proxy. If you plan to use write-through, remove s3proxy.read-only-blobstore=true, and give the Entra ID identity the Storage Blob Data Contributor role instead.

Step 2: Run s3proxy​

Start s3proxy with your properties file.

java -jar s3proxy --properties s3proxy.conf

s3proxy now listens on the s3proxy.endpoint address. Test it with an S3 client before you continue.

caution

The example endpoint uses plain HTTP on port 8080. Tigris is a hosted service. Tigris must reach s3proxy over the public internet with a valid TLS certificate. Before you configure the shadow bucket, put s3proxy behind a public HTTPS endpoint. See the s3proxy SSL support guide for TLS options.

Step 3: Configure the shadow bucket​

Point the shadow bucket at your s3proxy endpoint. Use the s3proxy.identity and s3proxy.credential values from Step 1 as the access key and secret key.

  1. Go to the Tigris Dashboard
  2. Click Buckets in the left menu
  3. Select the bucket you want to migrate data into
  4. Click Settings
  5. Find Enable Data Migration and toggle it on
  6. Enter your s3proxy connection details:
    • Endpoint: The public HTTPS URL of your s3proxy instance
    • Region: auto (s3proxy does not require a region)
    • Access Key ID: The s3proxy.identity value from Step 1
    • Secret Access Key: The s3proxy.credential value from Step 1
    • Bucket: The Azure Blob container name
note

s3proxy maps each S3 bucket name to an Azure Blob container. Set the shadow bucket Bucket field to the name of the Azure Blob container that holds your data.

Step 4: Update your application​

Tigris exposes an S3-compatible API, not the Azure Blob Storage API. You must change your application to use the Tigris SDK or an AWS S3 SDK. The Azure Blob Storage SDK will not work against Tigris. Set the endpoint and credentials for Tigris in your S3 client configuration.

With the AWS CLI:

aws s3 ls s3://your-bucket/ \
  --endpoint-url https://t3.storage.dev \
  --region auto

Or with boto3:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="https://t3.storage.dev",
    region_name="auto",
    aws_access_key_id="your-tigris-access-key",
    aws_secret_access_key="your-tigris-secret-key",
)

Your bucket names and object keys stay the same after migration. You must change your application source code to use the Tigris SDK or an AWS S3 SDK, not the Azure Blob Storage SDK.

Step 5: Verify the migration​

Once your application points to Tigris, objects migrate on first access. To verify:

  1. Request an object that exists in your Azure Blob container
  2. Verify that it returns successfully through Tigris
  3. Verify that later requests are served directly from Tigris

Step 6: Enable write-through (optional)​

To keep your Azure Blob container in sync during the migration, enable write-through in the shadow bucket settings. With write-through enabled:

  • New objects written to Tigris are also written through s3proxy to Azure Blob Storage
  • Deletes apply to both Tigris and Azure Blob Storage
  • Object listings include the full contents of the Azure Blob container

This keeps your Azure Blob container current so you can fall back at any point.

Write-through needs write access through s3proxy. Remove s3proxy.read-only-blobstore=true from s3proxy.conf and restart s3proxy. If you use Entra ID, give the identity the Storage Blob Data Contributor role on the container.

Step 7: Complete the migration​

caution

Run tigris buckets migrate and verify that it completes before you disable the shadow bucket configuration. The command copies all objects to Tigris server-side. Lazy migration copies an object only on first access. Objects that no request ever touches stay only in Azure Blob Storage. If you disable the shadow configuration and decommission Azure Blob Storage before the full migration completes, Tigris cannot serve those objects. The data becomes unavailable.

Once your workloads run well on Tigris, disable the shadow bucket configuration. Tigris becomes your primary object store. You can then shut down s3proxy and decommission your Azure Blob Storage.

Access Tigris from your application​

Tigris exposes an S3-compatible API, not the Azure Blob Storage API. After you move to Tigris, your application must talk to Tigris through the Tigris SDK or an AWS S3 SDK. The Azure Blob Storage SDK will not work against Tigris. Set the endpoint to https://t3.storage.dev and the region to auto. The SDK reads your credentials from the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The examples below configure an S3 client against the Tigris endpoint and list the objects in a bucket.

import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";

const S3 = new S3Client({
  region: "auto",
  endpoint: "https://t3.storage.dev",
  s3ForcePathStyle: false,
});

const response = await S3.send(
  new ListObjectsV2Command({ Bucket: "your-bucket" }),
);

for (const object of response.Contents ?? []) {
  console.log(object.Key);
}

Azure-specific considerations​

  • s3proxy is an extra component. You must run, secure, and monitor it for the length of the migration. If s3proxy goes down, Tigris cannot reach objects that are not yet migrated.
  • Public reachability and TLS. Tigris must reach s3proxy over the public internet with a valid TLS certificate. Do not expose the plain HTTP example endpoint to production traffic.
  • Container-to-bucket mapping. s3proxy maps an S3 bucket name to an Azure Blob container. Use the container name as the shadow bucket Bucket value.
  • Credential handling. The s3proxy.conf file and the s3proxy environment hold your Azure credentials. Restrict access to both. Tigris never receives the Azure credentials. Tigris authenticates to s3proxy with the separate s3proxy.identity and s3proxy.credential values.
  • Least privilege. Prefer an Entra ID identity with the Storage Blob Data Reader role on one container. An account access key grants full control of the whole storage account and cannot be restricted. Set s3proxy.read-only-blobstore=true in every setup that does not use write-through.
  • Authentication support. s3proxy 4.0.0 supports account keys and Entra ID through DefaultAzureCredential, which includes service principals and managed identities. Releases before 4.0.0 default to an older backend that supports account keys only.
  • Archived blobs block the migration. s3proxy cannot read blobs in the Archive access tier. Rehydrate them before you migrate. See Rehydrate archived blobs before you migrate.

FAQ​

Can Tigris read from Azure Blob Storage directly?​

No. Azure Blob Storage is not S3-compatible. A Tigris shadow bucket needs an S3-compatible source. You must front Azure Blob Storage with an S3 shim such as s3proxy.

Why does tigris buckets migrate fail with "an internal error occurred, try again"?​

A blob in the Azure Archive access tier is a common cause. s3proxy cannot read archived blobs, and the error message does not name them as the cause. List the archived blobs in the container and rehydrate them, then run the drain again. See Rehydrate archived blobs before you migrate. Each retry walks the full container again, so rehydrate all archived blobs before you retry.

Does migration require downtime?​

No. Shadow bucket migration happens transparently. Your application continues to serve requests while objects migrate on first access.

Do I need to change my application code?​

Yes. You need to change your application code. Tigris is not compatible with the Azure Blob Storage API. You must change your application source code to use the Tigris SDK or S3 API instead of the Azure Blob Storage API.

Can I roll back to Azure Blob Storage?​

Yes. If you enable write-through mode, your Azure Blob container stays in sync with all new writes. You can revert your endpoint configuration to switch back at any point.

What happens to objects I do not access?​

They remain in your Azure Blob container. Tigris only copies objects when they are first requested. Objects that are never accessed are never transferred.

Can I remove s3proxy after the migration?​

Yes. s3proxy is only required while Tigris migrates from Azure Blob Storage. Once you disable the shadow bucket configuration, you can shut down s3proxy.

Last updated on by Mantas Mikšys