As a result, we really want to have our own business logic define which side of a conflict wins. FoundationDB doesn’t let us do that out of the box, so we had to make our own layer. The neat part about being able to do this is that this lets us control the replication behaviour of buckets based on user needs. The three main ways it differs are with single-region buckets, multi/dual-region buckets, and global buckets. I’m going to cover the replication differences in the order of complexity.

Single-region buckets​

One of the easiest conflict resolution methods we have is a single-region bucket, which prevents any need for it. In this mode all actions are reverse proxied to the bucket’s region and any metadata changes are local to that region in particular. This means that any other regions trust the changes made by the bucket’s region and reject any changes made by any other regions. As a side effect this also means that the data does not move between regions like other buckets do. You’d think that would mean “if a region goes down, my bucket goes down.” But we have implemented data proxying, which means that if that region’s block storage is still available, other gateways can route requests to it and you can still access your data come hell, high water, or Giant Meteors.

The conflict resolution flow is like this:

  ┌────────────┐      ┌────────────────────┐      ┌───────────────────────────────┐
  │ client     │ ───▶ │ nearest Tigris     │ ───▶ │ owning region                 │
  │ PutObject  │      │ gateway            │      │                               │
  └────────────┘      │                    │      │ 1. bytes ──▶ block store      │
                      │ which region owns  │      │ 2. metadata ──▶ FoundationDB  │
                      │ this bucket?       │      │    (one gRPC commit)          │
                      └────────────────────┘      │ 3. return success             │
                                                  └───────────────┬───────────────┘
                                                                  │
                                                      reads queue │ (async)
                                                                  ▼
                                                  ┌───────────────────────────────┐
                                                  │ replication worker            │
                                                  │ every other region gets       │
                                                  │ a read-only copy              │
                                                  └───────────────────────────────┘

The main tradeoff with a single-region bucket is that you trade strict consistency (because there is only one possible writer) with higher latency (unless all of your workload is geographically close to that Tigris region in particular). This makes a Tigris bucket mostly behave like a region-locked S3 bucket with the exception of being able to query the files globally without having to configure your client to access that particular region.

Global buckets​

That was nice and simple. In comparison, global buckets are not.

Global buckets let any region be authoritative for any aspect of the bucket or any object in the bucket. Any changes get committed to the local FoundationDB cluster and then lazily replicated out to the other regions. This also means that we expect there to be some level of conflict. Imagine an object storage bucket like a git repository. You end up having conflicts when multiple people push to the same files. How do we decide who wins? You can’t just have a person sit there and decide which version of an object is right all the time.

Remember that we run a separate FoundationDB cluster per region. FoundationDB has a sequencer that gives version identity to everything in the cluster. Those versions are also based on time so you think we’d just be able to use those and compare them to know which version is the newest, right? Well, it turns out that FoundationDB versionstamps aren’t comparable across clusters as they’re partly based on the cluster’s creation date and all of our clusters were created at different times as we scaled globally. So we have to use something else that changes fairly constantly between clusters in a way that’s easy for us to validate without too much extra effort. Ideally, it’s something we already keep in sync for making sure everything else in the system works.

Like many other things in Tigris, we use time to determine who wins a conflict.

Here’s a paraphrased and reformatted version of our conflict resolution function:

  compare(previous, new, force)
  │
  ├── is prev or next unset, or a brand-new key in FDB?
  │      yes: different objects ──────────▶  drop, log the conflict  ✗
  │
  ├── is prev older than the new data?
  │      yes: the new version is newer ───▶  apply, row replaced     ✓
  │
  └── tie, or new is older: is this a forced change?
         yes: tie forcibly broken ────────▶  apply, row replaced     ✓
         no ──────────────────────────────▶  drop, log the conflict  ✗

Or: we prioritize the most recent entry when possible, accounting for deletions and the like such that a newer delete wins over an older update. This combined with eventual consistency means that there are potentially situations where one region updates an object and another region deletes it at the same time. During that small replication window you can get a situation where Singapore says an object exists that Chicago says doesn’t exist, but in practice the replication delay is small enough (single digit seconds or thereabout) that it doesn’t matter.

Time keeps ticking​

At this point clock skew is also a factor. We run NTP clients on all our infrastructure and that usually keeps us within about 10 microseconds (10,000 nanoseconds) off of NTP time. Given that we measure timestamps as nanoseconds to decide conflicts, clock skew-based ordering conflicts can genuinely be a factor.

Imagine that the two operations in Chicago were served by different servers that just so happen to have their clocks off by a fraction of a fraction of a second. A DELETE could be sequenced before a PUT and the object could be shown as deleted in Chicago but present in Singapore because that PUT replicated out after the delete was committed locally.

According to all the rules of the game, every region but Chicago sees that the DELETE is slightly older than the current data, so it ignores that DELETE and continues as if that PUT is the right state of the world.

 client          Chicago (bucket owner)         Singapore
    │                       │                       │
    │ PUT thing.txt         │                       │
    ├──────────────────────▶│                       │
    │                       │                       │   stamped …000200
    │                       │                       │   (the clock ran a hair fast)
    │                       │                       │
    │                       │ replicate @ …000200   │
    │                       ├──────────────────────▶│
    │                       │                       │   Singapore stores the
    │                       │                       │   live copy @ …000200
    │                       │                       │
    │ DELETE thing.txt      │                       │
    ├──────────────────────▶│                       │
    │                       │                       │   the clock still reads …000100,
    │                       │                       │   so the tombstone is stamped
    │                       │                       │   BEHIND the row it deletes
    │ 204 No Content        │                       │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤                       │
    │                       │                       │
    │                       │ tombstone @ …000100   │
    │                       ├──────────────────────▶│
    │                       │                       │   …000100 is not newer than
    │                       │                       │   …000200: the tombstone is
    │                       │                       │   dropped, the live copy survives
    │                       │                       │
    │ GET thing.txt         │                       │
    ├───────────────────────┼──────────────────────▶│
    │ 200 OK, the deleted object is back            │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
    │                       │                       │
    │ PUT If-None-Match: *  │                       │
    ├──────────────────────▶│                       │
    │ 412 Precondition Failed                       │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤                       │

This is the kind of race condition you can only really see in distributed systems at our scale or larger. Nothing that happens in this situation is a failure in the logic or implementation, it’s just a desync because of unfortunate timing.

I wish I could tell you an epic tale of time synchronization and other fun things along the way of fixing this particular issue, but the fix was sadly boring. We just refuse to process a DELETE when the DELETE is older than the data in the database. If the DELETE doesn’t land after the database row’s date, we re-read the clock and try again every 100 milliseconds up to 5 (five) times. If even that can’t result in a timestamp that sorts correctly we give up loudly to the client instead of quietly doing the wrong thing.

Multi-region buckets​

Finally we have multi-region buckets. These are the most complicated as they combine aspects of both single-region and global buckets. There wasn’t a protocol off the shelf that would do this for us– FoundationDB uses Paxos* internally, and I’ve read about MultiPaxos (and Matchmaker Paxos, Matchmaker MultiPaxos, it goes on…). It seems like everyone modifies the algorithm for their use case, I wonder if there is a “true” Paxos outside of academia. Maybe it was implemented by the one true Scotsman.

We replicate multi-region bucket metadata by having one region be the leader for a group and having that leader actively push out changes as well as enqueueing them like a global bucket. This means that the regions in the group get the data faster than they would otherwise and commits to the leader mean that the data is committed to all members of the group. It’s kinda like this:

 client          gateway         leader        follower 1      follower 2
    │               │               │               │               │
    │ PUT object    │               │               │               │
    ├──────────────▶│               │               │               │
    │               │ bytes + meta  │               │               │
    │               ├──────────────▶│               │               │
    │               │               │               │               │   upload to the leader's block
    │               │               │               │               │   store, commit the metadata:
    │               │               │               │               │   ONE transaction writes the row
    │               │               │               │               │   and enqueues replication
    │               │ committed     │               │               │
    │               │◀─ ─ ─ ─ ─ ─ ─ ┤               │               │
    │               │               │               │               │
    │               │ apply this right now          │               │
    │               ├───────────────┼──────────────▶│               │
    │               │ apply this right now          │               │
    │               ├───────────────┼───────────────┼──────────────▶│
    │               │               │               │               │   pushed to every follower
    │               │               │               │               │   at once, in parallel
    │               │ ok            │               │               │
    │               │◀─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ┤               │
    │               │ error         │               │               │
    │               │◀─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ┤
    │               │               │               │               │
    │ 200 OK        │               │               │               │
    │◀─ ─ ─ ─ ─ ─ ─ ┤               │               │               │
    │               │               │               │               │   one follower is enough
    ·               ·               ·               ·               ·
    │               │               │               │               │   seconds later, the queue delivers
    │               │               │               │               │   the same change all over again
    │               │               │ queued copy   │               │
    │               │               ├──────────────▶│               │
    │               │               │               │               │   same timestamp, not newer:
    │               │               │               │               │   follower 1 drops it
    │               │               │ queued copy   │               │
    │               │               ├───────────────┼──────────────▶│
    │               │               │               │               │   nothing here yet, so it applies:
    │               │               │               │               │   follower 2 catches up

When a client writes to a multi-region bucket, Tigris forwards the write to the leader and blocks there. If the leader can’t take it, the write errors out and nothing changes. If the leader commits the change, the gateway actively pushes that change out to the other regions in the group in parallel and waits for them to commit before answering the client. Interestingly enough, it uses the same codepath that the global replication workers use. Every region in the group is going to get this change twice: once from the fan-out on the leader’s commit and the other from the queue worker pushing it out a moment later.

This may also seem like a race condition, but remember that the replication rows contain the before and after state, a-la Postgres logical replication or git commit syncing. If the current data in the database is newer or the same as the data being pushed out, the change is ignored and the cluster moves on with life:

 client            gateway           leader          every follower
    │                 │                 │                   │
    │ PUT object      │                 │                   │
    ├────────────────▶│                 │                   │
    │                 │ commit the metadata                 │
    │                 ├────────────────▶│                   │
    │                 │                 │                   │   the row is written and durable.
    │                 │                 │                   │   there is no undo from here
    │                 │ committed       │                   │
    │                 │◀─ ─ ─ ─ ─ ─ ─ ─ ┤                   │
    │                 │                 │                   │
    │                 │ apply this right now                │
    │                 ├─────────────────┼──────────────────▶│
    │                 │ error           │                   │
    │                 │◀─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
    │                 │                 │                   │
    │ 5xx, your write failed            │                   │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ┤                 │                   │
    │                 │                 │                   │   no follower succeeded
    ·                 ·                 ·                   ·
    │                 │                 │                   │   but the queued copy was already
    │                 │                 │                   │   durable, committed in the same
    │                 │                 │                   │   transaction as the row
    │                 │                 │ queued copy       │
    │                 │                 ├──────────────────▶│
    │                 │                 │                   │   applies normally
    │                 │                 │                   │
    │                 │                 │                   │   the write you were told failed is
    │                 │                 │                   │   now readable in every region

At some level you can think about multi-region buckets as a latency optimization for the regions in the multi-region group. The replication queue is what actually guarantees the global convergence of metadata, but the active fan-out over regions is what gets there first. This means that if you have a multi-region bucket in the EU and all your workloads are in the EU, you get a lot of the same availability advantages of global buckets without a lot of the consistency risks of normal global buckets.

Replication brings its own challenges​

Our current system has handled all customer load without too many issues. So that any region can answer questions about the data, all metadata is replicated to every region, even if it’s not requested anywhere else. This means that for even single region buckets, there’s the same load on the replication queue as there is for global buckets.

To work around this, we can compartmentalize the activities for these single region buckets by sharding our FoundationDB clusters by bucket type, or even by tenant, so the replication queue doesn’t risk lagging. We have a finite number of replication workers and if too many objects change all at the same time it can cause the replication delay to be minutes instead of seconds. The technical term for this is “bad”.

Right now there’s one central FoundationDB cluster per region which stores everything. In the future we plan to have one cluster that’s used to map metadata about which buckets/organizations belong to which clusters, and from there we will scale out depending on customer request pressure. If you’re a large enough tenant, you may end up getting your own dedicated FoundationDB cluster!

  S3 request
  │
  ▼
  ┌─────────┐ ── which replicaset holds this tenant? ──▶ ┌────────────────┐
  │ gateway │                                            │ shard mapping  │
  └─────────┘ ◀─ ─ ─ ─ ─ ─  "replicaset 2"  ─ ─ ─ ─ ─ ─  │ cluster        │
       ║                                                 └────────────────┘
       ╚═══════════════════╗
                           ▼
  ┌──────────────┐   ╔══════════════╗   ┌──────────────┐
  │ replicaset 1 │   ║ replicaset 2 ║   │ replicaset 3 │
  │ 2-3 FDB      │   ║ 2-3 FDB      ║   │ 2-3 FDB      │
  │ clusters +   │   ║ clusters +   ║   │ clusters +   │
  │ worker pool  │   ║ worker pool  ║   │ worker pool  │
  └──────┬───────┘   ╚══════╤═══════╝   └──────┬───────┘
         │                  │                  │
         └──────────────────┼──────────────────┘
                            ▼
                 ┌───────────────────────┐
                 │ global pointer table  │
                 └───────────────────────┘

This does come with the obvious downside of having to manage multiple clusters, but we think the tradeoff is worth it when it eliminates the problem of noisy neighbors causing replication delay. It will mean that our caching layer has to be a bit more complicated (at the very least we expect the mapping of organizations/buckets to FoundationDB clusters to be fairly stable), but that’s just a simple matter of programming at this point. It’s also gonna make replication “fun”, but we’ll cross that bridge when we come to it.

Conclusion​

There’s three types of buckets but all of them use different shades of the same global replication logic. Most of the levers we offer are all around consistency, latency, and availability. Single region buckets remove the question of where the data is stored at the cost of making that one region a single point of failure. Global buckets embrace conflicts but can be a more latent when changes are replicated out. Multi-region buckets double-replicate your data across the cloud. All of these happen in regions that are far apart enough that synchronous round trips are genuinely expensive, but everything is just implementation details for the same basic object storage operations.

Computers really are something at this scale, aren’t they? The joys of our industry know no end.

Hello everyone! I'm Himank, the CTO and a co-founder of Tigris Data. Tigris is a globally distributed object store.

Tonight other speakers talked about the scale of data involved with training foundation models. Something has to hold all of that data. That's the part that Tigris works on.

My talk will be focusing more on the metadata storage of Tigris, like how we are using FoundationDB in our whole stack to power our metadata.

What is Tigris?​

First, Tigris is a global storage platform that is fully S3 compatible. From a user perspective you don't need to do any code changes. It will simple work with your existing S3 stack. All you need to do is switch from your existing S3-compatible provider to Tigris and it will just work.

Second, we're rethinking the object store from the ground up. Traditionally object storage is known for cold caching. We're focusing on reframing object storage as global infrastructure. How can we provide an object store as global infrastructure? Users should not need to think about replication or caching. We want users to get a bucket, upload data, and the data is available everywhere in the world.

Finally, we want to make the data globally available and optimize the read latency so that it doesn't matter where your compute is located. Tigris makes your data follow your compute so you can pick from any compute or GPU provider without worrying about your data being slow.

In order to make Tigris global by default, we're using FoundationDB. We store all object metadata, users, and buckets in FoundationDB.

Read and write in any regions​

When we were designing our system, we were thinking a lot about how we can make the components loosely coupled, but have an aggregated architecture where storage is decoupled from other components. This would let us scale any of these components independently.

For example, in each region we have our gateway, caches, metadata clusters, block storage backends, and asynchronous queue workers. All of these services can be scaled independently based on our needs, running on metal. The gateway and workers are stateless, everything else is where the state lives.

So the obvious question at that point is what should we use for our metadata storage? Metadata is very important. Tigris is mostly exciting ways to arrange metadata with boring ways to store data.

Why FoundationDB?​

There were a few options, either we built something on our own or used something from outside. We picked FoundationDB for a few reasons:

• ACID metadata: In FoundationDB, mutations are one transaction. There is no possibility of having split-brain metadata.
• Ordered keyspace: FoundationDB keys have an inherent order, so versioned keys encode bucket, object, and index order.
• Serializeable layout: Indices, metadata, and chunks mutate together.
• Operational safety: FoundationDB is high availability almost to a fault, replicates cleanly without human intervention, and has simulation testing to the level that Aphyr didn't even bother to evaluate it.
• Battle-tested in production: FoundationDB is used by companies like Apple and Snowflake as the foundation of their cloud infrastructure.

At some level, it's best to think about FoundationDB as a distributed filesystem that handles the hard parts for you: sharding, consensus, replication, and transactions. You then get to build your own layer on top of it. FoundationDB doesn't provide a schema, table layout, or indices. That's in the part you are expected to provide.

FoundationDB at the core​

Once we committed to FoundationDB we had to make a few changes to how we did things. In FoundationDB it's impossible to read or write data without a transaction. This sounds like a lot of overhead until you realize what it gives us. A single object write usually involves reading the current state, updating the new state, and then enqueueing the object for replication and indexing.

We also had to design our row layout, so we designed one that lets us support efficient scanning for our users:

  key   ·   lexicographic, one contiguous range per subspace                 value
 
  ┌──────────┬──────────┬────────────┬────────────────────────────────┐      ┌────────────────┐
  │ tenant   │ bucket   │ subspace   │ object / index / key           │      │ value          │
  ├──────────┼──────────┼────────────┼────────────────────────────────┤      ├────────────────┤
  │ t_9f3a   │ photos   │ obj        │ 2026/08/img_001.jpg            │  ──▶ │ manifest ptr   │
  └──────────┴──────────┴────────────┴────────────────────────────────┘      └────────────────┘
 
  └─────────────────── one FDB key, tuple-encoded ────────────────────┘

This lets us serve efficient queries for a single bucket. We don't need to worry about sharding or multi-hit transactions. This comes by default with FoundationDB.

What do writes do?​

So what does a PUT look like? As I mentioned before, we use strictly ACID transactions. A single write for us reaches the layer and we break the write into the data block and the metadata. We write the block into block storage first and then we start a FoundationDB transaction. In this transaction we write object metadata and then update our indices along with that metadata. We also have a bunch of other work that needs to be done after a write completes, so we write to the queue in the same transaction.

What is that other work? Tigris is global, so we have to replicate the data globally. We also have caches that need to be updated, so we have asynchronous queue workers that can handle all this. But to run that async machinery we need to have some kind of task mechanism that we do as part of our write.

Normally you end up having to juggle two transactions: one to your database and another to your message queue. We implemented our message queue in FoundationDB using the fact that both FoundationDB and time are ordered. We don't have to run a distributed transaction between two systems or add expensive recovery logic to ensure tasks don't get lost. Either everything commits or nothing commits, which is one of the best parts of FoundationDB.

Once this write commits and we know that any work items have been added to their queues, we return to the user and the transaction completes successfully.

How do we resolve GETs?​

How do we ensure that objects can be served from any region when the data may be stored in any other region?

One way to think about Tigris is that it's a multi-tier cache that has endpoints all over the world. When users request objects, a combination of anycast routing and geo-DNS make sure that requests go to the closest datacenter. All our metadata is eagerly replicated between FoundationDB clusters in each region, and that metadata includes where the object actually lives.

  ┌────────────┐   ┌────────────────┐   ┌────────────────────┐   ┌────────────┐
  │ client GET │──▶│ nearest region │──▶│ local FDB metadata │──▶│ location   │
  └────────────┘   └────────────────┘   └────────────────────┘   └──────┬─────┘
  choose the byte source                                                │
             ┌───────────────────────┬───────────────────────┬──────────┘
             ▼                       ▼                       ▼
  ┌─ HIT ──────────────┐  ┌─ LOCAL ────────────┐  ┌─ MISS ─────────────┐
  │ SSD / block cache  │  │ local block store  │  │ remote source      │
  └──────────┬─────────┘  └──────────┬─────────┘  └──────────┬─────────┘
             │                       │                       ├──▶ enqueue warm-cache / move
             └───────────────────────┬───────────────────────┘
                                     ▼
                      ┌──────────────┬─────────────┐
                      │ return bytes to client     │
                      └────────────────────────────┘
  // the read can complete before data placement catches up

If the object is in the local block cache, that gets served directly to the client. If the object is in the local block store, that also gets served directly to the client. If the data isn't stored locally, Tigris needs to fetch it from another region. In order to do that it reverse proxies the read to the block store in the region where the data actually lives. Since a user requested it, we enqueue a block store replication job so that the next GET request is faster. This makes future GETs much more efficient.

Our queueing system​

We built our own queueing system on top of FoundationDB by following what Apple did with their QuiCK: A Queueing System in CloudKit paper. The queue is just an ordered FoundationDB keyspace. Workers claim rows transactionally. The work itself happens asynchronously outside of a transaction, but the coordination layer is FoundationDB.

Building our own queue on top of FoundationDB lets us ensure that we have ACID semantics when we add tasks to the queue. This also lets us ensure that we have at-least-once semantics for tasks in the queue. We cannot lose tasks because they are always present in the FoundationDB queue.

These task writes are atomic with the metadata so either an object writes successfully with its tasks and metadata or nothing happens. No in-between state. An interesting side effect of this is that our workers can crash all they want. Another worker will pick the task up without any human intervention.

┌─ server / API plane ───────────────────┐   ┌─ worker deployment ──────────────────┐
│ ┌────────────────────────┐             │   │ ┌─────────────┐       ┌──────────┐   │
│ │ api gateway            │             │   │ │ scheduler 1 │──┬───▶│ worker 1 │   │
│ └────────────┬───────────┘             │   │ └─────────────┘  │    └──────────┘   │
│              │                         │   │                  │    ┌──────────┐   │
│              ▼                         │   │                  └───▶│ worker 2 │   │
│ ┌────────────────────────┐             │   │                       └──────────┘   │
│ │ transaction layer      │             │   │                       ┌──────────┐   │
│ └────────────┬───────────┘             │   │                       │ worker 3 │   │
│              │                         │   │                       └──────────┘   │
│              ▼                         │   │                                      │
│ ┌────────────────────────┐             │   │ ┌─────────────┐       ┌──────────┐   │
│ │ query processing       │             │   │ │ scheduler 2 │──┬───▶│ worker 4 │   │
│ └────────────┬───────────┘             │   │ └─────────────┘  │    └──────────┘   │
│         ┌────┴──────────┐              │   │                  │    ┌──────────┐   │
│         ▼               ▼              │   │                  └───▶│ worker 5 │   │
│ ┌──────────────┐ ┌─────────────┐       │   │                       └──────────┘   │
│ │ metadata row │ │ queue item  │       │   │                       ┌──────────┐   │
│ └───────┬──────┘ └──────┬──────┘       │   │                       │ worker 6 │   │
│         └─────────────┬─┘              │   │                       └──────────┘   │
│                       ▼                │   │                                      │
│            ┌──────────┬─────────┐      │   │                                      │
│            │ commit             │      │   │                                      │
│            └──────────┬─────────┘      │   │                                      │
│                       │                │   │                                      │
└───────────────────────┬────────────────┘   └─────────┴────────────────────────────┘
                        │                              ▲
                        ▼                              │
┌─ FOUNDATIONDB ────────┬──────────────────────────────┴─────────────────────────────┐
│                ┌──────┬─────┐               ┌────────┴───────┐                     │
│                │ data       │               │ queue          │                     │
│                └────────────┘               └────────────────┘                     │
└────────────────────────────────────────────────────────────────────────────────────┘
// one transaction writes the metadata row and the queue item together

Our global replication system is built on the back of this queue, meaning that you can list objects anywhere in any region and you will get the same view of your bucket in every region. This is all driven by our queue.

When a user does a read on an object and there isn't a copy of it locally, this async queueing system automatically kicks in to make sure that the data is copied over so it's hot and ready for next time. Future reads can just happen from the cache on that particular region.

Conclusion​

FoundationDB is more than just our metadata storage layer, it's the foundation to the Tigris platform. All objects, indices, and tasks use FoundationDB as the bedrock for storage. This lets us have data propagate everywhere in the world without having to build a multi-region FoundationDB cluster or add an expensive second coordination system into our stack. Every region has their own FoundationDB clusters that are all kept in sync automatically.

Storing metadata, indices, and task queues in the same database lets us focus more of our time on delivering users global object storage and less of our time fiddling with the details on how to make Kafka performant. Without FoundationDB we would not have the primitives we need to make Tigris happen.

And that's it, thank you!

Further reading​

• Skipping the boring parts of building a storage platform using FoundationDB — why we didn't build our own distributed transaction layer.
• Tigris metadata layer on FoundationDB — the key encoding, integer ID compression, and value format in detail.
• Backing up FoundationDB

Tigris has a pretty advanced replication scheme for writes. What happens when you actually need to delete things? Turns out deleting things is hard in distributed systems. Especially when you have a geo-replicated active-active database like Tigris does. We can (and do) use tombstones to mark where data once was, but how do you let people undo an accidental delete?

Tigris wants to turn storage inside out, so our implementation of soft deletion is by giving users the Recycle Bin for objects and buckets. Today we’re going to dig into how this works, why it works, and what this gives you in terms of using object storage today.

Recycle bins and you​

In Windows and macOS, the Recycle Bin (or Trash can) is a form of purgatory where deleted files wait for their storage to be deallocated by the user. This allows users to hit “delete” fearlessly because if they made a mistake they can just drag it back out and go on with life.

This works great in your local filesystem because there’s only one writer in one region. This kinda falls apart when you have multiple regions in your database and any one of them could be writing to it. How do you name things in the recycle bin? How do you handle the conflict of an update happening in one region before the deletion was fully replicated out from another region?

This is the fun of distributed systems, which is the kind of problem space that Tigris lives in.

One way to think about how the Recycle Bin works is that the file metadata gets moved there when the user hits delete. No data bytes move around on the disk, but the file doesn’t show up in My Documents anymore. In a distributed systems context you can’t just move the metadata around, you have to leave a tombstone behind to record where that metadata once was. This prevents other regions from being confused when actions happen really close to each other in time.

Soft deletes in some universes​

At a high level, a soft-delete is when a DELETE action doesn’t actually remove the data. When data is soft-deleted, it’s still there but just not visible in the main usage flow. This lets you get the data back when a delete is made by accident.

Your database becomes your API​

One of the interesting side effects of designing any API is that you end up leaking the internals of how your database works to your users. Many object storage systems were designed with overwriting or deleting data as one of the primary operations, and as such have had to bolt versioning onto the side. For the most part this does work; but once you get into advanced versioning schemes everything starts to fall apart. Tigris doesn’t suffer from the same problems because we built immutability into the core from day one, and in immutable systems you have to append new data on the end instead of overwriting data.

At the least, actually storing the data en masse is a boring problem. You put the data somewhere, maybe name it after the checksum of its contents, and then have a daemon make sure it’s copied three places. That daemon also handles cases when drives go offline and new ones are added to make sure data is shuffled around the cluster. This is largely a solved problem with projects like Ceph, Longhorn, or other distributed storage systems.

S3 uses delete markers​

Some object storage systems like S3 expose platform internals to make soft deletion work. In S3 deleting an object creates a delete marker (tombstone). A delete marker is an explicit marker that the object is deleted and should not be returned in normal operation. Here’s what that looks like in practice:

  ┌───────────────────┐  ┌───────────────────┐  ┌───────────────────┐ ◀── DeleteObject
  │ v1                │  │ v2                │  │ v3        current │
  │ report.pdf        │  │ report.pdf        │  │ report.pdf        │
  │ …198086           │  │ …198088           │  │ delete marker     │
  └─────────┬─────────┘  └─────────┬─────────┘  └───────────────────┘
            │                      │              no data
            ▼                      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ sea of data                                                       │
  │    ┌──────────┐         ┌──────────┐                              │
  │    │ v1 bytes │         │ v2 bytes │                              │
  │    └──────────┘         └──────────┘                              │
  └───────────────────────────────────────────────────────────────────┘
 
  // the bytes stay. only the newest record says the object is gone.

I don’t know how I feel about this flow. Based on reading between the lines in the delete marker documentation it really feels like this is a leaked internal implementation detail of how S3’s eventually consistent database works instead of a full fledged feature of the storage system. If I had to choose between leaking internal database details in the API and implementing a higher level API for something complicated like soft deletion, I’d want to implement the higher level API.

Tigris’ soft deletes are external references​

Let’s rethink what soft deletes really are. What if they were like the Recycle Bin in Windows?

Soft-deletes are external references to buckets or objects that live in a different namespace from normal buckets or objects. We implemented them as external references instead of tombstones because this is effectively moving object metadata to the recycle bin. Tombstones mark the data as not being there, but soft-delete markers are a copy of the data that was there. This makes it easy to put the object back in place if you deleted it by mistake.

Garbage collection roots​

One way to think about objects and buckets is that they are garbage collection roots for points in the endless sea of data. Any data in the sea without a root anchoring it down is eligible to be deleted. Uploading multiple versions of an object with a forkable bucket creates multiple metadata entries at their different timestamped version numbers. You can then fork a bucket from any one of those timestamps to see what the bucket was like at that point:

  every write appends a new version entry
 
  v1              v2              ╎   v3              v4
  ┌────────────┐  ┌────────────┐  ╎   ┌────────────┐  ┌────────────┐
  │ …198086    │  │ …431907    │  ╎   │ …764522    │  │ …055310    │
  │ 4.1 MB     │  │ 4.3 MB     │  ╎   │ 4.4 MB     │  │ 5.0 MB     │
  └─────┬──────┘  └─────┬──────┘  ╎   └─────┬──────┘  └─────┬──────┘
  ──────┴───────────────┴─────────╎─────────┴───────────────┴───────▶
  earlier                         ╎ fork point                later
  the fork inherits these         ╎   written later — the fork never sees them
                                  ▼
                      ┌──────────────────────────────┐
                      │ uploads/report.pdf           │
                      │ current version v2           │
                      │ as of 1775929812004431907    │
                      └──────────────────────────────┘
 
  // appending metadata instead of overwriting keeps any past instant addressable.

This would solve the soft-delete problem, but our existing database schema using FoundationDB requires us to enable forking and snapshots at bucket creation time. In essence, we need something that’s halfway between what we have (each bucket being a globally mutable namespace) and the bucket forking land of every action being appending metadata onto the end.

To do that, we basically implemented most of that appending metadata on the end trick but to a different place: the soft deletion corner. When you enable soft-deletion and delete an object, its metadata gets moved to the trashcan so you can pluck it back into place:

  main table · live keyspace                soft-delete keyspace · newest first
  ┌────────────────────────────┐            ┌──────────────────────────────────┐
  │ uploads/report.pdf         │            │ uploads/report.pdf   3rd delete  │
  │ live metadata record       │───────▶    │ deleted …768707198086            │
  │ in ListObjectsV2 output    │◀╌╌╌╌╌╌     └──────────────────────────────────┘
                                            ┌──────────────────────────────────┐
                                            │ uploads/report.pdf   2nd delete  │
                                            │ deleted …412888100731            │
  ┌────────────────────────────┐            └──────────────────────────────────┘
  │ uploads/notes.md           │            ┌──────────────────────────────────┐
  │ untouched by the delete    │            │ uploads/report.pdf   1st delete  │
  └────────────────────────────┘            │ deleted …104233715492            │
                                            └──────────────────────────────────┘
 
  ───────▶  DeleteObject moves the record out — one entry per delete
  ◀╌╌╌╌╌╌   RestoreObject moves the same metadata back
 
  // the bin entry is a copy of the metadata, not a marker. restoring is a move.

It’s the same basic idea as the recycle bin on your desktop. Any buckets or objects left in the recycle bin for long enough become eligible to be deleted, which then makes the backend go and securely erase things. Effectively, any bits of metadata in the soft deletion corner are still considered garbage collection roots, they’re just not shown when you do a normal ListObjectsV2 call.

Distributed systems are fun*​

The real fun comes into play when you remember that Tigris has a globally replicated active-active database where any region can change any object at any time. Most of the time things work out and objects are replicated without too much strife. The annoying part comes when two events are ordered weirdly. Imagine a scenario where one agent in one datacentre deletes an object after another agent in another datacentre:

  ORD (Chicago)                               IAD (Ashburn)
  ┌──────────────────────────┐
  │ PutObject  · agent A     │
  │ uploads/report.pdf       │  ──── replicates PUT ────▶
  │ t = …768707198086        │
  └──────────────────────────┘
 
                                              ┌──────────────────────────┐
                                              │ DeleteObject  · agent B  │
  ◀── replicates DELETE ──                    │ uploads/report.pdf       │
                                              │ t = …768984210773        │
                                              └──────────────────────────┘
 
  ORD applies   PUT  ▸  DELETE    deleted, as expected
  IAD applies   DELETE  ▸  PUT    the put looks brand new
 
  // two writers, one key. the regions disagree about whether it exists.

How would this replicate out? Well for one each change is timestamped by when it’s done in terms of Unix nanoseconds, so the replication messages kinda look like this:

  produced first                          produced 277 ms later
  ┌──────────────────────────────────┐    ┌──────────────────────────────────┐
  │ uploads/report.pdf               │    │ uploads/report.pdf               │
  │ op: PUT                          │    │ op: DELETE                       │
  │ LastModified  1775929768707198086│    │ LastModified  1775929768984210773│
  │ block 0x3f2ac701 · origin ORD    │    │ block 0x3f2ac701 · origin IAD    │
  └──────────────────────────────────┘    └──────────────────────────────────┘
 
  last write wins
  984210773   is greater than  707198086
 
  // the delete carries the newer LastModified, so the delete survives

This means that in theory, a user could DELETE an object before an update is processed by another region, and that would make the regions disagree about if the object exists or not. This is a horrible state to be in and usually requires support intervention or to recreate/re-delete the object.

The root cause boils down to deleting objects actually deleting metadata from the database doesn’t scale past a single region. Updates to metadata include the entire metadata object, so if you delete it locally and a new version is pushed remotely, the object will gain the remote state.

We don’t want users to have to deal with that, so we added the concept of anti-resurrection to Tigris. Any write to an object must prove it is newer than the deletion.

  ORD · agent PutObject                     IAD · user DeleteObject
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ uploads/report.pdf       │              │ uploads/report.pdf       │
  │ new version · v2         │◀─────────────│ row deleted · marker kept│
  │ t = 15                   │   delete     │ tombstone t = 25         │
  └────────────┬─────────────┘              └────────────┬─────────────┘
               └────────────────────┬────────────────────┘
                                    ▼
    ┌────────────────────────────────────────────────────────────┐
    │ at IAD: is the write newer than the marker?                │
    │ write t = 15  ·  marker t = 25  ·  is 15 > 25 ?            │
    │ no — not strictly newer, so the write is dropped           │
    │ equal timestamps lose too · the guard runs for every bucket│
    └────────────────────────────────────────────────────────────┘
 
  // without the marker, an empty slot looks exactly like a key that never existed.

In this circumstance, a user sent a DeleteObject request to the IAD datacentre at time 25, but an agent sent a new version of the object with PutObject to the ORD datacentre at time 15. The user’s delete is newer than the agent’s put, so the new version is rejected and the delete gets sent back to ORD.

Using soft deletes​

Tigris extends the S3 API by having users add headers to their requests. For example, to create a bucket with soft deletion enabled:

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/tigrisdata/storage-go"
)

client, err := storage.New(ctx,
	storage.WithGlobalEndpoint(),
	storage.WithAccessKeypair(
		os.Getenv("TIGRIS_STORAGE_ACCESS_KEY_ID"),
		os.Getenv("TIGRIS_STORAGE_SECRET_ACCESS_KEY"),
	),
)

_, err := client.CreateBucketWithSoftDelete(ctx, &storage.CreateBucketWithSoftDeleteInput{
	CreateBucketInput: &s3.CreateBucketInput{Bucket: aws.String("my-bucket")},
	RetentionDays:     30, // 0 uses the 7-day default
})

Or to list soft-deleted objects:

out, err := client.ListSoftDeletedObjects(ctx, &storage.ListSoftDeletedObjectsInput{
	Bucket: "my-bucket",
	Prefix: "uploads/",
})
if err != nil {
	return err
}
for _, o := range out.Objects {
	log.Printf("%s v=%s %d bytes deleted=%s", o.Key, o.VersionID, o.Size, o.LastModified)
}

Or to permanently delete one soft-deleted version:

_, err := client.PermanentlyDeleteObject(ctx,
	"my-bucket", "uploads/report.pdf", "1775929768707198086")

When you have a soft-delete enabled bucket, you can also forcibly delete an entire bucket:

_, err := client.ForceDeleteBucket(ctx, &s3.DeleteBucketInput{
	Bucket: aws.String("my-bucket"),
})

And then bring it back from the dead:

trash, err := client.ListSoftDeletedBuckets(ctx, nil)
if err != nil {
	return err
}
for _, b := range trash.Buckets {
	log.Printf("%s (%d day retention)", b.Name, b.RetentionDays)
	if _, err := client.RestoreBucket(ctx, &storage.RestoreBucketInput{Bucket: b.Name}); err != nil {
		return err
	}
}

Now what?​

Object storage entered our stacks as an unlimited FTP server we all used for backups. A distressing amount of the world’s most important data lives in object storage buckets because it’s the best place to put it. This is why having an “undo” button matters, it’s what makes it safe to trust your backups in the cloud. To err is human, and mistakes are a “when” to plan for, not an “if” that you hopefully never have happen. The blast radius of one overly wide --recursive flag is measured in years of people’s lives.

One of the biggest usecases that comes to mind is ransomware prevention. Imagine a case where an attacker downloads everything in your bucket, deletes it, and asks for a ransom to send you the files back. With Tigris, soft deletes means that the ransom can be ignored, you can un-delete your data, and be on your merry way with incident response. The other big usecase is for agents, where they somehow get the idea that deleting production data is the right way to solve a problem. Both cases mean you need a quick and fast way to go back to before things went wrong.

If you want true isolation instead of recovery, that’s why we have bucket forking. Bucket forking needs to be enabled before a bucket is created, but you can enable soft deletion on any bucket in the dashboard whenever you want.

Every storage system is going to make you choose between ones that hide how the platform works and ones that expose the gorey internals to users. I think that hiding the internals and exposing the high level operations built on top of them is the right way to go, if only because the higher level operations are much easier to make safe in our globally distributed future.

Tigris is a drop-in replacement for AWS S3 (or GCS, anything S3API compatible). As such, we need to be fully compatible with both the mechanisms and semantics of S3 including the SigV4 authentication protocol. This is the lingua franca of authentication in the object storage landscape; even Google Cloud Storage has a way to enable SigV4 support so you can use existing applications against its object storage service.

At first I thought that SigV4 was fairly simple. Clients sign requests, servers do the same work and make sure the result matches. The main sticking point is that the cryptography involved is symmetric cryptography, the kind where both parties need to have the same secrets. This makes some scaling issues weird, but we'll get into that in the future.

SigV4 in a shellnut​

At a high level when a client signs a request with SigV4 you get an access key ID and secret access key. The access key ID is functionally a username and the secret access key is functionally a password. Admins can identify keypairs by the access key ID (without special training or tools) and services use the owner of the access key or policies delegated to that access key to determine what actions that client may take.

SigV4 uses HMAC (hash-based Message Authentication Code) and SHA-256 (SHA-2 with a 256 bit hash width) to do authentication by creating salted hashes based on request metadata.

In order to send a SigV4 request, clients take the outgoing request, reduce it to a canonicalized form, and sign it with a symmetric key derived from the secret access key, the current date, region of the service, and service name, kinda like this Go code:

func HMAC(key, data []byte) []byte {
	h := hmac.New(sha256.New, key)
	h.Write(data)
	return h.Sum(nil)
}

var (
	kDate    = HMAC("AWS4"+secretAccessKey, nowDate)
	kRegion  = HMAC(kDate, region)
	kService = HMAC(kRegion, service)
	kSigning = HMAC(kService, "aws4_request")
)

As an example, let's see what a signed GET request to a HTTP debugging endpoint looks like on the wire with and without the signature:

$ curl http://localhost:3000 -v

GET /
User-Agent: curl/8.7.1
Accept: */*

And when you add the signature with --aws-sigv4:

$ curl \
  --user tid_YOISC719YLXSONFU:tsec_DiYqeH8t0IKjKUKfqhzTsqrCCUl9Wm0m+6MXNhhi1fU \
  --aws-sigv4 aws:amz:auto:s3 \
  -v \
  http://localhost:3000

GET /
User-Agent: curl/8.7.1
Accept: */*
Authorization:
  AWS4-HMAC-SHA256
  Credential=tid_YOISC719YLXSONFU/20260720/auto/s3/aws4_request,
  SignedHeaders=host;x-amz-date,
  Signature=879bcdd43749cfc9782b876d9ceb3ff153d79ab1482290cca7ab915bb7f8785d
X-Amz-Date: 20260720T153748Z

Breaking it down we have two extra headers in the request:

• Authorization: The fixed string AWS4-HMAC-SHA256 to signal to the server which authentication mechanism is in use. The rest of the string is information about the request signature so the server can properly canonicalize the request.
• X-Amz-Date: The date and time (UTC) of the request so the server knows when the request was signed. Servers will use this request date in order to reject old requests to prevent replay attacks.

Request canonicalization and signing​

On the wire, HTTP/1.1 requests look kinda like this:

GET /api/list?page=0&count=30
User-Agent: curl/8.7.1
Accept: */*
Host: myawesomesite.example

However the headers could be sent in any order, and changing the order of request headers doesn't result in different requests. Additionally any query string parameters could be formatted in any way a client (or server) could imagine, including the use of semicolons to separate values. All attempts to canonicalise HTTP requests MUST deal with this ambiguity and define their own rules.

SigV4 canonical requests are made up of a few parts:

• The HTTP method (GET, PUT, POST, DELETE, etc.)
• The URI path of the request (/api/list, etc.)
• The sorted canonical query string (you must exactly match the server-side canonicalization logic)
• The signed headers terminated with two newlines
• The sorted list of signed headers joined by semicolons
• The SHA256 checksum of the request body

For that example /api/list request, the canonical form would look like this:

GET
/api/list
count=30&page=0
host:myawesomesite.example
x-amz-date:20260715T204745Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

As the request has no body, the empty sha256 checksum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is put as the body checksum.

I'm reading the blogpost at <link> and Xe mentioned AWS SigV4's
STREAMING-AWS4-HMAC-SHA256-PAYLOAD method. I would like to learn more about how
this works. Please research how this works and give me code and request body
samples.

To make the signature, you take the sha256 checksum of the canonical request and then HMAC it against that derived signing key:

finalRequestSignature := HMAC(kSigning, reqSig.Bytes())

And construct the Authorization header based on your access key ID, service region, and service name:

req.Header.Set("Authorization", fmt.Sprintf(
	"AWS4-HMAC-SHA256 Credential=%s/%s/%s/%s/aws4_request, SignedHeaders=%s, Signature=%x",
	accessKeyID, nowDate, region, service,
	strings.Join(signedHeaders, ";"),
	finalRequestSignature,
))

What about SigV4a?​

AWS has made an extension to SigV4 that uses asymmetric cryptography called SigV4a (the "a" means asymmetric). Instead of using symmetric cryptography on both the client and server in ways that means the server needs to either know the client's secret access key (or a value derived from the secret access key), SigV4a uses key derivation functions to derive a cryptographic keypair. Servers authenticating requests fetch the public key from IAM. Only the client and IAM know what the private key is, and that private key is what signs outgoing requests.

I'd love to use SigV4a more because it makes adding additional services to the mix (such as a git service) a lot safer as you can have those additional services exist in different trust domains than the core product. This is the core of how microservices end up happening. However, it's not super widely used even within AWS. The only SigV4a use I can find in Amazon is S3 Express Zones, however they may end up using it in other services I'm just not aware of.

When I did my own experimentation with SigV4a (where I was implementing my own IAM server so that I really understood this all at a low level), I had to copy a lot of internal AWS SDK code into my repo in order to get it working.

I'll talk about SigV4a some more another time.

Replay attacks and you: a young coder's illustrated primer​

One of the weaknesses of using signatures for API authentication like this is the problem of replay attacks. When you make a naïve signature of a value, there's no real way to tell when that signature was created. If you sign a request to create a compute instance at time instance t0, it's still technically valid at any other time instance tN. This is why the canonical form of SigV4 requests includes the current date and time:

Authorization: [...] SignedHeaders=host;x-amz-date, [...]
X-Amz-Date: 20260715T205432Z

This means that the request was signed on July 15, 2026 at 20:54:32 UTC. Time changes constantly (at least at the rate of one second per second!) and the client has to have a working clock in order for TLS to work. Servers can trivially read the contents of X-Amz-Date and reject old requests. This means that you don't need to add or store nonce (number used once) values with each request because that doesn't scale.

As a side effect of this, you need to set a temporal skew window for validating requests. This window needs to be generous enough to accommodate slow clients, sloppy timekeeping on the client side, highly latent clients, leap seconds, or other exceptional temporal phenomena. In general time synchronization is a surprisingly hard problem, so it's best to just be tolerant of clients in order to make things more robust in practice. AWS uses a temporal skew window of 15 minutes for validating requests. I'm going to use a window of 5 minutes for my API because 300 seconds is a nice round number and I don't have to deal with the same amount of legacy code that AWS does.

How TAG changes the game​

So all of this SigV4 business had been working really well for Tigris. Then we worked with a few customers who needed a local cache to fully saturate their hungry GPUs. To be fair, Tigris is plenty fast, but the real thing that kills AI training is latency and something that runs locally will always be faster than the cloud.

In order to provide that sweet middle spot between making everything rely on the cloud and having everything local, we made TAG, the Tigris Acceleration Gateway. This effectively gives you most of a Tigris region in your own infrastructure.

When you connect to TAG, your code uses its existing access keypairs, buckets, and code. You point your code to TAG, you point TAG to Tigris, and then everything is cached for you. But how does TAG authenticate with your code? TAG doesn't have access to all your existing API keys (and to be honest it shouldn't), but it's still able to authenticate them with SigV4 authentication.

TAG and the IAM server both implement a signing key proxying feature that lets a client and TAG both prove their identity to Tigris. Once that proof is sent, then TAG gets the intermediate derived signing key and uses that for locally validating requests, kinda like this:

 client                    TAG                   Tigris
    │                       │                       │
    │ ListBuckets (signed)  │                       │
    ├──────────────────────▶│                       │
    │                       │ ListBuckets (proxied) │
    │                       ├──────────────────────▶│
    │                       │                       │   TAG adds signed proxy headers;
    │                       │                       │   Tigris checks both signatures:
    │                       │                       │   2xx, derived keys returned
    │                       │                       │
    │                       │ ListBucketsResponse   │
    │                       │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
    │                       │                       │   + derived signing keys,
    │                       │                       │   AES-encrypted for TAG only;
    │                       │                       │   TAG decrypts and caches them
    │                       │                       │
    │ ListBucketsResponse   │                       │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤                       │
    ·                       ·                       ·
    │                       │                       │   later: the next request
    │ ListBuckets (signed)  │                       │
    ├──────────────────────▶│                       │
    │                       │                       │   TAG verifies the signature
    │                       │                       │   locally with the cached key:
    │                       │                       │   no round trip to the cloud
    │                       │                       │
    │ 200 OK                │                       │
    │◀─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤                       │

The actual implementation in TAG involves some derived AES logic so that the derived signing keys are very much limited to the client that requested it (namely: the AES key is the SHA256 encoded form of the proxy secret access key). One of the weird parts is that the canonical form of the proxied requests differ from the normal SigV4 canonicalization process, namely looking like this:

tag.default.svc.cluster.local # Host header from the client
1784577479                    # Unix timestamp of the request (X-Tigris-Proxy-Timestamp)
GET                           # HTTP method of the client
/                             # HTTP path of the client

This is signed using the same SigV4 signature process as before but added differently to the request:

• X-Tigris-Forwarded-Host: the HTTP Host of client requests (EG: tag.default.svc.cluster.local)
• X-Tigris-Proxy-Access-Key: the Tigris keypair used to authenticate TAG itself (must be in the same organization as the client)
• X-Tigris-Proxy-Timestamp: the time of the request in unix timestamp format
• X-Tigris-Proxy-Signature: the hex output of signing the canonical form of the request against TAG's secret access key

And then TAG reads the response from Tigris, caches those derived signing keys, and then uses those in the standard SigV4 process to authenticate clients: no round trip to the cloud required.

I was wrong about the simple part​

The happy path is exactly what I thought it was. Reduce a request to a canonical form, run four HMACs, compare the result. That part fits in an afternoon.

Everything expensive lives in the questions around it. Which bytes count as the request? Whose clock decides that a signature is still good? Who gets to hold the key that proves any of it? Each question has an obvious answer, and each obvious answer is wrong in some specific way you only find by implementing it.

That last question is the one that surprised me. I read symmetric cryptography as a hard limit: if the verifier needs your secret, the verifier has to be Tigris. It isn't. SigV4 derives its signing key through a chain of four HMACs, each one scoped tighter than the last: date, then region, then service. Those intermediate values can travel without the secret behind them. TAG rides that. The key it holds stops working when the UTC date rolls over. It covers one region and one service. You can't walk it backwards into a secret access key.

We also didn't write any of this, which is its own kind of relief. SigV4 is old, widely deployed, and hammered on by every S3 client in existence. Any compatibility bugs here are ours. The protocol's bugs are everyone's.

The place a protocol bends is usually some intermediate value that somebody already designed to be thrown away.

The era of manual installation scripts for Blessed Frameworks™ is over. Developers should spend their time shipping things, not fighting their environment into getting your product integrated into their development flow. It’s the future, the environment should just configure itself for you. We’ve shipped this with tigris init --agent and we think you should do this too.

Keep developers in flow​

Right now your onboarding flow likely funnels developers into two categories: people that use Blessed Frameworks™ and everyone else. This is great for the developers that do use those Blessed Frameworks™, but developers like me get knocked out of flow and suddenly have to read the docs to figure out what I need to do. This friction results in customer churn from people you never hear from again. Onboarding is one of the highest risk phases in the customer journey because the easiest time to walk out the door is right after you just walked in it.

Nobody chooses that outcome on purpose. It’s the residue of the reasonable decision around picking something you already know well enough to confidently implement an unattended setup process. This could be a 5 minute quickstart, a template repo, or the fabled one-click deploy button. Everyone outside that blessed path gets the API reference and wished good luck. That’s been me, closing the tab and reading reference docs to try and figure out where to start the hard way.

With the rise of modern AI agent tools, we don’t have to do this anymore. We can use agents to help people get started in ways that help keep developers in the development flow. The agent in your users’ editor already knows their stack, their conventions, and where their configuration lives. It’s a better onboarding engineer for any given project than all of us will ever be.

Setup scripts are sunk-cost fallacies​

A setup script is an easy to describe and easy to validate problem, but the real devil is in the details of actually implementing it in a way that’s portable enough to be reliable. This is the sweet spot for AI agents. AI agents have all of the book smarts in every stack, language, framework, and environment. This massive volume of data can help customize the getting started flow without you having to do anything.

Consider what the portable version of a setup script actually has to do. It has to identify what language the project is in, what tool you use to manage dependencies, and more importantly decide which dependency manager is more authoritative in a project that uses many of them at the same time. Your script also has to be right about knowing Linux, macOS, Windows, devcontainers, CI, and somebody’s virtual shell on an iPad. It’s a program that has to be correct about a machine it’s never seen and any environment it fails in becomes another support ticket.

The agent needs none of that. It doesn’t need to be told which package manager the project uses because worst case it can explore and figure out how you build and deploy your software.

We shipped tigris init --agent​

None of this is hypothetical. We just shipped tigris init to help you wire Tigris into whatever coding agent you use, and tigris init --agent prints a setup recipe that your agent can follow along with you.

Our agent setup doesn’t do anything special, it’s just returns a prompt that’s formatted like how a user would ask to get started:

Help the user set up their project with Tigris:

1. If `tigris` isn't on $PATH (`command -v tigris`), ask permission, then `npm install -g @tigrisdata/cli --ignore-scripts`.
2. If `tigris` is on $PATH, ensure it's latest version using `tigris update`.
3. Check if the user is already authenticated using `tigris whoami`. If not, authenticate using `tigris login oauth`.
4. Run `tigris orgs list --format json`. If there's more than one org, ask which to use, then make it active with `tigris orgs select <org>` so the following commands target it.
5. Infer the project name from metadata (package.json, go.mod, pyproject.toml, ...). Check `tigris buckets list --format json` first; if the bucket is absent, `tigris buckets create <name> --format json`. Ask the user whenever anything is ambiguous.
6. Create the access key into a private temp file (overwrite, never append), then print only the ID:
   `umask 077; tmpfile=$(mktemp); tigris access-keys create <username>-<project>-devel --format json > "$tmpfile" && jq -r '.id' < "$tmpfile"`
7. Grant bucket access: `tigris access-keys assign <id> --bucket <bucket> --role Editor --format json`.
8. Detect whether the code uses the Tigris SDK or the AWS SDK, then have a small script append the right vars to .env, reading `.id`/`.secret` from the temp file. Do NOT read the secret into your context — append it via script. Delete the temp file when done: `rm -f "$tmpfile"`.

   Tigris SDK (@tigrisdata/storage, storage-go):
     TIGRIS_STORAGE_ACCESS_KEY_ID     = .id
     TIGRIS_STORAGE_SECRET_ACCESS_KEY = .secret   (secret)
     TIGRIS_STORAGE_BUCKET            = <bucket>

   AWS SDK:
     AWS_ACCESS_KEY_ID       = .id
     AWS_SECRET_ACCESS_KEY   = .secret                   (secret)
     AWS_ENDPOINT_URL_S3     = https://t3.storage.dev    (required)
     AWS_ENDPOINT_URL_IAM    = https://iam.storage.dev   (required)
     AWS_REGION              = auto                      (required)

9. Congratulate the user and point them to:
   - JS:    https://www.tigrisdata.com/docs/sdks/tigris/
   - Go:    https://pkg.go.dev/github.com/tigrisdata/storage-go
   - Docs:  https://www.tigrisdata.com/docs/
   - Discord: https://community.tigrisdata.com/
   - Skills: https://www.tigrisdata.com/docs/skills/

   Suggest adding to their agent config:
     > ## Tigris object storage
     > This project uses Tigris. For any Tigris questions, consult https://www.tigrisdata.com/llms.txt before acting; look it up rather than relying on memory.

The entire onboarding flow becomes documentation that people can follow by hand if they need to. There’s no framework in here, no template, and no operating system assumptions. The important parts are that we aggressively delegate any decisions to the user instead of having the AI agent guess. This leaves the user in charge of their environment even though they got assistance from their AI agent.

One of the key things we did here is ship this prompt in the Tigris CLI instead of a copy->paste->Claude flow in the docs. This seems weird at first, but this prevents the locally installed Tigris CLI from lagging behind the more up to date content in the docs. If you have a user with an older version of the CLI, telling them to use something only on the new version of the CLI isn't helpful; it will just confuse the agent and get it much more likely to go down a debugging rabbit hole. Coupling the prompt with the CLI makes things much more reliable in practice.

How do I prompt good?​

Wording is where this approach lives or dies, and it’s a different craft from writing docs for people. Four things matter more than the rest.

Ask permission before taking action​

Agents can stop and ask, and a prompt that tells them when to do it turns from an install flow from something done to your user into something being done with them. This is also the part that keeps your flow from tripping over an agent’s own safety behaviour because a request for consent to mint credentials looks nothing like an agent quietly minting credentials out of its own volition.

Prefer heuristics instead of mandates​

Notice that my example said to infer the project name by exploring with a few examples instead of specifying exactly where to find it. Agents know the project better than your setup script ever will, so give them the goal and let them use what they can see.

Reserve instructions for the things it must not guess such as the account, organization, environment, permission scope, and credential storage. Instructions should be policy, not details about the mechanism.

Keep secrets out of the context window​

Assume anything in the model context window is one breach, log pipeline, or screenshot away from being public information. When your agents provision credentials, write them to a file and operate on that file instead of reading them into the context window. This also reduces the risk of agents on lesser models hallucinating your credentials.

Phrases things positively​

Models follow “ask before creating access keys” better than “don’t create access keys silently”. Models act unreliably on prohibitions when it’s buried far upstream of the action and competing with other instruction sources. Make sure to use active voice when writing your instructions too, it’s good for people and agents.

Something to keep in mind is that positive phrasing is a reliability technique, not a security mechanism. If a step must never happen, the thing that should stop it is a confirmation prompt in your CLI or a scope on the credential, not magic adverbs.

Agents can’t manage their own configuration​

One lesson I learned the hard way is that generally speaking agents can’t manage their own configuration seamlessly. Tigris ships an agent plugin and we want to have agents set it up by themselves, giving agents object storage superpowers out of the gate. I wrote that, tested it, everything installed, and then the agent behaved like nothing in particular changed. All the skills were installed but none of them were usable.

After thinking about it a bit, I realized that this makes sense. Skills are effectively executable code, and the agent has no way of knowing if they're added for legitimate reasons (integration with Tigris) or illegitimate reasons (cryptocurrency mining, making your agent part of a residential proxy botnet, etc).

This is why tigris init has two doors: human use and AI agent use. The wizard is for humans in the shell where installing things to an agent is safe because the agent hasn’t started yet. The --agent recipe is in plain text because text is the only thing that works reliably from inside an agent session.

Whatever your flow installs into agents belongs outside of the agent.

Agent-native is the new baseline​

In the long run I don’t see shell installers and golden paths for Blessed Frameworks™ disappearing, I think that they’re just going to stop being the only way to onboard. The onboarding surface is moving to a couple of kilobytes of text that your CLI prints and someone else’s agent executes, which means the job changes from writing your instructions as code to writing your instructions for a collaborator who already knows the codebase better than you ever could.

For anyone building developer tools: what parts of your setup flow are easy to describe, easy to verify, but really annoying to implement? This is the hole that agents fit into the stack. Put these directions in your tool and then send it off to the races. Be sure to test against open weights models such as Kimi K3 or Qwen 3.5 36b-A3b to be sure you haven't accidentally optimized for a single model from a single provider.

As time goes on I can see this getting more and more useful. Models circa last year had trouble with two or three step processes and today’s models can easily handle 9 step flows without issue. I don’t know how things are going to develop in the future, but I do know that the setup flow was written for machines before we admitted it and making it with machines in mind is only going to make things easier in the future.

If you self-host ClickHouse, you probably run it on Hetzner, OVH, or your own hardware. That's the whole appeal: ClickHouse is fast on cheap metal, and hyperscaler compute prices are hard to justify when your queries scan terabytes for fun.

But every guide to running ClickHouse in production assumes there's an S3 bucket nearby. Backups go to object storage. Partitions older than 30 days get tiered off to object storage. If you're not on AWS, there is no bucket nearby, so most teams point everything at S3 anyway and eat the bandwidth charges. AWS bills $0.09/GB to read your own data back.

What that actually costs​

Old data doesn't stay untouched. Someone digs through it during an incident, an auditor asks for it, or a backfill re-reads it. If an incident has you scanning 50 TB of old partitions, that's around $4,500 in bandwidth before you've learned anything. A full restore of a 500 TB backup costs about $45,000 at list price.

The worse problem is the one that doesn't show up on any bill: because restores cost real money, nobody tests their backups. You find out whether they work during the outage. That's the worst possible moment to learn something new about your infrastructure.

The fix is an endpoint URL​

Tigris speaks S3 and doesn't charge egress. Nothing about your setup has to change, because these are the same stock ClickHouse features you'd use with any bucket. Backups are the five-minute version:

BACKUP TABLE logs
TO S3(
  'https://t3.storage.dev/ch-cold/backups/logs-2026-07-16',
  'tid_your_access_key',
  'tsec_your_secret_key'
);

That endpoint is the entire integration. Incremental backups and restore rehearsals work the same way, and our ClickHouse guide walks through both. If you have retention-heavy tables, add a tiered storage policy too. Recent partitions stay on NVMe, and everything older moves to Tigris on its own:

TTL timestamp + INTERVAL 30 DAY TO VOLUME 'cold'
SETTINGS storage_policy = 'tiered'

Your queries don't change. A dashboard query that filters to the last few days gets pruned down to the hot partitions and never leaves local disk. Queries that reach further back read from Tigris at object storage speed, and a cache disk soaks up repeat reads of the same old partitions. The full disk and policy config is in the ClickHouse guide.

To be clear about what this is not​

Your hot data stays on NVMe. Local disk answers in microseconds and object storage answers in milliseconds, and that difference is most of why ClickHouse feels fast. Tigris is for your cold tier and your backups, not your primary storage.

One more thing: don't try zero-copy replication over S3 in open-source ClickHouse, tempting as it looks. The design shares one copy of the data between replicas, but each replica keeps its own local metadata about which blobs it uses, with reference counts coordinated through Keeper. That bookkeeping can race when one replica runs a merge while another runs a mutation. A replica can delete blobs its siblings still need, or it can strand orphaned blobs in your bucket forever. That's why the feature has been disabled by default since 22.8 and marked not production-ready. The clearest signal is what ClickHouse did about it. Rather than fix the bookkeeping, they built a new engine with shared metadata, called SharedMergeTree, and kept it in their cloud. Use Tigris for your backups and cold tier, and don't ask replicas to share a disk.

What changes when reads are free​

A lot changes. You can restore a backup just to prove it works, and you can do it monthly, because it costs nothing. Backfills and audits stop being budget conversations. The restore, the number that scared you most, drops to zero.

Your data was never the expensive part. Getting it back was. Fix that, and it's just storage. The ClickHouse guide has everything you need to set it up, from the backup command to the full tiering config.

A presigned URL is a replay attack you did on purpose.

Replayable auth tokens are the textbook way to create vulnerable systems, but Tigris ships them as a first-class feature with presigned URLs and so does every other object storage system on the planet. However this isn’t an oversight because presigned URLs turn a weakness into a feature.

Replay attacks are a real problem and the classic fix is miserable​

When you authenticate a request with Amazon’s SigV4 protocol for Tigris, your client boils down the request to a canonical form: a SHA256 hash of the request’s method, path, query parameters, signed headers and a SHA256 hash of the payload. It runs the result of that through HMAC with a signing key derived from your secret access key. Nothing secret ever crosses the wire. The server derives the same key as the client, does the same canonical form transformation, and compares the result.

Being able to make a valid signature proves that the request came from someone holding the secret access key, but it proves nothing about when that request was made. A signature that was made a year ago would still be valid today or any other time you send it, so in theory an attacker could warehouse your signed requests only to replay them en masse later. Imagine sitting on a pile of signed “create EC2 instance” calls only to spam them all out at a later date. You would be a twirling moustache villain able to spawn dozens of servers at a moment’s notice.

Traditionally the fix is to bake a nonce (number used once) into the signature (sorry to any British readers in the audience). This makes every signature differ because that nonce differs.

However with great power comes great responsibility and making sure that something used once is only used once is a surprisingly hard distributed systems problem. You can’t verify that something is only used once locally. Say you store them all for a 15 minute smear window at a low request rate like 10,000 Bq. That’s 9 million live nonces, and every frontend node needs to have a consistent view of the whole set as it churns.

You have made your fast authentication check slow from having to ensure things are only used once.

What you want instead is something that changes constantly without coordination and invalidates those old signatures for free. For an added bonus you want this to also be in the standard library of every programming language.

Sign the clock​

There’s exactly one value that changes constantly, (mostly) monotonically, and is already actively coordinated across all elements of the stack: the clock. Your OS already keeps time in sync with the public NTP pool (or a private NTP pool if you are cool enough to have radioactive PCI cards laying around). Without an accurate view of time you can’t make TLS connections, which means you can’t make API calls to Tigris at all, so the auth layer gets to assume a working clock exists.

SigV4 signs the current time into the request. If an attacker gets their greasy hacker paws on a signature, they have about 15 minutes to use it before it becomes a digital paperweight. If time is an input to the signature and the time changes enough to invalidate the signature, the signature is null and void. Sure in theory a sufficiently funded attacker could create a black hole in your datacentre and disrupt temporal flow, but at that point the planet is probably toast which makes the attack profile moot. Commit mass object storage fraud with this one neat trick! The department of temporal investigations will have hated it!

This makes your verification stay stateless. Everything gets checked against the system clock the server already needs and you can give clients a 15 minute signature smear window as a grace period for old or delayed clients (exponential backoff is a good thing and Tigris will reward you for doing it).

Of course the real thing keeping the signatures safe on the wire is TLS (HTTPS). If that is broken we have bigger problems and object storage fraud is the least of our problems.

Time is the only nonce you need because both sides already agree on it anyways.

Some thorns have roses​

Presigned URLs take the replay tolerance that SigV4 spends all this effort nerfing and then buffs it into the feature. The entire auth dance gets flattened into URL parameters that any HTTP client can use, be it a browser, curl, Go’s net/http, or something you made by bit-banging HTTP over a socket. Here’s a real presigned URL I sundered into visibility:

https://xe-sophia-base.t3.tigrisfiles.io/moby-dick.txt
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=tid_ubYBNEYAmTciLVwszw_QrUXDmtcyQisryryGfxgznDsCnOvNqh/20260714/auto/s3/aws4_request
&X-Amz-Date=20260714T043308Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=0dcaf4972911527a7582ff36ea457e9760a8efccb6655a178685aaa281637a36

Here are the parts (forgive the AI looking listicle because this is genuinely the best way to format this):

• X-Amz-Algorithm: the signature scheme. Effectively always AWS4-HMAC-SHA256.
• X-Amz-Credential: the access key ID plus the credential scope — date, region, service, and the literal terminator aws4_request. The signing key is derived by chaining HMAC through exactly those parts, so a signature is only ever valid for that day, that region, that service.
• X-Amz-Date: the second the URL was born, in UTC.
• X-Amz-Expires: how many seconds it gets to live, chosen by the signer.
• X-Amz-SignedHeaders: which HTTP headers are folded into the signature. Usually just host, because you can't force whoever you hand a URL to into sending exotic headers.
• X-Amz-Signature: 64 hex characters of HMAC-SHA256 over the canonical request — the method, the path, every parameter above, the signed headers, and the payload hash. Change any of them and the math stops agreeing.

All of these are normally HTTP headers in standard SigV4 requests.

GET /moby-dick.txt HTTP/1.1
Host: xe-sophia-base.t3.tigrisfiles.io
X-Amz-Date: 20260714T043308Z
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256
Credential=tid_ubYBNEYAmTciLVwszw_QrUXDmtcyQisryryGfxgznDsCnOvNqh/20260714/auto/s3/aws4_request
SignedHeaders=host;x-amz-content-sha256;x-amz-date
Signature=0dcaf4972911527a7582ff36ea457e9760a8efccb6655a178685aaa281637a36

Note that this request is not a legal request, it’s an example to illustrate the point, here be dragons, etc etc etc.

It’s best to think about this presigned URL as a capability grant. Whoever holds it gets to make exactly one (1) kind of API call with one (1) HTTP method against one (1) object in one (1) bucket. They can do this as many times as they want until the presigned URL expires. The signature covers the method, the path, and the signed headers so a user can’t take a presigned request for GETting a copy of Moby Dick from a development environment and weaponize it into a way to delete everything in your production bucket.

Possession is authorization until the clock says no.

What it costs you​

Capability grants like this can have some sharp edges. There is no real way to revoke any individual presigned URL short of killing the access key it was signed with. When that key dies, everything it signed dies too. This includes any URLs you may have wanted. This cuts both ways and it kinda has to unless you make a new keypair per presigned request, which is probably out of scope.

Expiry has fine print too. A presigned request can live anywhere from one (1) second to one (1) week (seven (7) periods of twenty-four (24) hours).

There’s no limit to the number of times a client can use a presigned request. If you give a mouse permission to GET one cookie, they can GET that same cookie over and over. You end up having to pay for the GetObject calls in the end, so keep that in mind.

URLs also leak, but these URLs are born to die. Presigned URLs will end up in API responses, chat messages, GitHub comments, and your browser history. The tradeoff is acceptable because all the links self-destruct, but it’s a tradeoff you need to keep in mind when you design your services, not a panacea for access control.

Presigned URLs sound like a great way to prevent hotlinking. At some level they are (a few of my services use them as such), but what they actually do is put a lifetime on hotlinking. This makes things annoying enough that it usually gets people to stop.

The hole in the fence is the gate​

SigV4 makes a lot of API authentication challenges so much easier. It spent most of its innovation budget on making signatures die quickly because replay attacks are the classic way that signed requests go wrong. Presigned URLs looked at that property, shrugged, flipped it on its head, and made it into a feature.

The thing that looked like a problem becomes a fundamental construct to build your apps upon.

Networking exists to move data around. You can easily spin up your agents on any platform under the sun. But storage is how The Big Cloud™ gets you. When you move your operations to a new storage platform like Tigris you have to migrate your data to its new home.

Tigris makes it easy with bucket migration, but what if you want to move over everything all at once? The Tigris CLI has your back:

t3 migrate t3://world-domination-plans/agents/chatlogs

And your laptop will run the migration automatically, scanning over every object in your bucket and safely moving them over to Tigris. Here's how you set it up:

Bucket migration and you​

First, grab a Read-only keypair for your bucket in your old storage system. This will be the access key that Tigris uses to migrate your data. Use it in the tigris bucket set-migration command:

tigris buckets set-migration world-domination-plans \
  --bucket old-bucket-name \
  --endpoint https://old.provider.domain \
  --region region-such-as-yow \
  --access-key <access key id> \
  --secret-key <secret access key>

If you want the migration to be two-way (so new files added to your Tigris bucket get added to your old bucket just in case you don't want to continue your migration), make that keypair an Editor and pass the --write-through flag:

tigris buckets set-migration world-domination-plans \
  --bucket old-bucket-name \
  --endpoint https://old.provider.domain \
  --region region-such-as-yow \
  --access-key <access key id> \
  --secret-key <secret access key> \
  --write-through

Then you can run tigris bucket migrate:

tigris bucket migrate t3://world-domination-plans/

Kick back and relax! The Tigris CLI will keep on chugging until the job is done and all your data is safely on Tigris.

Caveats​

Keep in mind this has a few caveats. Namely that this migration isn't free. Running the tigris bucket migrate command issues a lot of ListObjectsV2 and HeadObjects calls, which will cause you to incur Class A request costs according to Tigris' pricing table.

This is also a very long-running operation, so if you run it on your laptop and go for a walk to the local cafejo, the migration will be suspended if your laptop is closed. Use something like a vibe coding laptop wedge to keep it running while you go touch grass. Worst case, spin up a small VPS / cloud server to run the command on.

If the migration does get interrupted, don't worry: the next time you run tigris bucket migrate it picks up where it left off instead of starting from scratch. No wasted work, no re-copying objects that already made it over.

We'll keep innovating for your agents​

At Tigris we believe that you(r agents) deserve a better experience with object storage. That's why we will keep innovating with bucket migration, forks/snapshots, bundles of objects, and more as we reinvent object storage based on how you wish it could work.

"Where does the agent run?" is a solved question.

It runs in a sandbox: a Daytona microVM that boots in milliseconds, executes whatever the model just wrote, and gets torn down without ceremony. Disposability is the whole value proposition: the sandbox is supposed to die.

It raises a question I didn't think to ask until one of my agents lost three days of accumulated context to a sandbox that did exactly what sandboxes are supposed to do: if the place the agent runs is built to be destroyed, where does the agent live? And if you're building an agent platform rather than an agent, the question doesn't come up once; it comes up for every agent in the fleet, on every run, concurrently.

An agent is a storage-shaped problem. You can swap the sandbox, the harness, and even the model without losing anything; the one part you can't swap is the state: the files, memory, and metadata that make Wednesday's agent the same agent as Tuesday's. That state is the agent, and it belongs in one namespace with one snapshot boundary, which is to say a bucket you can fork. Put it there and experiments stop threatening production: fork the world, try the new prompt or model, and merge or discard. A twenty-line shim is all it takes to wire your sandboxes to it.

An agent is four things in four places​

Be precise about what "an agent" physically is, because it isn't one process on one machine. It decomposes into four pieces, hosted in four different places:

1. The model lives at the inference provider: Anthropic, OpenAI, or your own GPUs. It's stateless per request and shared across every customer the provider has. Nothing agent-specific survives there between calls.
2. The harness is the orchestration loop: it holds the conversation, dispatches tool calls, and decides when to stop, from an app server or queue worker on your compute. It's the piece people usually point at and call "the agent," but a well-built harness is stateless: kill it, and any replica picks the loop back up, provided the state it was looping over still exists.
3. The execution environment is the sandbox, where tool calls and generated code run. It is disposable on purpose; that's the feature you're paying for.
4. The state is everything that has to survive between runs for the agent to still be the same agent tomorrow.

Three of these four layers are replaceable at will; the fourth is the one you can't regenerate, and that is the honest answer to the question in the title. The agent doesn't live in the sandbox, the harness, or the model; it lives in its state, and everything else is a way of moving that state forward in time.

The agent's world, frozen mid-task​

So what exactly is in that fourth layer? If you froze an agent mid-task and took inventory of everything it would need to become itself again, you'd find four things:

• Working files. The workspace: the repo it's editing, the artifacts it has produced, the half-finished output sitting in a scratch directory. Already bytes, already belongs in object storage.
• Memory. Conversation history, scratchpads, whatever the agent has learned across sessions. The thing that makes session forty-one smarter than session one.
• Structured state. The run metadata, task queue, tool results, config. The reflex is to put this in a database and store a pointer to the blobs, but object storage is a key-value store, and namespaced keys like runs/{id} and memory/{agent}/{session} hold structured state in the same namespace as the blobs it describes. No pointer pattern, no two systems to keep in sync.
• Provenance. What the world looked like before the agent touched it. The layer you don't think about until an agent does something destructive and you need last Tuesday back.

Notice what this inventory has in common: all four are bytes under keys, which means you can hold the agent's entire world, frozen mid-task, in one namespace with one snapshot boundary. The bucket at that moment is the agent at that moment.

Today, that world is scattered across five systems​

That's the theory. In most real deployments the fourth layer looks nothing like one namespace:

Each piece is fine on its own; the problem is the seams, because there is no moment at which you can capture all five systems in a consistent state. You cannot snapshot the agent, because the agent has no snapshot boundary, and you cannot fork it to try two continuations. You cannot replay Tuesday's run byte-for-byte either, because Tuesday exists in five places that were never consistent with each other in the first place.

Consolidating the world into one bucket is a real trade, and it's worth naming: you give up SQL over your run metadata for atomicity over your agent. If your access pattern is get, put, and list (and for agent state it almost always is), that trade is heavily in your favor. If you need rich queries over metadata, keep a queryable index and treat the bucket as the source of truth it's derived from.

"But my sandbox has snapshots"​

It does, and the objection deserves a direct answer: Daytona snapshots and volumes do persist state across sandbox restarts, and if persistence were the whole requirement, you could stop there. This isn't a knock on Daytona either; their own docs treat Tigris as a first-class mount target precisely because the two layers are built to split the work this way: Daytona owns the compute, the bucket owns the world. Each layer checkpoints what it owns: Daytona checkpoints the machine your agent runs on, and the bucket checkpoints the agent itself, the whole world as one point-in-time unit you can restore, branch, and compare.

The requirement is forkability, and the difference is not subtle. A sandbox snapshot is a photograph of a machine: it restores exactly one state, and it hauls the full image around every time.

A bucket fork is a branch of a world. On Tigris a fork is copy-on-write: metadata rather than bytes. You can branch a 40 TB world for exactly what a 40 MB one costs, and branch it fifty times for the price of once. The fork outlives every sandbox, and because it's just a bucket, you can read it from any cloud or provider with zero egress. You want the machine frozen approximately never; you want the world branched constantly: before every risky action, for every parallel variant, at the start of every eval run.

The loop is always the same three moves: fork the world, work against the fork, promote the winner.

That loop is where fleets live or die. A platform running a thousand concurrent agents takes a thousand forks before lunch, and zero-copy is what makes that a non-event: creating a fork doesn't scale with world size, a fork stores only what its run actually writes, and deleting a dead one frees exactly those deltas. Per-run isolation at fleet scale is an economics problem before it's an engineering one, and copy-on-write answers both. We've written before about fifty agents sharing one bucket; one fork per run is the same idea with the blast radius set to zero.

The shim: boot the sandbox already bound to its fork​

Once the world is one bucket, the lifecycle writes itself: fork before the run, mount the fork in the sandbox, merge or discard after. You wire it up with three small pieces: a few lines in your harness that fork the world and mint credentials scoped to that fork, an entrypoint wrapper around the Tigris CLI that binds the sandbox to its fork before your agent's first instruction runs, and the storagesdk MCP server that lets your agent work against the fork without copying it.

Before the code, a quick sketch of how Daytona works, because the shim leans on it. A Daytona sandbox is a full computer with its own kernel, filesystem, and network stack, created from a snapshot image you define once in code and ready to execute in under 90 milliseconds. The Python and TypeScript SDKs drive the whole lifecycle from your harness: build the image, create the sandbox, exec the process, destroy it. We pair with Daytona because the economics match: fork-per-run only pays off if both halves of the loop are cheap, and a zero-copy fork plus a sub-100ms sandbox means per-run isolation costs milliseconds on the storage axis and on the compute axis alike. One layer makes the computer disposable, the other makes the world durable, and neither has to pretend to be the other.

The harness piece comes first, and it's where the isolation becomes enforceable:

# In the harness, before the sandbox boots.
FORK="run-${RUN_ID}"

# 1. Fork the world. Copy-on-write, metadata-only:
#    constant-time whether the bucket holds megabytes or terabytes.
tigris buckets create "$FORK" --fork-of agent-world

# 2. Mint a key that can see only the fork. Capture the key ID and
#    secret from the output; the secret is shown exactly once.
tigris access-keys create "sandbox-${RUN_ID}"
tigris access-keys assign "$KEY_ID" -b "$FORK" -r Editor

Pass the fork name and the scoped key into the sandbox's environment when you create it. In Daytona's Python SDK that's one call:

# Still in the harness. Daytona() reads DAYTONA_API_KEY; fork,
# key_id, and key_secret were captured from the tigris commands above.
from daytona import CreateSandboxFromSnapshotParams, Daytona

sandbox = Daytona().create(
    CreateSandboxFromSnapshotParams(
        snapshot="agent-runtime",  # your image, entrypoint baked in
        env_vars={
            "FORK": fork,
            "TIGRIS_ACCESS_KEY_ID": key_id,
            "TIGRIS_SECRET_ACCESS_KEY": key_secret,
        },
    )
)

Those three variables are the only credentials the sandbox ever holds, which means the base bucket isn't hidden from your agent; it's unreachable: a prompt injection, a buggy tool, or a model on a bad day can name agent-world all it wants and the API will refuse it. Inside the snapshot, the entrypoint does the rest:

#!/usr/bin/env bash
# entrypoint.sh, baked into the Daytona snapshot. The sandbox
# arrives holding fork-scoped credentials and nothing else.
set -euo pipefail

# 0. The harness set these three at sandbox creation. If any are
#    missing, fail loudly now rather than partway through a run.
: "${FORK:?}" "${TIGRIS_ACCESS_KEY_ID:?}" "${TIGRIS_SECRET_ACCESS_KEY:?}"

# 1. The scoped key arrives as TIGRIS_ACCESS_KEY_ID and
#    TIGRIS_SECRET_ACCESS_KEY (what storagesdk reads); mirror it
#    to the AWS names the Tigris CLI reads.
export AWS_ACCESS_KEY_ID="${TIGRIS_ACCESS_KEY_ID}"
export AWS_SECRET_ACCESS_KEY="${TIGRIS_SECRET_ACCESS_KEY}"

# 2. Pull the working set only. Memory, run metadata, and artifacts
#    stay in the fork and are read by key, never copied into the VM.
tigris cp -r "t3://${FORK}/workspace/" /workspace/

# 3. Bind every storage tool to the fork. The MCP server below and
#    any `storage` command the agent runs see the fork, not the base.
export STORAGE_ADAPTER=tigris
export TIGRIS_BUCKET="${FORK}"
exec "$@"

Set it as the snapshot's entrypoint and the agent process (node agent.js, python -m agent, whatever you run) inherits a world that is entirely its own. The worst run your agent will ever have is scoped to a fork you were always free to throw away.

If you'd rather not copy even the working set, Daytona documents mounting a Tigris bucket straight into the sandbox with mount-s3 and the https://t3.storage.dev endpoint. Bake mount-s3 into your Daytona snapshot, point the mount at the fork, and the whole world shows up as a local directory, so your agent's existing file tools work against the fork without a byte pulled up front. The fork-scoped key works unchanged, because mount-s3 reads the same AWS_* variables the entrypoint already exports.

The second piece is how your agent touches the rest of its world: the memory, run metadata, and artifacts the entrypoint deliberately didn't copy, because copying a terabyte world into a microVM would throw away the point of a zero-copy fork. Instead, your agent reads and writes them by key through storagesdk, whose MCP server exposes every storage verb as a tool. Install it when you build the Daytona snapshot (npm install -g @storagesdk/cli) and add one entry to the agent's MCP config:

{
  "mcpServers": {
    "world": { "command": "storage", "args": ["mcp"] }
  }
}

Because the entrypoint exported TIGRIS_BUCKET as the fork, every tool the server offers (download, upload, ls, snapshot_create) is already aimed at the fork, and the fork-scoped key is what makes the aim binding. Your agent can also checkpoint its own world before a risky migration and roll back if the result looks wrong, which is a nice trick, but don't build your safety story on the model remembering to do it. The guarantees live at the boundaries: the fork the entrypoint created, and the snapshot the harness takes before promotion.

When the run ends, you decide what the fork was worth, and you decide it from the harness, not the sandbox:

# Run passed validation: promote the fork's changes back to the base.
tigris cp -r "t3://run-${RUN_ID}/workspace/" "t3://agent-world/workspace/"

# Run failed, or was an experiment you're done with: drop the fork.
# Deleting a fork frees only the deltas the run actually wrote.
tigris rm "t3://run-${RUN_ID}" -f

# Either way, the run's credentials die with the run.
tigris access-keys delete "$KEY_ID" --yes

One sandbox.delete() in the same harness retires the compute the same way. Be honest about the asymmetry here: the fork was free, and promotion isn't. It's a copy, the one step in the loop that costs what it weighs, so promote the prefixes the run actually changed rather than the whole fork.

If you want provenance on top, enable snapshots on the base bucket and take one before each promotion with tigris snapshots take agent-world, and "what did the world look like before run 4187" becomes a timestamp you can fork from.

Every disposable layer sits above the line, and the one layer you can't rebuild sits below it, forkable:

What the bucket does, and doesn't​

One bucket doesn't run anything: you still bring the harness, the model, and the sandboxes, and the bucket is the state they orbit rather than a replacement for any of them. It doesn't merge divergent forks for you either: deciding which run's changes deserve promotion is your validator's job, and the fork only makes that decision cheap to act on.

It's also not the first tool to reach for in every deployment. If your agent's entire world is a git repo, git is already a forkable, snapshotted store, and if the world is a few megabytes, a runs/{id}/ prefix convention inside one bucket buys most of the isolation with none of the new moving parts. The bucket earns its keep when the world spans files plus memory plus metadata (git only covers the first), when it's big enough that copying is the bottleneck, or when runs you don't fully trust are mutating state other runs depend on.

What it does give you is the one property the scattered version can't: a snapshot boundary around the whole world. Files, memory, structured state, and provenance move as one unit: forked together, promoted together, deleted together. Every operation in this post falls out of that single fact.

Hand this prompt to your agent​

If your agent builds itself (and whose doesn't, lately), here's the setup as a prompt instead of a runbook. Paste it into Claude Code or the coding agent of your choice:

Set up durable, forkable state for my agent fleet on Tigris.
Assume the tigris CLI is installed and authenticated, and my
harness already creates Daytona sandboxes; wire this into it.

1. Create the base bucket:
   `tigris buckets create agent-world --enable-snapshots`.
2. Lay out the namespace: `workspace/` for working files,
   `memory/{agent}/{session}` for memory, `runs/{id}` for run
   metadata as JSON values under keys (object storage as KV:
   no separate database, no pointer pattern).
3. In the harness, before each sandbox boots: fork the base
   (`tigris buckets create run-${RUN_ID} --fork-of agent-world`),
   mint a key scoped to the fork (`tigris access-keys create`,
   capture the printed ID and secret, then
   `tigris access-keys assign $KEY_ID -b run-${RUN_ID} -r Editor`),
   and pass them into the sandbox env (env_vars on the Daytona
   SDK's create call) as FORK, TIGRIS_ACCESS_KEY_ID, and
   TIGRIS_SECRET_ACCESS_KEY, its only credentials.
4. Write an entrypoint.sh that mirrors the two TIGRIS_* vars to
   AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, pulls the
   working set with `tigris cp -r t3://${FORK}/workspace/
   /workspace/`, and exports STORAGE_ADAPTER=tigris and
   TIGRIS_BUCKET set to the fork. Bake it into my Daytona
   snapshot as the entrypoint, alongside
   `npm install -g @storagesdk/cli`.
5. Register `storage mcp` in my agent's MCP config so it reads
   and writes memory, run metadata, and artifacts by key against
   the fork instead of copying them.
6. After each run, in the harness: if validation passes, take
   `tigris snapshots take agent-world`, then promote with
   `tigris cp -r t3://run-${RUN_ID}/workspace/
   t3://agent-world/workspace/`; otherwise drop the fork with
   `tigris rm t3://run-${RUN_ID} -f`. Either way, delete the
   run's access key: `tigris access-keys delete $KEY_ID --yes`.

An agent you can't fork is an agent you can't trust​

So, where does the agent live?

Your agent runs in a Daytona sandbox, calls a model, and takes orders from a harness, and all three can vanish mid-task without costing you anything but a restart. The one layer that can't be rebuilt is the bucket. Treat that bucket as the agent, and the operations you've been missing fall out for free: snapshot it and you can replay any run, fork it and you can try anything, delete a fork and the worst run your agent ever had never happened. And the pattern doesn't change between one agent and a thousand: a fleet is just more forks of more worlds, each one scoped, cheap, and disposable. That's where the agent lives, so make it forkable.

The last time I shipped a "small" prompt change to a production agent, it passed every test I had and made the agent worse. It got a little more generic, a little more likely to ask a customer something they'd already told it. My tests didn't catch it because my tests were three hand-written threads, and three hand-written threads have no past. The behavior that mattered only showed up against customers with history, and history is exactly what a fixture doesn't have.

If you run an agent platform, that's not your problem once. It's every tenant's problem, multiplied. Each agent accumulates state you don't own and can't recreate, and your job is to keep all of them in line as you change them. To know a change actually helps, you have to eval(uate) it against the real, accumulated world the agent lives in. Then you can hand that workflow to your users, but only if their state lives somewhere you can branch.

This is an exercise in building a small harness that answers one question without touching production: will this change make my agent better or worse against real users?

All the code for this example lives here: examples/eval-on-real-state.

What a state-blind regression looks like​

Here's what one of your tenants' customers actually sees after the change ships:

Ana told the agent weeks ago that she's vegetarian with a severe tree-nut allergy, and that fact now lives in her thread. When she asks for "a dinner from this week's menu," the old prompt answered straight from memory with "a vegetarian, nut-free pick," but my "small" edit made the agent a little more generic and it started replying "any allergies or preferences?", re-asking something Ana had already told it.

The reason my tests missed it is the whole point. On a blank fixture there is no prior turn to recall, so both the old and new prompt ask the same clarifying question and look identical. The regression only exists where there's a past to forget, and a fixture has none.

The whole loop on one page​

The whole thing is four steps: fork prod into a throwaway bucket, replay real threads through the candidate, score it against the baseline, then drop the fork. The trick that makes it practical is something you can only do on object storage: forking the bucket. When you fork a bucket holding agent data, you clone the entire agent: every thread, checkpoint, and message. The clone is made by reference, so it lands instantly no matter how much history has accumulated.

The checkpointer is langgraph-checkpoint-tigris, a drop-in BaseCheckpointSaver that stores each checkpoint as an object in a Tigris bucket. Install it and let's build each box in that diagram.

pip install -U langgraph-checkpoint-tigris

1. An agent whose only variable is the change under test​

Hold everything fixed except the thing you're evaluating: the graph, the model, and the accumulated thread history all stay the same, so any difference in the verdict is attributable to your change. Here the change is the system prompt, prepended at call time so it never gets baked into the stored thread. That is what lets us replay the same real conversation under two different prompts.

from langchain.chat_models import init_chat_model
from langgraph.graph import START, MessagesState, StateGraph

BASELINE_PROMPT = (
    "You are a helpful customer support assistant. Answer the user's question."
)
CANDIDATE_PROMPT = (
    "You are a helpful customer support assistant. Before answering, recall "
    "everything you already know about THIS customer from the conversation so "
    "far (their plan, preferences, constraints, and past issues) and tailor "
    "your answer to it. Never ask them to repeat something they've told you."
)

def build_agent(system_prompt: str, temperature: float = 0.2) -> StateGraph:
    model = init_chat_model("claude-haiku-4-5-20251001", temperature=temperature)

    def call_model(state: MessagesState) -> dict:
        messages = [{"role": "system", "content": system_prompt}, *state["messages"]]
        return {"messages": model.invoke(messages)}

    builder = StateGraph(MessagesState)
    builder.add_node("call_model", call_model)
    builder.add_edge(START, "call_model")
    return builder

The temperature is low on purpose. We want to measure the prompt's effect, not sampling noise. That candidate prompt is the kind of one-liner that looks like nothing on a fixture and proves itself only against memory-rich threads.

2. The real state you evaluate against​

Each thread is one real customer conversation. It carries the memory we seed in its history, a held-out probe we score both variants on, and the keywords that prove a reply actually used the memory.

from dataclasses import dataclass, field

@dataclass
class Thread:
    thread_id: str
    history: list[str]                 # prior turns that establish the memory
    probe: str                         # the held-out question we score on
    recall_markers: list[str] = field(default_factory=list)  # proof of recall

In the demo we seed four of them, each holding one fact a good answer has to use. The marker is a deliberately simple substring check: it is not the score, it just makes the lesson concrete.

Your real prod bucket already has state like this. On a platform, every tenant's bucket already has it. The seeding just gives you something to run against out of the box.

3. Fork prod and replay (the load-bearing call)​

This is the heart of it: each variant gets its own zero-copy fork of prod, runs every real thread's probe inside that fork so each answer is produced with the customer's actual history in context, and the fork is dropped when we're done.

def run_variant_on_fork(prod, system_prompt, threads, *, name, run_id, keep=False):
    fork = prod.fork(f"{prod.bucket}-eval-{name}-{run_id}")
    graph = build_agent(system_prompt).compile(checkpointer=fork)

    responses = {}
    for t in threads:
        out = graph.invoke(
            {"messages": [{"role": "user", "content": t.probe}]},
            {"configurable": {"thread_id": t.thread_id}},
        )
        responses[t.thread_id] = out["messages"][-1].content

    if not keep:
        drop_bucket(prod.client, fork.bucket)  # erase the branch, no residue
    return responses

What makes this safe comes down to two properties.

First, the fork inherits all of prod: every thread, checkpoint, and learned fact. Replaying conversations against it runs them against the real world, not a mock that won't reflect it.

Second, the fork is a separate copy. The candidate's responses are written into the fork and never into prod, so nothing the eval does can touch live data.

Together that lets you iterate on live state without fear. To evaluate ten variants instead of two, call this in a loop; each fork is independent and costs storage only where it diverges.

4. Judge head-to-head, and cancel the judge's bias​

Pairwise LLM-judging is how most teams actually grade agents, and it needs no answer key. The one trap is position bias: judges tend to favor whichever reply came first. So we ask twice with the order swapped and only count a win when it is consistent across both.

def judge_report(threads, baseline, candidate, judge):
    results = []
    for t in threads:
        a, c = baseline[t.thread_id], candidate[t.thread_id]
        v1 = judge(t.probe, a, c)   # A=baseline, B=candidate
        v2 = judge(t.probe, c, a)   # swapped
        if v1 == "B" and v2 == "A":
            winner = "candidate"
        elif v1 == "A" and v2 == "B":
            winner = "baseline"
        else:
            winner = "tie"          # inconsistent: position bias, call it a tie
        results.append((t.thread_id, winner,
                        recalled(a, t.recall_markers),
                        recalled(c, t.recall_markers)))
    return results

Alongside the verdict we run the cheap recall check from step 2: did each reply contain the fact we know lives in that thread's history? Against fixtures that number is 0/0 for both variants and the whole comparison is a wash. Against real state, the gap is the regression your fixtures were hiding.

5. Run it​

thread       winner      baseline recall   candidate recall
------------ ----------- ----------------- ----------------
cust-ana     candidate   no                yes
cust-ben     candidate   no                yes
cust-cleo    tie         yes               yes
cust-dan     candidate   no                yes

head-to-head: candidate 3 / baseline 0 / ties 1
used real memory: candidate 4/4  vs  baseline 1/4

VERDICT: SHIP — candidate beats baseline on real state

Your exact numbers will vary, but the judge runs at temperature 0, so the verdict is stable across re-runs. The same change, evaluated against three blank fixtures, would have come back a tie, and you'd have shipped it never knowing which way it went.

Why this only works on object storage​

You can't bolt instant whole-agent forking onto a relational checkpointer, because the cost model is wrong. A database reads rows to copy them, so branching costs more the more history you have. That's exactly backwards. Tigris shares immutable blocks and layers new writes on top, so a fork is a pointer, not a duplication, and dropping it leaves nothing behind.

The shape of the problem changes once you're running a fleet. A database puts every tenant behind one primary and a bounded connection pool, so the thing you scale is a bottleneck you have to manage. Give each agent its own bucket and the bottleneck disappears: the requests are independent, the state scales to zero when idle, and any single agent forks in constant time.

The same storage gives a platform the rest of what it needs. Lined up against a database, the differences are not small:

LangGraph already defines the checkpointer seam, and Tigris just fills it, so none of your graph code changes. The first four rows make object storage the right home for a platform's agent state. The last row makes that home a capability you can resell: per-PR eval environments, "clone this agent," instant rollback, all the same fork() wearing different hats.

What it doesn't do​

Use a Single-region or Multi-region bucket. Those give the strongly consistent reads the saver relies on to find the latest checkpoint, where Global and Dual-region buckets can hand back a stale one. Forking makes the state free, not the inference: an N-variant sweep is N times the tokens. And the verdict is best read as a confidence signal that informs the decision, rather than a hard gate that blocks the merge on its own.

The short version​

If you're on LangGraph, swapping in the Tigris checkpointer is one pip install and one connection string. You keep resume, time travel, and human-in-the-loop. What you gain is the move databases make too expensive to bother with: cloning your entire agent in constant time, with no data copy. That's what lets you keep an agent in line as you change it, evaluating each tweak against the real thing instead of three fixtures.

The checkpointer: langgraph-checkpoint-tigris on PyPI.

The full runnable example, seeded so it runs end to end against your own bucket: examples/eval-on-real-state.

What happens if I just point a git server at an object storage bucket?

Back when I was porting agent sandboxes to Go, I built everything on top of billy, a filesystem abstraction for Go. The whole trick of the project was teaching a Tigris bucket to act enough like a filesystem that a shell interpreter and its tools couldn’t tell the difference. Billy was the key layer that made the entire façade fall into place.

After I had gotten things working, I learned that I’m using billy way outside its normal usecase. It was originally made for go-git, a pure-Go implementation of git’s protocols and data formats. It doesn’t rely on the /usr/bin/git binary existing at all. Every method on billy’s filesystem interface exists purely because go-git needs it. This gave me a terrible idea: I already have a bucket that can quack like a filesystem and go-git’s native language is “filesystem”.

Can this Just Work™? Let's find out.

Git was always an object store​

If you strip away the porcelain, a git repository is 4 basic things:

• Objects, or compressed blobs of data. Most of the objects in any individual repository are files.
• Trees, or objects that map to other objects. TL;DR: trees are folders.
• Commits, or objects that point at one tree and their parent commit. This lets you pin down which files belong to one logical change set.
• Refs, branches and tags, they are tiny mutable pointers into the pile of objects.

For example, let’s say I just made a new git repository and committed a README.md to it. The tree for the .git folder looks something like this:

$ tree .git
.git
├── COMMIT_EDITMSG
├── config
├── HEAD
├── index
├── objects
│   ├── 5e
│   │   └── b8151eb669aa4467b6dea2c4bce19183cd0b41
│   ├── 6a
│   │   └── 6a8ecfcae2632152486aca3d9150ef83dedd66
│   ├── f4
│   │   └── d2487a1c6d742c8037c0296ddf80625190bd80
│   ├── info
│   └── pack
└── refs
    ├── heads
    │   └── main
    └── tags

As you can see there are three objects. One of them is the commit 5eb8151eb669aa4467b6dea2c4bce19183cd0b41, the next is the tree, and the last one is the README file. The main branch also points to that commit:

$ cat .git/refs/heads/main
5eb8151eb669aa4467b6dea2c4bce19183cd0b41

The cool part is that half of this is content-addressed. The content-addressed bits never change once they’ve been committed. Git objects are a great fit for Tigris’ internal model because they are append-only storage, just like the fundamental model Tigris is built upon. The things that do change often are the refs, which are updated to point to the latest commit. These are tiny files though, which means that Tigris can handle them with no effort required.

However, when we host git repositories on a server, we end up creating single points of failure. Our git repos are hosted on single machines that can and will break. The entire implementation relies on git objects being 1:1 correlated with filesystem objects because everyone (even GitHub) shells out to the git binary to actually store files. Hosting git repos becomes one of the most stateful services in our stateless cloud-native environment.

Sure git is in-theory decentralized, but most of us have ended up using that to put our git repositories in one big store that has questionable uptime practices: GitHub. To be fair to hubbers, GitHub operates at a scale that none of us can really think about. They’ve been pushing the limits since their inception where they had to get Engine Yard to keep building them bigger servers to handle the load. They have to do everything with a big mounted filesystem because git’s tooling gives them no other option.

A travesty of horrors beyond human comprehension​

Now suppose this weirdness bothers you enough to do something about it. To build a git server without storing everything in the local filesystem, you have to speak git somehow, and the conventional options aren’t really all that great:

• If you shell out to the git binary, now your “library” is the argv of the git process and your error handling is screen-scraping output. Internally, git implements its functionality with a billionty subcommands rather than exposing it all as a library. The codebase is held together by load-bearing calls to die(), which kills the process.
• If you link into git’s guts with libgit, you inherit the “when things go bad, die()” behaviour and your app now suddenly starts crashing at random. This is not good for uptime.
• If you try to use libgit2 (the rewrite-that’s-actually-a-library), you have to reckon with the fact that it’s addled by the GPL (with a linking exception, try explaining that to your lawyers), you have to eat the jump to C every time you do anything with git (very often), development has stalled, the Go bindings have been archived, and it still assumes a local filesystem despite assurances it does not.

It might sound hopeless, right? You may be able to use WebAssembly or something to contain the madness (assuming you have a good way to implement fork()/exec() or posix_spawn() or something similar), but what if there was a pure Go library that could handle this all for us?

Enter go-git, a pure-go implementation of the git protocol and internals from scratch. This doesn’t rely on cgo or /usr/bin/git and it does not assume the repositories are stored in the local filesystem. Its storage interface is written against billy, the exact interface I’ve already taught to speak Tigris. I wanted a git server that was just in a bucket and the pieces were sitting there and calling to me.

Oh no, it works​

So I hacked up objgit, a git server backed by object storage. The only filesystem call I had to add to get it booting was MkdirAll. I wired up the transport package to a socket to implement the plaintext git protocol, hooked it up to a bucket, and pushed the repo I was currently working on.

To my absolute astonishment, it worked.

Git pushed, pulled, logged, blamed, tagged, the whole kit and kaboodle. I didn’t have to implement git myself, I just committed an egregious amount of shoving a square peg into a round hole until the peg went in.

In hindsight this makes an annoying amount of sense. A bare repo is those four kinds of things on a filesystem; swap the filesystem for object storage and everything else Just Working™ is perfectly logical. Git’s on-disk format is its database schema and if you fake open/stat/rename convincingly enough the entire façade keeps working because APIs are the lies we tell ourselves to make us sleep at night.

After a lot of hacking, I ended up with a feature list kinda like this:

• Push and pull over three transports: HTTP, classic git://, and SSH
• Repositories upserted on first push
• Absolutely no effort put into authentication as this is an experiment and authentication is annoying and complicated
• Prometheus metrics so I could optimize the filesystem layer

Everything comes out of one Go binary with no local state, even the generated SSH keys are stored in the bucket. You can run this in a Kubernetes cluster with only the mutable storage required being temporary files for an optimistic cache when doing smart git clones.

The rest of this post is what it took to get from “oh no, it works” to something close to usable.

Obligatory disclaimer (like the best things in life): this is an experiment. It has not been tested thoroughly or vetted for correctness. If it breaks in half, you get to keep both pieces. Please do not move your company’s monorepo onto this and then email me when it catches fire.

That one POSIX idiom that survived​

Git is paranoid about durability, and its entire strategy is one Unix idiom that you end up seeing many places: write new data to a temporary file and then rename(2) it into place after you’ve assured it’s correct. POSIX guarantees that rename is atomic, so readers either see the old file or the new one, not an intermediate state inbetwixt the two. Packfiles (bundles of objects) land as temporary files when uploaded then moved to their permanent home. Refs are written as locked temporary files and then renamed over the ref. It’s rename all the way down.

Object storage traditionally does not have rename as one atomic operation. S3’s answer is to create exactly that intermediate state: CopyObject to the new place and DeleteObject on the old one. This makes the most load-bearing idiom in Git’s philosophy fall to pieces.

Luckily, Tigris has an extension for this: RenameObject. To use it, pass an additional X-Tigris-Rename: true header to a CopyObject call and instead of copying then deleting on the client, it moves the metadata around on the server. One round trip, no data movement, and the Unix idiom maps on the bucket 1:1. Objgit’s implementation of Rename is trivial:

// internal/s3fs/basic.go
// RenameObject is a Tigris extension that renames in place (no data copy),
// so we don't need a separate CopyObject + DeleteObject.
copySource := fs3.bucket + "/" + src
_, err := fs3.client.RenameObject(ctx, &s3.CopyObjectInput{
    Bucket:     &fs3.bucket,
    CopySource: &copySource,
    Key:        &dst,
})

A second, sneakier violation hides in the same codepath. When go-git writes a temporary file, it creates that temporary file and then immediately starts opening it for reading so it can build the pack index. You cannot do that with a single live object in any object storage system, you are either reading or writing, never both. I ended up working around this by cheating a bit and buffering the contents of newly written pack files into memory so that this game of chicken kept working. I may have to change this to write that pack cache to the filesystem as trying to push gcc.git made me run out of RAM. At the very least, everything lies consistently enough that git doesn’t care, so win!

Death by a thousand stat() calls​

With this correctness sorted, I tried pushing the golang/go repository to objgit to see how long it would take. It did work, but it took forever. Using the prometheus metrics I mentioned before, I saw that it was making biblical amounts of HeadObject calls. Some blocking profile analysis pointed to the fact that the git library was using the stat() call to detect if a file exists. The flow was like:

• Client has object x
• Check if object x exists
• Check if any pack has object x

And so on ad infinitum. This is fine-ish on a local filesystem because those syscalls resolve in microseconds, not the tens of milliseconds it takes to get from my office to the nearest Tigris region (please expand to Ottawa, I would love that so much).

This was compounded with a discovery that the transport I was using (SSH — classic git:// shares the same code path) was exploding every packfile into loose objects when pushing it. Each loose object write was costing two round trips: stat() to check if a file exists and then open() / write() to actually put the data into Tigris. This made a 100,000 object packfile cost 200,000 object storage calls. Call it 10ms of latency for each one, and that’s over half an hour of waiting for responses that mostly say “404 not found”.

Caching can’t really save you here either, read caches would absorb the repeated reads; but this is a firehose of writes to 100,000 paths that probably have never been read and likely will never be seen again.

The reason only two transports had this problem is a deadlock story. The git library's fast path stores an incoming pack whole through its PackfileWriter, by copying from the connection until io.EOF. Over HTTP that's fine: the request body ends, EOF arrives, everyone goes home. Over git:// and SSH, the connection is a persistent socket and the client is holding it open, politely waiting for the server's status report. EOF never comes. The copy waits forever, the client waits forever, and you have invented a distributed deadlock with two participants. The original workaround was to hide the PackfileWriter capability on those transports so go-git fell back to its streaming parser that writes every object loose. Hence the stat storm.

So the solution was to stop depending on EOF at all. Packfiles are self-delimiting: the header says how many objects are coming and a trailing checksum marks the end, so a packfile scanner walks the stream and stops at the trailer while a TeeReader mirrors exactly those bytes into the PackfileWriter. This makes the rest of the façade fall into place and the git library is happy. This made pushes into two uploads: a packfile and its index instead of a torrent of round trips that mean nothing.

What about cloning?​

Once I got pushing fixed, I moved on to the read path. In order to emulate ReadAt, I used ranged GetObject requests so that the git library could read individual objects out of packfiles. I was happy with this hack, but there was one problem: the latency curse struck again. Cloning a simple repo with 318 objects and a 200KiB packfile made over 8,500 GetObject calls before I killed it.

A git client cloning a repository reads repository packfiles thousands of times with random access, walking objects and candidate delta bases over and over. On a local disk you never notice because your page cache eats that for breakfast. When every call is an HTTP request, a 200KiB repo turns into dozens of megabytes of round trips. A 20MiB repo was effectively unservable.

In other words, I had un-cached the one workload that caching was designed to solve.

The fix leans on a gift from git: pack files are immutable and content-addressed. pack-<sha>.pack will never change for as long as it exists. This makes them trivially cacheable to a faster local medium, such as the filesystem. No invalidation logic is required. I made objgit download packs to a local temporary folder and serve reads from there. To be on the safe side, I did add least-recently-used caching to the mix so that my temp folder wouldn’t blow up unexpectedly. This does mean that the first request for pack files is slower, but then everything else is at filesystem speed.

Yes, this relies on the local disk again, but only as a cache that can and will be thrown away. I think trading a stateless ideal for clones that terminate in reasonable amounts of time is a worthwhile bargain.

Why so ListObjectsV2, Batman?​

Once the other disasters were out of the way, one more remained: the metrics showed a flood of ListObjectsV2 calls every time a clone was made and didn’t stop making those calls after it was done.

Two things compounded. First, when git looks up an object that isn't packed, it probes for a loose object at objects/<xx>/<rest-of-hash>. objgit keeps packs whole, so there are no loose objects, so every probe misses, and each miss across a distinct two-hex prefix triggered a directory listing to find out. There are 256 possible prefixes. A single clone could issue up to 256 ListObjectsV2 calls whose collective answer was a resounding "there is nothing here."

Second (and more embarrassing), the listing cache already had an optimization for this. It collapsed entire subtree lookups into recursive scans so a single listing of the repository could answer every stat() and probe beneath it. It was completely dead in production. The cache matched recursive prefixes against the repo root (refs/), but every repo is chrooted to its own directory, so real keys look like myrepo.git/refs/heads/main. The prefix check wasn’t aware of chroots so it never actually matched anything. Nobody noticed because a cache that degrades to “no caching” still returns the correct answer, just slowly. To rub it in, a cache warmer was dutifully re-listing every one of those useless prefixes every 30 seconds for 10 minutes after each clone. Thousands of background list calls were burned in the service of caching nothing of use.

The fix was insultingly small: when a repo’s filesystem gets chrooted, register that chroot as a recursive subtree root within the cache. This made the cache actually useful and resulted in only one ListObjectsV2 call instead of hundreds. Every sufficiently advanced cache is indistinguishable from a no-op until someone graphs the miss rate.

None of these disasters were exotic. They’re the things filesystems and kernels give you for free — and every perfectly reasonable disk assumption fell to pieces once a network round trip sat at the core. Serving Git repositories is an accidental filesystem latency benchmark. If your storage abstraction has a weak point, Git will find it and the metrics will show you where that problem is.

Post-receive hooks go in clown jail​

One of the most useful parts of hosting your own git server is setting up post-receive hooks. These have been used since time immemorial for things like automatic deployments when you push code to the server. The heart of this is how we get systems like GitHub Actions: it’s code that runs when you are done pushing.

When you push to objgit with --allow-hooks enabled, it looks for a post-receive hook in .objgit/hooks/receive-pack (this corresponds to the git plumbing action, the name can and will be changed) in the tree of the commit you just pushed. It will then spin up a kefka sandbox with a checkout of the git repository at the commit you just pushed mounted at /src and mutable temporary files at /tmp. It gets coreutils and nothing else. No host filesystem, no network, no arbitrary binaries. Output streams back into the pusher as remote: lines just like when you git push heroku main. Eventually I want to make custom commands to allow you to deploy Tekton pipeline changes and kick off CI jobs that way, but for now I’m happy with this working at all.

You can’t implement policy using these hooks yet. I’m working on it.

Now what?​

I taught a bucket to speak git. Where this goes next, roughly in order of how much the ideas keep me up at night.

CI is the obvious next step. I would wire up commands for things like “apply kubernetes object” and “create tekton pipeline run” so that CI would run via your friendly neighborhood Kubernetes cluster and then notify you through some reasonable mechanism. That’s the first thing I’ll build when I have the time.

It would be nice to have a web UI for this, which is complicated for reasons that have nothing to do with git trees, object storage, or anything else and everything to do with the current state of the internet. Git lookups are expensive in the best cases and with the current torrent of unethical scraping ransacking git servers for every scrap of RAM they have, it’s probably a bad idea to implement this without a lot of clever optimizations. Maybe the fact that this doesn’t have load-bearing dependencies on /usr/bin/git would make it more resilient against scrapers. The fact that this is based on object storage could also mean that caching would be a bit easier (having basically unlimited storage is kind of a low-key superpower for caching), but then the main issue would be server load. It’s a tough cookie to handle.

Performance and stability are another place this needs to improve. I’ve tested this on my developer workstation but that is far different from testing it in production. There’s some other performance issues that are easy to fix, but the big one is latency to Tigris. Maybe I can get the devops team to set me up a k3k cluster in production.

Right now this is an experiment as I plug along and feel out the shape of what git-on-object-storage can be. A git server with no disk, no git binary, and no database. If you want to take a look, check it out on GitHub.

Back in 1979, the people building Unix had a very physical problem: how do you get a directory full of little files onto a magnetic tape so you can back it up? Tape is sequential. You can't seek around on it like you can a disk. So they invented a format that streams every file's metadata and contents back to back into one continuous blob: the tape archive, or tar. A tarball is just a header describing a file, then that file's bytes, then the next header, then the next file's bytes, all the way down until you hit the end.

Fifty-some years later I keep running into the exact same problem, except the tape is an object storage bucket and the files are training samples. You've got a few million tiny objects sitting in a bucket, and you need to pull thousands of them at a time, fast. The old solution turns out to be the new solution. Tigris now lets you grab a whole pile of objects in one request with bundles, and the thing it hands back is a tar stream.

The problem: lots of little files​

When you're assembling a dataset, you almost always end up with a smattering of small objects. Images get ingested one at a time as they're discovered. Audio clips, JSON samples, parquet shards, whatever it is; they land in your bucket as individual keys because that's how they showed up. You don't get to pick.

Training is the opposite shape of work. A dataloader wants to pull a batch of a few thousand samples, do it again for the next batch, and keep your GPUs fed so they're not sitting idle burning money. The access pattern wants big sequential reads. The data is stored as tiny scattered ones.

Most object storage makes you reconcile that mismatch the hard way: one GET per object. If your batch is 4,000 images, that's 4,000 separate HTTP requests, each with its own request line, its own headers, its own round trip to the server and back. Even if your client is smart enough to reuse a connection and fire requests concurrently (or lucky enough to get HTTP/2 multiplexing), you're still paying per-object request overhead thousands of times per batch.

Here's the napkin math. Say a round trip to the bucket is 30 ms. Do those 4,000 GETs strictly one after another and you've spent 120 seconds just waiting on latency, before counting a single byte of actual image data. Crank concurrency up to 64 in flight and you're down to roughly 1.9 seconds of pure latency overhead per batch. That's better, but it's 1.9 seconds your GPUs spent doing nothing, every batch, forever.

A bundle collapses all of that into one request. One round trip, one response, one stream of bytes that contains every object you asked for.

How it works​

When you request a bundle, you send a list of object keys to Tigris and the format you want to get the bundle in. The server starts writing the archive directly to you, then walks down your list of keys and appends each object on the end as it goes. Tigris never buffers the whole archive server-side, so the first bytes reach you while it's still pulling the last objects off disk.

Your client can read that and then unpack it with any tar library such as the one in Go's standard library, Python's stdlib tarfile library, or tar-stream in JavaScript. Worst comes to worst, you can write it to a file and shell out to tar by hand, or pipe the HTTP response straight into tar's standard input — whatever you (or your agent) can write the code for.

Nothing exciting is happening here, which is the point: it's just another authenticated request, and the only permission you need is GetObject. You can even do it with curl.

Why you'd actually want this​

Feeding training batches is the obvious win, the case we just did the math on, and the one bundles were built for. You need thousands of samples per batch without eating thousands of round trips, and that's where the latency savings are the most dramatic.

It's not the only place this shows up, though. A few others:

• GDPR export requests. Someone exercises their right to get a copy of their data. Their data is spread across a few hundred objects in your bucket. Instead of orchestrating hundreds of downloads and stitching them together, you hand the server one list of keys and get back a single tarball you can stream straight to the user. Their whole pile, one file.
• Shipping game assets. Games are made of seemingly infinite numbers of tiny files. In an MMO with customizable armor across, say, five playable races, adding a single shirt to the game can mean shipping 20-plus different 3D models, textures, and material definitions just to make that one shirt render correctly on everybody. Pulling those as a bundle beats pulling them one at a time while a player stares at a loading bar.

Anywhere you know the exact set of objects you need up front and you need them together, a bundle turns N requests into one.

Using it from the SDKs​

Every SDK wraps the same flow: make the request, get a stream back, and read members straight out of it without ever touching disk. Here's that call in each language:

curl -X POST "https://t3.storage.dev/my-bucket?bundle" \
  --aws-sigv4 "aws:amz:auto:s3" \
  --user "$TIGRIS_STORAGE_ACCESS_KEY_ID:$TIGRIS_STORAGE_SECRET_ACCESS_KEY" \
  -H "x-tigris-bundle-format: tar" \
  -d '{"keys":["dataset/train/img_00001.jpg","dataset/train/img_00002.jpg"]}' \
  | tar -tv

curl -X POST "https://t3.storage.dev/my-bucket?bundle" \
  --aws-sigv4 "aws:amz:auto:s3" \
  --user "$TIGRIS_STORAGE_ACCESS_KEY_ID:$TIGRIS_STORAGE_SECRET_ACCESS_KEY" \
  -H "x-tigris-bundle-format: tar" \
  -d '{"keys":["dataset/train/img_00001.jpg","dataset/train/img_00002.jpg"]}' \
  | tar -x

tigris bundle my-bucket \
  --keys dataset/train/img_00001.jpg,dataset/train/img_00002.jpg \
  --output batch.tar

cat keys.txt | tigris bundle t3://my-bucket --compression gzip -o batch.tar.gz

import tarfile
from tigris_boto3_ext import bundle_objects

response = bundle_objects(s3_client, "my-bucket", [
    "dataset/train/img_001.jpg",
    "dataset/train/img_002.jpg",
])

with tarfile.open(fileobj=response, mode="r|") as tar:
    for member in tar:
        if member.name == "__bundle_errors.json":
            # Handle any errors here
            continue
        f = tar.extractfile(member)
        if f is not None:
            image_bytes = f.read()
            # do something with image_bytes

output, err := client.BundleObjects(ctx, &storage.BundleObjectsInput{
    Bucket: "my-bucket",
    Keys: []string{
        "dataset/train/img_001.jpg",
        "dataset/train/img_002.jpg",
    },
})
if err != nil {
    log.Fatal(err)
}
defer output.Body.Close()

tr := tar.NewReader(output.Body)
for {
    hdr, err := tr.Next()
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
    if hdr.Name == "__bundle_errors.json" {
        // handle any errors here
        continue
    }
    data, _ := io.ReadAll(tr)
    // process hdr.Name, data
}

import { bundle } from "@tigrisdata/storage/server";
import tar from "tar-stream";

const result = await bundle("my-bucket", [
  "dataset/train/img_001.jpg",
  "dataset/train/img_002.jpg",
]);

if (result.error) {
  throw result.error;
}

const extract = tar.extract();
extract.on("entry", (header, stream, next) => {
  if (header.name === "__bundle_errors.json") {
    // handle any errors here
    stream.resume();
    next();
    return;
  }
  const chunks = [];
  stream.on("data", (chunk) => chunks.push(chunk));
  stream.on("end", () => {
    const data = Buffer.concat(chunks);
    console.log(`${header.name}: ${data.length} bytes`);
    next();
  });
  stream.resume();
});

const { Readable } = await import("stream");
Readable.fromWeb(result.data.body).pipe(extract);

Since the dataloader case is the whole reason this exists, here's what it looks like wired into a PyTorch IterableDataset that prefetches a chunk of batches per bundle request:

import random
import tarfile
from io import BytesIO
import torch
from PIL import Image
from tigris_boto3_ext import bundle_objects

def build_batches(metadata_path, batch_size):
    """Load object keys from a metadata file and split them into batches."""
    ...

class TigrisBundleDataset(torch.utils.data.IterableDataset):
    def __init__(self, s3_client, metadata_path, bucket, batch_size=32, prefetch=20):
        self.s3_client = s3_client
        self.bucket = bucket
        self.batch_size = batch_size
        self.prefetch = prefetch
        self.batches = build_batches(metadata_path, batch_size)

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:
            my_batches = self.batches
        else:
            my_batches = self.batches[worker_info.id::worker_info.num_workers]
        random.shuffle(my_batches)

        for i in range(0, len(my_batches), self.prefetch):
            chunk = my_batches[i : i + self.prefetch]
            keys = [row["key"] for batch in chunk for row in batch]

            with bundle_objects(self.s3_client, self.bucket, keys) as response:
                with tarfile.open(fileobj=response, mode="r|") as tar:
                    for member in tar:
                        if member.name == "__bundle_errors.json":
                            # handle any errors here
                            continue
                        f = tar.extractfile(member)
                        if f is None:
                            continue
                        image = Image.open(BytesIO(f.read())).convert("RGB")
                        yield {"image": image}

What happens when a key is missing​

Remember the __bundle_errors.json file every example skips? That's how missing keys get reported.

By default, bundles run in skip mode: if a key doesn't exist, the server leaves it out instead of failing the whole request, and lists what it dropped (and why) in __bundle_errors.json. It's metadata, not one of your objects, so read it if you care which keys went missing and skip it otherwise.

If you'd rather fail loudly, send x-tigris-bundle-on-error: fail (the SDKs have an equivalent flag). The server then checks every key up front and returns a 404 listing what's absent, instead of a partial tarball. Skip mode keeps you training through a few gaps; fail mode is for when the bundle has to be complete.

There are a handful of limits worth keeping in your back pocket: up to 5,000 keys per request, up to 50 GB assembled, a 5 MB cap on the request body itself, and a 15-minute timeout on the whole thing. If you're pulling more than 5,000 objects, batch your batches.

One honest tradeoff: a bundle is a sequential stream, not a random-access archive. You can't range-request a single member out of the middle of it, and one enormous object in the list will stream in its entirety before you get to the next one. For the "I know exactly which thousand small files I want" workload that's exactly right. For "I want one specific chunk of an 8 GB file," a plain old GET is still the better tool.

The web console does it too​

Don't want to write any code? The web console downloads a bundle straight from the object browser: select the objects you want, hit download, and you get back the same tar stream the API hands you.

Go forth and bundle​

Object storage spent a long time pretending that the only thing you ever want is one object at a time. The reality of how people actually use buckets, especially for AI training, is that you constantly want a known set of objects together right now without paying a latency tax per file. Borrowing the oldest trick in the Unix book turns out to be a clean way to give that to you.

Today we're shipping soft delete for Tigris buckets and objects. When you turn it on for a bucket, deletes stop being permanent. They go into a holding state for a retention window you set, anywhere from 7 to 90 days. Restore inside that window and the bucket or object comes back. Wait the window out and Tigris removes the data on its own.

A wrong delete is no longer a one-way door. Work that used to need a staging environment or a careful permissions audit can happen on the real bucket, because if something goes wrong, you have a few days to fix it.

Deletes go into a holding state​

Retention is set per bucket. Turning soft delete off later doesn't purge data that's already in the holding state. That data keeps aging out on the schedule it was created with. While a bucket itself is in the holding state, its name stays reserved. You can't reuse the name until the bucket is restored or fully removed.

Your agent writes get a safety net​

Tigris already gives you snapshots and forks when you want to isolate an agent's work into a copy of the bucket. Soft delete adds another option for the cases where the agent or CI job writes directly to a real bucket. Any delete it makes is recoverable inside the retention window. A coding agent cleaning up stale prefixes, a pipeline rotating documents in a RAG corpus, a CI job resetting test fixtures between runs. All of these can write directly, and if something goes wrong, you click restore.

It also changes how you run destructive work against production. A lot of teams keep a staging bucket around as an insurance policy for migrations, cleanups, and retention sweeps that touch real data. You run the script there first, watch what happens, point it at production only after you're sure. With soft delete on, the bucket holds its own undo history. You can run the migration directly, watch what breaks, and put back whatever you didn't mean to touch.

And it's not always about recovering from the last mistake. Every delete on a key gets its own restorable version, stamped with the time it happened. So if a bad write goes out and a string of good ones follow, you can still reach back past the noise to the exact prior state you wanted. The good writes stay where they are.

Every version is its own restore point​

Inside any bucket with soft delete on, the dashboard splits the file list into All files and Deleted files. Pick a deleted object and the Version history panel opens on the right. It lists every soft-deleted version of that key with a timestamp, version ID, size, and ETag. Click restore on the one you want and the object is live again.

Figure 1. The Version history panel for a soft-deleted object, listing each version with its timestamp, version ID, size, and ETag. Each one is restorable on its own.

Enabling it takes one toggle, one header, or one command​

In the dashboard, you can toggle Enable Soft Delete when you create a bucket, or open Bucket Settings → Data Management on a bucket you already have.

Figure 2. Enabling soft delete in the Create Bucket dialog.

Figure 3. Enabling soft delete from Bucket Settings → Data Management.

From the S3 API, set the X-Tigris-Soft-Delete header on CreateBucket. Use true for the default retention of 7 days, or pass a number between 7 and 90 for a custom window.

func createBucketWithSoftDelete(ctx context.Context, client *s3.Client, bucketName string) error {
	_, err := client.CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String(bucketName)}, func(options *s3.Options) {
		options.APIOptions = append(options.APIOptions, http.AddHeaderValue("X-Tigris-Soft-Delete", "30"))
	})
	return err
}

From the CLI, the same setting is a flag on tigris buckets set.

tigris buckets set my-bucket --soft-delete enable --retention-days 30

Restore is one click or one command​

On the buckets page, a Deleted buckets segment lists every soft-deleted bucket that's still inside its retention window. Open the row's menu and click restore. The bucket reappears in the regular list. Every object it held when you deleted it is live and readable again.

Figure 4. The Deleted buckets segment in the Tigris dashboard, with a soft-deleted bucket's menu opened to the Restore action.

Objects work the same way. Open the Deleted files segment, pick a version, click restore.

From the CLI, list soft-deleted buckets and restore one by name.

tigris buckets list --deleted
tigris buckets restore my-bucket

tigris buckets delete my-bucket --yes also respects the bucket's soft delete setting. If soft delete is on, the bucket moves into the holding state and shows up under tigris buckets list --deleted until you restore it or the retention window expires.

Your cost won't change​

Data in the holding state is billed at the same per-GB rate as live data for as long as it sits inside the retention window. There's no separate add-on fee for turning the feature on. When the window expires and Tigris removes the data, the storage is reclaimed and the bill goes down.

Soft delete is recovery, not backup​

Pick a retention window that matches the cycle your team actually operates on. The default is 7 days, which is enough time for most teams to notice that something went wrong and recover. If your incident review runs on a longer cadence, or you want a wider buffer for audits, push the window out to 30 or 60 days. Already soft-deleted data keeps the retention it was created with, so changing the window later only affects new deletes.

Soft delete is recovery, not backup. It protects you against the kind of mistake you can catch inside the window you set. It doesn't protect you against a long-running corruption that goes unnoticed for months, and it doesn't replace whatever you do for long-term retention or disaster recovery. Use it alongside those, not instead of them.

If you're on a snapshot-enabled bucket, soft delete sits underneath snapshots. Deleting a specific version moves that version into the holding state and it shows up in the Deleted files segment, where you can restore it the same way as any other soft-deleted object. A plain delete still records a delete marker and leaves earlier versions live, the same as before.

For the full dashboard walkthrough and the API reference, see the soft delete documentation.

If you use Go for your web services and want to take advantage of Tigris' advanced features like bucket forking, snapshots, and object renaming, normally your code has to look like this:

func WithRename() func(*s3.Options) {
	return func(options *s3.Options) {
		options.APIOptions = append(options.APIOptions, http.AddHeaderValue("X-Tigris-Rename", "true"))
	}
}

// rename the object in the bucket
_, err = client.CopyObject(ctx, &s3.CopyObjectInput{
  Bucket:     aws.String(bucketName),
  CopySource: aws.String(bucketName + "/" + keyName),
  Key:        aws.String(targetName),
}, WithRename())
if err != nil {
  log.Fatalf("Unable to rename object. Here's why: %v", err)
}

With the new SDK, it looks like this:

_, err := client.RenameObject(ctx, &s3.CopyObjectInput{
	Bucket:     aws.String(bucketName),
	CopySource: aws.String(bucketName + "/" + keyName),
	Key:        aws.String(targetName),
})
if err != nil {
	log.Fatalf("Unable to rename object. Here's why: %v", err)
}

The Go Storage SDK fixes this. It gives you dedicated methods for Tigris features, in two modes:

• package storage: Helper methods wrapping the AWS S3 client you already use.
• package simplestorage: A reimagined high-level object storage client for Go applications.

If you want to get started with it today, go get it:

go get github.com/tigrisdata/storage-go@latest

An SDK so nice we did it twice​

One of the best ways to think about Tigris is that it's like S3, but more. Tigris handles global replication for you. Tigris handles migrating your data for you. Tigris also lets you snapshot, fork, and download bundles of objects all at once. These operations extend S3, so the SDK extends the S3 client. Here's what you get:

• BundleObjects: fetch a bundle of objects all at once.
• CreateBucketFork: fork a bucket into a new bucket so your agents have their own sandboxes.
• CreateBucketSnapshot: create a new point-in-time snapshot for your buckets so you can save your data from digital destruction.
• CreateSnapshotEnabledBucket: create a new bucket with forking and snapshots enabled.
• HeadBucketForkOrSnapshot: fetches the fork/snapshot metadata for a bucket.
• ListBucketSnapshots: list all the snapshots for a bucket so you can create forks from those snapshots.
• RenameObject: rename (move) an object from one place in your bucket to another.

As Tigris gets more features, we plan to just add more methods.

This mode is designed to be a drop-in replacement for your existing S3 client to make migration trivial. When I moved some of my own projects over from the AWS S3 library to package storage, it took about 30 seconds at most. Everything compiled normally, everything worked as expected, and I got access to the extra features I needed. Win/win/win!

But wait, there's more​

Honestly, we could have stopped there, but we didn't. Alongside this I also added package simplestorage: a brand new interface to object storage. Most of the time your apps end up using a single bucket. Why should you have to pass the bucket name every time you do anything? Take a gander at how easy it is to copy a file from Tigris to your local filesystem:

client, err := simplestorage.New(ctx)
if err != nil { panic(err) }

obj, err := client.Get(ctx, "my/key")
if err != nil { panic(err) }
defer obj.Body.Close()

slog.Info(
  "object metadata",
  "key", "my/key",
  "size", obj.Size,
  "content-type", obj.ContentType,
)

fout, err := os.Create("./var/object-data")
if err != nil { panic(err) }
defer fout.Close()

io.Copy(fout, obj.Body)

The bucket name and credentials were inferred from environment variables:

• TIGRIS_STORAGE_BUCKET: The default bucket to operate on.
• TIGRIS_STORAGE_ACCESS_KEY_ID: The access key ID for your app's keypair.
• TIGRIS_STORAGE_SECRET_ACCESS_KEY: The secret access key for your app's keypair.

If you don't want to change the environment variable names, simplestorage will use the standard AWS configuration resolution flow you're already used to.

Forked buckets work with the For method:

newBucket, err := client.ForkBucket("my-bucket", "my-agents-bucket")
if err != nil { panic(err) }

newBucketClient := client.For(newBucket.Name)

// do whatever you want here

The main innovation here is reducing cognitive load so that you can focus on what you're doing with your buckets instead of plumbing the minutiae of your object storage API being automatically generated from Java classes. There's also nothing stopping you from using this with AWS S3 or another object storage provider like Hetzner object storage:

s3Client, err := simplestorage.New(ctx,
  simplestorage.WithRegion("fsn1"),
  simplestorage.WithEndpoint("https://fsn1.your-objectstorage.com"),
  simplestorage.WithAccessKeypair(accessKeyID, secretAccessKey),
)
if err != nil { panic(err) }

Advanced Tigris features like bucket forking will not work (Hetzner doesn't implement our extensions to the S3 API), but basic object manipulation will work just fine.

Note that any object manipulation functions will use the client's default bucket, but any bucket manipulation functions require you to spell out the bucket's name just to be sure you're operating on the right bucket.

Get go-ing today​

If you want to try this out, install it in your Go project with go get:

go get github.com/tigrisdata/storage-go@latest

Please give us feedback in the storage-go repo. We want to make this the best way to use Tigris for Go developers and your feedback can only make it better.

You've spent fifteen years branching, tagging, and rebasing your code without thinking twice. Then you reach for object storage and the API is PUT, GET, LIST, DELETE, and "good luck." Buckets are the one piece of your stack you can't safely fork, can't tag a known-good state on, can't mutate on a side branch without a sinking feeling.

StorageSDK is what happens when you decide that's a bug. It's a TypeScript SDK for object storage with snapshots and forks as first-class primitives: branch a bucket per agent run, mutate safely, replay from the same baseline. Same API on top of whichever backend you pick.

The other half of the picture​

ComputeSDK is a vendor-neutral project that gives you one consistent API to control sandboxes across every major sandbox provider. We've partnered with the ComputeSDK team to launch StorageSDK as the matching half — same shape, same goal, but for storage instead of compute. Pick a backend, get out of your way.

The thesis underneath both: Git === storage. Not literally. Conceptually. Buckets need what Git gave us fifteen years ago — tags, branches, isolated mutations, replayable state. Once you've spent a career with those primitives in your VCS, going back to a flat object store feels like editing without undo.

The GitHub adapter is where the metaphor stops being a metaphor: storage.snapshots.create() makes a tag, storage.forks.create() makes a branch, and storage.forks.get(name) hands you a writable storage handle on that branch. The rest of this post is about applying the same idea to every other backend.

One interface. Many backends.​

StorageSDK gives you one interface across basic operations that are portable across every storage provider: get, put, list, delete, copy, and make a URL. Changing backends is one import and one adapter config — swap the tab to see for yourself:

import { Storage } from "@storagesdk/core";
import { tigris } from "@storagesdk/adapters/tigris";

const storage = new Storage({
  adapter: tigris({
    bucket: "agent-runs",
    accessKeyId: process.env.TIGRIS_ACCESS_KEY_ID,
    secretAccessKey: process.env.TIGRIS_SECRET_ACCESS_KEY,
  }),
});

await storage.upload("hello.txt", "Hello, storage SDK!", {
  contentType: "text/plain",
});

const text = await storage.download("hello.txt", { as: "text" });

import { Storage } from "@storagesdk/core";
import { s3 } from "@storagesdk/adapters/s3";

const storage = new Storage({
  adapter: s3({
    bucket: "agent-runs",
    region: "us-east-1",
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    },
  }),
});

import { Storage } from "@storagesdk/core";
import { r2 } from "@storagesdk/adapters/r2";

const storage = new Storage({
  adapter: r2({
    bucket: "agent-runs",
    accountId: process.env.R2_ACCOUNT_ID,
    accessKeyId: process.env.R2_ACCESS_KEY_ID,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
  }),
});

import { Storage } from "@storagesdk/core";
import { gcs } from "@storagesdk/adapters/gcs";

const storage = new Storage({
  adapter: gcs({
    bucket: "agent-runs",
    projectId: process.env.GOOGLE_PROJECT_ID,
    keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS,
  }),
});