There’s a new data lake on the market: DuckLake. It’s worth the hype because it gives you an open standard that not only enables you to run queries on your data lakes from anywhere, it outright encourages you to run your query, metadata, and storage layers separately in whatever platform works best for you. We've been thinking about DuckDB as a solution for Small Data, but now with the limitless storage capability of object storage, it can support massive scale datasets. Big Data doesn't have to be complicated.

One of the key changes with DuckLake is moving the metadata layer out of object storage and into a dedicated database. But why would an object storage company like Tigris be excited about putting less data in object storage?

Data migrations are risky. Data is sticky; once you put it somewhere, you need to spend a lot of time, energy, and egress fees trying to move it somewhere else. Let’s say you want to move all your data from The Big Cloud™ to another cloud. The gold standard process usually goes something like this:

By day I work on DevRel at Tigris Data, and by night I’m a virtual anime person that makes software deployed by the United Nations. A side project of mine took off recently, a Web AI Firewall Utility named Anubis. Anubis blocks abusive scrapers from taking out web services by asserting that clients that claim to act like browsers do, in fact, act like browsers. As usage has grown (yes, the UN actually uses Anubis), we’ve gotten creative about detecting bots, including setting honeypots that give us more detailed data on their requests. These honeypot logs are sent to Tigris, and we analyze them using DuckDB to find patterns and improve bot detection.

In this article I'm going to be calling those abusive scrapers "AI scrapers", but to be clear I'm not really sure if they are for training generative AI or not. However, the worst offender has been Amazon's Alexa team so it's pretty easy to take those two points and draw a line between them.

As usage of Tigris has grown, so has the need to protect it from abuse. Tigris is a globally distributed, multi-cloud object storage service with built-in S3 API support, and as more developers rely on it for performance and scale, we've also seen a natural increase in attempts to misuse the platform.

Earlier this year, a campaign dubbed "DeceptionAds" came to light, where attackers used fake CAPTCHAs hosted on trusted services to trick users into running malicious commands. The end result: installation of the Lumma info-stealer malware, which exfiltrates sensitive information like passwords and financial data. The attackers used a combination of Bunny CDN (a legitimate content delivery network) and BeMob (an ad tracking service) to distribute and cloak their payloads - demonstrating how malicious actors increasingly hide behind reputable infrastructure to evade detection.

Tigris makes your storage multicloud for you, so you don't have to worry about all that.

Lifecycle rules automatically move your objects to archive or infrequent access tiers to help cut storage costs. This feature has become pretty standard for object storage systems, but under the hood, it's relatively complex to implement. Especially when you're building with global storage across many regions. Let's walk through how we built the feature, and where things got a little complicated.

“Do this and then do that” is hard. Models need specific, stepwise instructions to perform tasks, and even then, they aren’t reliable about completing each step in the desired order. When we built the Tigris Model Context Protocol (MCP) Server we intentionally kept it minimal and composable. We focused only on creating and managing storage. But in thinking through developer experience, we found many of the workflows we wanted to build have multiple steps. Seems easy enough, but there’s a problem: models are missing the entire concept of chaining tools together to accomplish multi-step tasks.

We’ve built a new tool that does two things in sequence: it uploads an object and then creates a shareable URL for it. This should help you use Tigris for your apps that require sharing files with your friends. The URLs this tool generates will expire in 24 hours or however long you ask. But why is it a separate tool in the first place?

AI is all the rage these days, huh? The main problem comes when you're trying to use it in anger to get Real World Apps™️. One of the most popular usecases for AI has you take your documents, shove it all into a search engine, and then go from there to have your model generate responses rooted in the docs.

There's only one problem: every one of those examples leaves the details as an excercise for the reader. Not today! Today I'm going to show you how to ingest and search documents with LanceDB and Tigris.

On Tigris, you can set any domain name to point to any bucket. Why does this matter?

Multi-cloud is all the rage these days. A new neocloud pops up every week offering cheap compute, and running your apps on Kubernetes means they’re easily ported to another provider. Compute is fungible, but storage is sticky. Using a custom domain to front your bucket of data makes it easy to swap out storage providers without updating all the code that uses that storage.

We’ve made it easy to share buckets with your team. Surprisingly, it isn’t so easy to share buckets on $BigCloud. Life doesn’t have to be hard when you have good developer experience.

Let’s say you want to share a bucket in a big public cloud. Suddenly you need permissions to author IAM policies, and you have to worry about user policies, bucket policies, and trawl through the documentation to remember which verb means “this user can do this policy on these resources”. Tigris supports all the IAM goop you’re used to if you want to copy and paste something you already have. But you don’t have to deal with IAM policies if you don’t want to. We’ve made it easy to share access to buckets in the UI, exactly how you’d expect it to work.

As the saying goes, there’s two hard problems in computer science:

1. Cache invalidation
2. Naming things
3. Off-by-one errors

Tigris takes care of cache invalidation, and we’d love to help with that last one, but now we can help you out if you named your object wrong. That’s right, we’ve added the ability to rename objects.