3 posts tagged with "cloud"

At Tigris we put your big data close to your compute so you don't have to do it yourself. However, there's been a small problem with that: most of the programs that are built to process that data such as AI training, document indexing, and other kinds of workloads expect to read data from a filesystem.

Not to mention, big data means big data. Bigger than ram. Bigger than your disk. Bigger than any one machine can have on any amount of disks. Sometimes even bigger than human minds can imagine. What if that data was as easy to access as your code folder, but had unlimited storage?

You’ve got your tunes in high gear, your editor is open, and you’re working on recreating some database tables with your AI agent in an epic duo. Your AI agent says “run this SQL query?” and you click yes. The tests pass, you make PR that’s quickly stamped and merged, when suddenly your pager goes off. And again. And again. You’ve just broken the analytics database and everything is on fire. What do you do?

There’s a new data lake on the market: DuckLake. It’s worth the hype because it gives you an open standard that not only enables you to run queries on your data lakes from anywhere, it outright encourages you to run your query, metadata, and storage layers separately in whatever platform works best for you. We've been thinking about DuckDB as a solution for Small Data, but now with the limitless storage capability of object storage, it can support massive scale datasets. Big Data doesn't have to be complicated.

One of the key changes with DuckLake is moving the metadata layer out of object storage and into a dedicated database. But why would an object storage company like Tigris be excited about putting less data in object storage?