5 posts tagged with "foundationdb"

Tigris is a globally available, S3 compatible object storage service.

Tigris uses FoundationDB's transactional key-value interface as its underlying storage engine. In our blog post Skipping the boring parts of building a database using FoundationDB we went into the details of why we chose to build on FoundationDB. To recap, FoundationDB is an ordered, transactional, key-value store with native support for multi-key strictly serializable transactions across its entire keyspace. We leverage FoundationDB to handle the hard problems of durability, replication, sharding, transaction isolation, and load balancing so we can focus on higher-level concerns.

We are starting a series of blog posts that go into the details of how Tigris has been implemented. In the first post of the series, we will share the details of how we have built the metadata storage layer on top of FoundationDB. We will cover the topics of data layout, and schema management.

I did a presentation at the FoundationDB Monthly Meetup. The talk is about, how building secondary indexes for a database is always about balance. A balance between a system that scales and is easy to manage and an API that is intuitive and delightful for a developer to use. Recently at Tigris Data, we have been adding secondary indexes to our metadata store and have been working hard to achieve a good balance between scale and developer delight. Tigris is a storage platform that leverages FoundationDB as one of its core components. In this talk I cover four aspects we had to balance:

1. Handling schema changes automatically in our secondary indexes.
2. The trade-off between auto-indexing all fields and indexing select fields.
3. Changes we made after performance testing.
4. How we plan build indexes in the background with minimal conflicts.

We are running FoundationDB with the official kubernetes operator. FoundationDB supports logical backups (with backup_agent) and disaster recovery (with dr_agent) through copying the database/streaming changes logically. It also supports binary backups through disk snapshots.

In this blog post, we will describe how to make a backup of FoundationDB via backup_agent. The FoundationDB operator supports making logical backups via backup_agent, but it does not support running DR with dr_agent. We decided to run backup_agent as a separate deployment to allow a symmetric setup with dr_agent.

Tigris is a globally available, S3 compatible object storage service. Tigris utilizes FoundationDB's transactional key-value interface for its underlying metadata storage. This blog delves into the topics of serializable transactions, the mechanics of transactions within Tigris, and concurrency control.

The most complicated and time-consuming parts of building a new storage system are usually the edge cases and low-level details. Concurrency control, consistency, handling faults, load balancing, that kind of thing. Almost every mature storage system will have to grapple with all of these problems at one point or another. For example, at a high level, load balancing hot partitions across brokers in Kafka is not that different from load balancing hot shards in MongoDB, but each system ends up re-implementing a custom load-balancing solution instead of focusing on their differentiating value to end-developers.

This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Most of them decide to reimplement common processes and components without substantially increasing the value gained from reimplementing them. For instance, many database builders start from scratch when building their own storage and query systems, but often merely emulate existing solutions. These items usually take a massive undertaking just to get basic features working, let alone correct.