Tigris is a globally available, S3 compatible object storage service.
Tigris uses FoundationDB's transactional key-value interface as its underlying storage engine. In our blog post Skipping the boring parts of building a database using FoundationDB we went into the details of why we chose to build on FoundationDB. To recap, FoundationDB is an ordered, transactional, key-value store with native support for multi-key strictly serializable transactions across its entire keyspace. We leverage FoundationDB to handle the hard problems of durability, replication, sharding, transaction isolation, and load balancing so we can focus on higher-level concerns.
We are starting a series of blog posts that go into the details of how Tigris has been implemented. In the first post of the series, we will share the details of how we have built the metadata storage layer on top of FoundationDB. We will cover the topics of data layout, and schema management.
Data layout
As Tigris is a multi-tenant system, when a user is created they are assigned to a tenant. All of their data is then stored under this tenant. Thus, the hierarchy of data storage looks something like this
Data layout
As Tigris leverages FoundationDB as the storage engine, which exposes a key-value interface, the data has to be stored as key-value pairs. This means there needs to be some translation from a logical layout of storing tenants, buckets, objects, and schemas to a physical layout.
Tigris maintains different key layouts depending on the information it stores. Each key layout has a custom encoder-decoder and has a prefix at the start of the key. The encoder adds this prefix; then, the decoder uses it to decode it according to the appropriate structure. The high-level concept of key encoding remains the same for all types of data (users or system data).
Key encoding
Within Tigris, metadata is stored in collections. A collection is identified by a tenant, database, and collection name. This is the minimum information we need in the key to identify a record. However, a collection may have secondary indexes as well. Therefore, an index identifier must also be part of the key.
This key structure is made extensible by having a version component allowing us to add or remove attributes in the future.
To summarize, we need to pack the following information in the key:
tenant | database | collection | index name | index value
Taking a more realistic example, let's say we have a tenant foobar with a database userdb, a collection users, and an id field defined as the primary key of the records. This translates to the following key structure
["fooApp", "userdb", "users", "pkey", [1]] =>
{"id": 1, "email": "alex@example.com", "phone_number": 12345}
The index values are seen as an array here because Tigris supports composite indexes as well, meaning a collection can have one or more than one field defined as index fields. These index values are packed in a single binary structure.
However, storing this information in every key as-is means unnecessary costs attached to each record. Therefore we implemented key compression.