Tigris is a globally available, S3 compatible object storage service.

Tigris uses FoundationDB's transactional key-value interface as its underlying storage engine. In our blog post Skipping the boring parts of building a database using FoundationDB we went into the details of why we chose to build on FoundationDB. To recap, FoundationDB is an ordered, transactional, key-value store with native support for multi-key strictly serializable transactions across its entire keyspace. We leverage FoundationDB to handle the hard problems of durability, replication, sharding, transaction isolation, and load balancing so we can focus on higher-level concerns.

We are starting a series of blog posts that go into the details of how Tigris has been implemented. In the first post of the series, we will share the details of how we have built the metadata storage layer on top of FoundationDB. We will cover the topics of data layout, and schema management.

I did a presentation at the FoundationDB Monthly Meetup. The talk is about, how building secondary indexes for a database is always about balance. A balance between a system that scales and is easy to manage and an API that is intuitive and delightful for a developer to use. Recently at Tigris Data, we have been adding secondary indexes to our metadata store and have been working hard to achieve a good balance between scale and developer delight. Tigris is a storage platform that leverages FoundationDB as one of its core components. In this talk I cover four aspects we had to balance:

1. Handling schema changes automatically in our secondary indexes.
2. The trade-off between auto-indexing all fields and indexing select fields.
3. Changes we made after performance testing.
4. How we plan build indexes in the background with minimal conflicts.

Client-side encryption refers to the practice of encrypting data on the client side (user's device) before it is transmitted to a server or stored in a remote location. This approach enhances the security and privacy of user data by ensuring that sensitive information is encrypted locally, and only the encrypted form is transferred or stored on the server.

Here's how client-side encryption typically works:

For the first time setup the keys are generated which are typically the master-keys. These keys are used to encrypt and decrypt. They are derived from user entered passphrases.

In cloud services before the client sends the data to the server, the client encrypts the data using the generated encryption key. Most commonly AES encryption is used as the encryption algorithm. Once the data is encrypted, it is transmitted to the remote server and stored in encrypted form, even if the data is intercepted it is in unreadable form without corresponding decryption key.

While reading the data the server transmits the encrypted data back to the client and client decrypts the encrypted data using the encryption key. This step ensures the sensitive information remains secure as decryption occurs on the user's trusted services.

In some cases, client-side encryption is used in conjunction with end-to-end encryption (E2EE), where only the end users (clients) have the keys to decrypt and access the data. This ensures that even service providers or server administrators cannot access the unencrypted data.

Client-side encryption is commonly implemented in various applications and services, such as:

Cloud Storage Services: Users can encrypt their files locally before uploading them to the cloud, ensuring that the cloud service provider cannot access the content.

Messaging Apps: End-to-end encryption in messaging apps ensures that only the intended recipients can decrypt and read the messages.

File Transfer Services: Encrypting files before transferring them over the internet ensures the confidentiality of the data during transit.

This approach empowers users with greater control over their data and adds an extra layer of security, especially in scenarios where users may not fully trust the server or the service provider with their sensitive information.

Benefits of Client-side encryption​

Client-side encryption offers several advantages, particularly in enhancing the security and privacy of user data. Here are some key benefits:

Enhanced Data Security:

Client-side encryption ensures that sensitive data is encrypted on the user's device before being transmitted or stored remotely. This significantly reduces the risk of unauthorized access or data breaches.

User Control: Users have greater control over their encryption keys and, consequently, their data. This empowers individuals to manage and safeguard their information, reducing dependence on service providers.

Privacy Preservation: Client-side encryption enhances user privacy by preventing service providers or third parties from accessing unencrypted user data. Only users with the corresponding encryption keys can decrypt and access their information.

Reduced Trust in Service Providers:

Users don't need to fully trust service providers with their unencrypted data. Even if the service provider is compromised, the encrypted data remains unreadable without the encryption keys.

Compliance with Privacy Regulations: Client-side encryption aids in compliance with data protection and privacy regulations by ensuring that sensitive information is secured at the source, reducing the risk of non-compliance.

Challenges with client-side encryption​

Client-side encryption, while offering enhanced security and privacy, comes with its own set of challenges and considerations:

Key Management: Effective key management is critical. Lost or forgotten encryption keys can result in permanent data loss, and securely storing and recovering keys can be challenging.

User Experience: Implementing seamless user experiences, especially around key generation and storage, can be complex. Striking a balance between security and user-friendliness is essential.

Key Distribution: In scenarios where multiple devices or users need access to encrypted data, securely distributing and managing encryption keys becomes a significant challenge.

Recovery Mechanisms: Implementing secure and user-friendly key recovery mechanisms is important. Users should have a way to regain access to their encrypted data if they forget their password or lose their encryption key.

Search and Indexing: Searching and indexing encrypted data becomes challenging, as the server cannot directly access the content. Techniques like searchable encryption may be used, but they add complexity.

Despite these challenges, client-side encryption remains a valuable approach for securing user data, particularly in scenarios where users prioritize privacy and want control over their information. Addressing these challenges requires careful planning, a user-centric approach, and ongoing efforts to stay abreast of advancements in encryption technologies and best practices.

Several well known implementations of client side encryption in cloud services are Amazon S3, AWS Wickr, Azure blob storage, Snowflake, MongoDB.

We are running FoundationDB with the official kubernetes operator. FoundationDB supports logical backups (with backup_agent) and disaster recovery (with dr_agent) through copying the database/streaming changes logically. It also supports binary backups through disk snapshots.

In this blog post, we will describe how to make a backup of FoundationDB via backup_agent. The FoundationDB operator supports making logical backups via backup_agent, but it does not support running DR with dr_agent. We decided to run backup_agent as a separate deployment to allow a symmetric setup with dr_agent.

In computer systems, username and password has become a widely used way to authenticate users. The evolution of password storage and authentication methods has gone through various stages to enhance security and protect user credentials.

Here's an overview of the progression from plain text passwords to using Secure Remote Password (SRP) and the key concepts at each stage:

Plain Text Passwords: In the early days of computing and network authentication, passwords were often stored in plain text on servers. This means that when a user entered their password, it was stored exactly as entered. The main problem with plain text storage is that if the server is compromised, attackers can easily obtain the passwords, potentially leading to unauthorized access to user accounts.

Password Hashing: The next step was to use password hashing to address the security weaknesses of plain text passwords. Instead of storing the actual password, this approach stores a one-way cryptographic hash of the password on the server. Hashing ensures that the original password cannot be easily retrieved from the stored value, even if the database is breached. Common hashing algorithms used for password storage include MD5, SHA-1, and more secure ones like bcrypt, scrypt, and Argon2. These newer algorithms are designed to be slow and computationally intensive, making it more difficult for attackers to crack passwords through brute force or dictionary attacks.

Salted Hashes: The concept of adding salt to password hashes was introduced to enhance security. A salt is a random value unique to each user, combined with their password before hashing. Salting prevents precomputed rainbow tables and ensures that even identical passwords result in different hashes due to the unique salts. Combining salt with password hashing significantly improves the resistance to offline attacks.

Secure Remote Password (SRP): SRP is an advanced authentication protocol that addresses the weaknesses in traditional password-based authentication systems. In SRP, the password is transformed into a verifier value stored on the server, and the actual password is never transmitted during authentication. SRP uses cryptographic techniques to provide mutual authentication, where both the client and server prove their identities without exchanging plaintext secrets. The protocol also defends against a wide range of attacks, including man-in-the-middle attacks, replay attacks, and offline password attacks.

In summary, password security has progressed from storing plain text passwords to using password hashing with salts and finally to the more advanced and secure Secure Remote Password (SRP) protocol. SRP protects passwords from exposure and provides strong security features for remote authentication, making it a valuable choice for applications that require robust security and user protection. Secure Remote Password (SRP) is a cryptographic protocol and authentication mechanism that allows two parties, typically a client and a server, to establish a secure connection and authenticate each other without exposing the user's password to potential eavesdroppers or attackers. The primary goal of SRP is to protect the user's password from being transmitted over the network, making it resistant to offline password attacks.

Here's a simplified overview of how SRP works:

Registration

• Client and server first negotiate on an SRP group and agree on one.
• Client collects a user-provided username & password and generates random salt.
• Client uses the key derivation function to derive a secure key from a user-provided password.
• Client then uses an agreed SRP group and derived secure key x to generate a verifier v.
• Client sends the username, salt, and verifier (v) to the server over a secure channel.
• Server stores the SRP group, username, salt, and verifier (v) in durable storage, with (v) stored securely.

Authentication

• We assume the client can access the secure derived key (x) and the agreed SRP group.
• Client creates an ephemeral public key (A) and sends it to the server.
• Server computes the ephemeral public key (B) and sends it to the client.
• Server uses its private key to encrypt and send username and salt to the client as proof.
• Client verifies the proof and sends its proof to the server using its private key.
• Once the server verifies the proof, both the client and server have verified that the other party has the username and password and mathematically agreed on the proof. Authentication is successful in this case.

SRP provides several security benefits:

• The password is never sent over the network, making it resistant to eavesdropping.
• It offers protection against offline dictionary and brute force attacks because the verifier stored on the server is derived from the user's password and a salt.
• It provides mutual authentication, meaning the client and the server verify each other's identity. It offers additional authentication over TLS.

SRP is widely used in secure authentication systems and can be particularly valuable in situations where secure remote authentication is crucial, such as online banking and other secure applications.

Tigris is a globally available, S3 compatible object storage service. Tigris utilizes FoundationDB's transactional key-value interface for its underlying metadata storage. This blog delves into the topics of serializable transactions, the mechanics of transactions within Tigris, and concurrency control.

The most complicated and time-consuming parts of building a new storage system are usually the edge cases and low-level details. Concurrency control, consistency, handling faults, load balancing, that kind of thing. Almost every mature storage system will have to grapple with all of these problems at one point or another. For example, at a high level, load balancing hot partitions across brokers in Kafka is not that different from load balancing hot shards in MongoDB, but each system ends up re-implementing a custom load-balancing solution instead of focusing on their differentiating value to end-developers.

This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Most of them decide to reimplement common processes and components without substantially increasing the value gained from reimplementing them. For instance, many database builders start from scratch when building their own storage and query systems, but often merely emulate existing solutions. These items usually take a massive undertaking just to get basic features working, let alone correct.

We're excited to announce that Tigris Data has joined Netlify's Jamstack Innovation Fund as one of the 10 most promising Jamstack startups.

As the world increasingly shifts to digital-first interactions, the need for fast, reliable, and secure web applications has never been greater. The Jamstack movement is a response to this need, focused on building web applications that provide rich experiences while at the same time being easy to deploy and scale. Data is, of course, crucial to building rich experiences, and the support from Netlify will accelerate our mission to provide the fast, reliable and secure data layer that Jamstack applications need to thrive.

In our inaugural blog post Hello world, we talked about the problem of data infrastructure sprawl that, over the years, has complicated modern application development, while putting a lot of burden on the operations team who must manage, maintain and secure many different databases and technologies.

In this blog post, we'll explain what we mean when we say "data infrastructure sprawl" by walking through a typical example, and then we'll explain why it doesn't have to be this way.

We're excited to announce the launch of Tigris Data, a company on the mission of simplifying storage for developers.

Over the years, data has become increasingly complex and difficult to manage. Developers have had their lives made exponentially more difficult due in large part to all the different technologies they're expected to put together to build modern applications.

The data sprawl also puts a lot of pressure on operations teams, who must manage, maintain and secure different system. Then there is the onerous task of operationalizing these system across multiple different cloud platforms.