Enabling awesome search experiences with Tigris Standalone Search

Database search is okay, at best. To build awesome search experiences you need a dedicated search product.

Earlier this month we released Tigris Search in BETA and covered how it supports automatic synchronization infrastructure between Tigris Database and Tigris Search, which improves developer productivity by enabling the team to focus on shipping features and not setting up and managing search and synchronization infrastructure.

However, you may be looking to quickly enhance your application experience by adding search capabilities using existing data without waiting to migrate your database to Tigris. Or, your data may not be in a database, and it's in some other type of data store, such as S3.

Whatever the reason, the Tigris Standalone Search feature allows you to quickly add search to your existing data.

Tigris Standalone Search was released with Tigris Search BETA

As part of our release of Tigris Search BETA we also made it possible for you to take advantage of Tigris Search as a standalone feature.

Tigris Standalone Search can be used with any existing data store via integrating with our APIs and SDKs. This gives you full control to determine which indexes you create and maintain.

Here are some examples in TypeScript that demonstrate how to use Tigris Standalone Search. For a more in-depth example and a demo, checkout the search demo below.

Define your Search data model:

export const CATALOG_INDEX_NAME = "catalog";

@TigrisSearchIndex(CATALOG_INDEX_NAME)
export class Catalog {
  @SearchField({ sort: true })
  name: string;

  @SearchField({ sort: true })
  price: number;

  @SearchField({ facet: true })
  brand: string;

  @SearchField({ elements: TigrisDataTypes.STRING, facet: true })
  @Field({ elements: TigrisDataTypes.STRING })
  tags: Array<string>;
}

Create your search index for the data model:

const client = new Tigris();
const search = client.getSearch();
const catalog = await search.createOrUpdateIndex<Catalog>(CATALOG_INDEX_NAME);

Insert a document in the search index:

const result = await catalog.createOne({
  name: "fiona handbag",
  price: 99.9,
  brand: "michael kors",
  tags: "purses",
});

That's it!

For more information, including indexing multiple documents, updating indexed documents, and deleted indexed documents, see the TypeScript Standalone Search docs.

Awesome email search demo with Resend & Tigris Search

To demonstrate Tigris Search, we've created a demo application with Resend - an API platform that enables developers to build, test, and deliver transactional email at scale - that simulates a series of onboarding emails to be sent during the onboarding journey.

1. Signed up = welcome
2. After trigger X or Y amount of time = join the community
3. After trigger X or Y amount of time = request feedback

After sending the emails, they are stored in a Tigris Search index. As the status of the emails change (delivered, bounced, complained, opened, clicked), Resend triggers a webhook which updates the Tigris search index with the updated email status. You can search across all the attributes of an email document within the Tigris Search index using:

• Full-text real-time search with fuzzy matching across all fields (status, to, from, subject, created [date], and email body)
• Filtering on email status
• Sorting by date
• Browsing via pagination

It's a simple but powerful use case of Tigris standalone search.

You can:

• 🫢 Try out the Awesome Email Search demo
• 💻 Grab the Awesome Email Search code on GitHub,

Or read through the code walk-through, below.

How to send transactional email with Resend

The demo application is built using Next.js and makes use of the Next.js 13 App Router.

When the Send email form is submitted, a route handles the POST request.

src/app/api/email/route.tsx

export async function POST(request: Request) {
  let emailIndex;
  let statusCode = 400;
  try {
    const formData = await request.formData();
    const stage = formData.get('stage');
    if (!stage) {
      throw new Error('"stage" is a required search parameter');
    }

    ...

In the code above, we define a emailIndex variable representing the Tigris Search index we're going to create and also statusCode variable to hold the status code to be used in the POST response. We grab the form data from the POST request and extract the stage value. stage maps to the email we'll send: "Welcome", "Join the community", or "Request feedback". The code also checks that a required stage value was passed at all.

Next, we lookup the Resend email template and create the email:

src/app/api/email/route.tsx

    if (!stage) {
      throw new Error('"stage" is a required search parameter');
    }

    const emailTemplate = EmailTemplates[stage as string];

    if (!emailTemplate) {
      throw new Error(`Could not find email template for stage "${stage}"`);
    }

    const body = emailTemplate.template({
      name: formData.get('name')?.valueOf() as string,
      link: formData.get('link')?.valueOf() as string,
    })!;
    log('created email');

    ...

Here, we perform a lookup based on the stage value and get a reference to the email template details. The template property is a react component - yep, Resend allows you to create emails using React - and we create the body of the email, passing in name and link values that have been submitted via the POST request.

Now that we have the email contents, we can send the email:

src/app/api/email/route.tsx

    log('created email');

    const testEmailStatus = formData.get('testEmailStatus')
      ? TestEmailStatus[
          formData.get('testEmailStatus') as unknown as TestEmailStatus
        ]
      : '';
    log(
      'testEmailStatus',
      testEmailStatus,
      `[${formData.get('testEmailStatus')}]`
    );

    const sendEmailRequest: SendEmailData = {
      from: process.env.DEFAULT_EMAIL as string,
      to: [getToEmail(testEmailStatus)],
      subject: emailTemplate.emailSubject,
      react: body,
    };

    const resend = new Resend(process.env.RESEND_API_KEY);
    const sendResponse = await resend.sendEmail(sendEmailRequest);
    log('sent email');
    statusCode = 201;

    ...

Resend provides several test email addresses (e.g. bounced@resend.dev) you can use to test sending emails, so we map the testEmailStatus form data to a to email address via the TestEmailStatus lookup. We then create a SendEmailData payload including from, to, subject, and body information, and make a request to send the email via await resend.sendEmail. Finally, we set a statusCode variable value of 201 to indicate the resource has been created.

How to index emails with Tigris Search

Before we continue with the POST handling code, let's look at how we define our Search data model and create the search index.

First, we define an Email class:

src/db/models/email.ts

export const EMAIL_INDEX_NAME = "emails";

@TigrisSearchIndex(EMAIL_INDEX_NAME)
export class Email implements EmailResult {
  @SearchField({ id: true })
  id?: string;

  @SearchField({ sort: true })
  firstTo!: string;

  @SearchField({ elements: TigrisDataTypes.STRING })
  to!: string[];

  @SearchField({ sort: true })
  from!: string;

  @SearchField({ sort: true })
  subject!: string;

  @SearchField()
  body!: string;

  @SearchField({ sort: true, facet: true })
  status!: EmailStatus;

  @SearchField({ sort: true })
  createdAt!: Date;
}

The Email class defines the Search document model to be created in an index called emails. Fields to be indexed are marked with a @SearchField decorator, if they are to be sortable, sort is set in the decorator, if they are a potential facet, they are also marked as such. Where possible, data types are inferred. Since to is an Array, { elements: TigrisDataTypes.STRING } must be passed to the decorator.

With the Search data model defined, we need to create the search index:

scripts/setup.ts

import { Email } from "../src/db/models/email";
import { Tigris } from "@tigrisdata/core";

const main = async () => {
  const client = new Tigris();
  const search = client.getSearch();
  await search.createOrUpdateIndex<Email>(Email);
};

main();

This simple script should be run whenever the application starts, data model code is changed, or the application is built. It informs Tigris of the Search Model structure and instructs Tigris to create a search index via the async call to search.createOrUpdateIndex<Email>(Email).

Now that we've created the search index, we can head back to the POST handling function.

With the email sent, we can create the email document in the search index.

src/app/api/email/route.tsx

    const sendResponse = await resend.sendEmail(sendEmailRequest);
    log('sent email');
    statusCode = 201;

    const bodyString = reactElementToJSXString(body);
    log('created body string', bodyString);

    emailIndex: Email = {
      to: sendEmailRequest.to as string[],
      firstTo: sendEmailRequest.to[0] as string,
      from: sendEmailRequest.from,
      status: EmailStatus.Pending,
      subject: emailTemplate.emailSubject,
      body: bodyString,
      createdAt: new Date(),
      id: (sendResponse as EmailResponse).id,
    };

    log('creating index', emailIndex);
    // uses environment variables for credentials
    const tigris = new Tigris();
    const search = tigris.getSearch();
    const emails = await search.getIndex<Email>(EMAIL_INDEX_NAME);
    const createResult = await emails.createOne(emailIndex);

    if (createResult.error) {
      console.error('Error occurred saving search index', createResult.error);
    } else {
      log('Index created', createResult);
    }

    ...

The emailIndex variable is assigned a structure matching the Email class data model we've defined. We then get the emails index using await search.getIndex<Email>(EMAIL_INDEX_NAME) and then create the email document within the index with await emails.createOne(emailIndex). Finally, we check the result and log success or failure

note

A 201 even if the index creation fails?

I made the decision to still return a 201 if the email is sent but the search index insertion fails. This is an interesting point for discussion.

With the email sent and the document inserted into the search index, we return the email index to the client if everything has gone as expected. Otherwise, we return a status code of 400 along with the error message.

src/app/api/email/route.tsx

    if (createResult.error) {
      console.error('Error occurred saving search index', createResult.error);
    } else {
      log('Index created', createResult);
    }
  } catch (ex: any) {
    console.error(ex);
    return NextResponse.json({ error: ex.toString() }, { status: statusCode });
  }

  return NextResponse.json(emailIndex, { status: statusCode });
}

Here's what the sending experience looks like which makes use of the POST endpoint:

How to search with Tigris Search

The whole point of adding documents to the search index is to enable us to build a search experience.

Let's walk through the GET search handler:

src/app/api/email/route.tsx

export async function GET(request: Request) {
  const response: SearchResponse = {
    results: [],
    meta: undefined,
    error: undefined,
  };

  const { searchParams } = new URL(request.url);
  log('searchParams', searchParams);

  try {
    const query = searchParams.get('search') || undefined;
    const statuses = searchParams.get('statuses') || undefined;
    const sortOrder = toSortOrder(searchParams.get('sortdir'));

    // Number(null) = 0
    const page = Number(searchParams.get('page'));
    if (page <= 0) {
      throw new Error(
        '"page" is a required search parameter and must be a number greater than 0'
      );
    }

    ...

At the top of the GET function, we create a response to hold the query result: a search result or an error. We also extract the searchParam from the request and the following parameters:

• query: the textual query
• statuses: one or more of the valid email status values
• sortdir: this is sent as asc or desc. We convert the latter into a sort order with the structure { field: 'createdAt', order: '$desc' | '$asc' } to be used with the Tigris SDK.
• page: used for search pagination. Required by this endpoint.

Next, we perform the search using the Tigris SDK:

src/app/api/email/route.tsx

    const tigris = new Tigris();
    const search = tigris.getSearch();
    const emails = await search.getIndex<Email>(EMAIL_INDEX_NAME);

    const searchQuery: SearchQuery<Email> = {
      q: query,
      sort: sortOrder,
      hitsPerPage: PAGE_SIZE,
    };

    if (statuses) {
      const orFilter: Filter<Email>[] = statuses.split(',').map((status) => {
        const enumAsStr = status as unknown as EmailStatus;
        const asEnum = EmailStatus[enumAsStr];
        log('map', status, enumAsStr, asEnum);
        return {
          status: asEnum,
        } as Filter<Email>;
      });

      if (orFilter.length > 1) {
        searchQuery.filter = {
          $or: orFilter,
        };
      } else {
        searchQuery.filter = orFilter[0];
      }
    }

    log('searchQuery', JSON.stringify(searchQuery, null, 2));
    const tigris = new Tigris();
    const search = tigris.getSearch();
    const queryResult = await emails.search(searchQuery, page);
    log('queryResult', queryResult);

    ...

Above, we get the search index with await search.getIndex<Email>(EMAIL_INDEX_NAME), and build a SearchQuery<Email> using query, the sortOrder (created via the sortdir search parameter), and set a paging size (PAGE_SIZE is defined elsewhere but can be any numerical value. In this app we use 10.).

If statuses is populated it will be a comma-separated list of values (e.g. Delivered,Bounced) so we split the string by ,, and map the string values to emums representing the EmailStatus values, and create a search filter on the status search document property. If there's only one statuses filter, this is used standalone. If multiple statuses have been sent, we need to split the query into multiple $or filters.

Finally, we execute the search with await emails.search(searchQuery, page), passing in the SearchQuery<Email> and page representing the desired page used in search pagination.

We're now ready to build the JSON response:

src/app/api/email/route.tsx

    response.results = queryResult.hits.map((hit) => hit.document);
    response.meta = queryResult.meta;
  } catch (ex) {
    console.error(ex);
    response.error = ex as string;
  }

  return NextResponse.json(response);
}

Once we have the search result, we map over the hits and create an Array of hit.document, which are of type Email from our data model. We also pass the queryResult.meta information to the client via result.meta. This contains information such as the total number of search results, the current page, and the page size.

With that, search functionality is in place.

How Resend webhooks trigger Tigris Search index updates

The final step is to handle inbound webhooks from Resend that indicate when the status of an email has changed. For example, when it's Delivered, Opened, or when a link in the email is Clicked.

export async function POST(request: Request) {
  let statusCode = 500;

  try {
    const body = (await request.json()) as ResendWebhook;
    log('WebHook', JSON.stringify(body, null, 2));
    const newStatus: EmailStatus = WebHookToEmailStatusLookup[body.type];
    log(`Updating status to`, newStatus);

    ...

The POST function starts by assuming the worst and setting the HTTP response statusCode to 500. It then gets the json payload sent from Resend and asserts it as a type defined by ResendWebhook. We then perform a lookup to determine the event type from the payload (Delivered, Bounced etc.).

Once we know the event type of Resend webhook, we want to retrieve the document from the search index that relates to the email the webhook is for:

    log(`Updating status to`, newStatus);

    log('getting index and document');
    const emailsIndex = await search.getIndex<Email>(EMAIL_INDEX_NAME);
    const toUpdate = await emailsIndex.getOne(body.data.email_id);
    log('found document', JSON.stringify(toUpdate, null, 2));

    if (toUpdate === undefined) {
      statusCode = 404;
      throw new Error(
        `Could not find document in index for Email with id ${body.data.email_id}`
      );
    }

    ...

Here, we get the index and assign it to an emailsIndex variable. We then use the body.data.email_id from the webhook to find the document to update with emailsIndex.getOne(body.data.email_id). If the document can't be found, we set the statusCode to 404 and throw an Error`.

If we've made it this far, we've found the document we want to update.

    if (toUpdate === undefined) {
      errorStatusCode = 404;
      throw new Error(
        `Could not find document in index for Email with id ${body.data.email_id}`
      );
    }

    toUpdate.document.status = newStatus;
    const updateStatus = await emailsIndex.updateOne(toUpdate.document);
    log('Update result', JSON.stringify(updateStatus, null, 2));

    if (updateStatus.error) {
      const msg = 'Error updating indexed document:' + updateStatus.error;
      console.error(msg);
      throw new Error(msg);
    }
    statusCode = 200;
  } catch (ex) {
    console.error(ex);
  }

  return NextResponse.json({}, { status: statusCode });
}

So, we update the status of the document with the new status from the webhook and perform an update with emailsIndex.updateOne(toUpdate.document). We then check the status of the update, and if it was successful set the statusCode to 200. Finally, we send a response using NextResponse.json({}, { status: statusCode }) which make use of the statusCode we've been setting throughout the POST function. A 200 tells Resend that the webhook was successfully handled. Error response codes are handled differently and may result in retries.

Try the demo to get started with Tigris Search and Resend

Ok, now you're definitely ready to:

• 🫢 Try out the Awesome Email Search demo
• 💻 Grab the Awesome Email Search code on GitHub,

Please also consider joining the Tigris Discord, following @TigrisData on Twitter, and giving Tigris a Star on GitHub.

Have questions? Get in touch.