Quickstart: Vector search with OpenAI and Tigris
In this guide, we'll show you how to use the OpenAI Embeddings API to generate embeddings for your documents and then use Tigris to index these embeddings for fast and scalable vector search.
This is a powerful combination that can be used for building semantic search applications, recommendation engines, and more.
Vector search workflow with OpenAI and Tigris
The vector search workflow with OpenAI and Tigris works as follows:
- Generate embeddings for your documents using the OpenAI Embeddings API.
- Index the embeddings in Tigris.
- Pass the search query through the OpenAI Embeddings API to generate an embedding.
- Use the embedding to search for similar documents in Tigris.
- Get back semantically similar documents, even if they don't contain the search query keywords.
Now, let's build a vector search application that implements this workflow.
Building the vector search application
To follow along with this guide, you'll need:
- An OpenAI account. You can sign up for an account here
- A Tigris Cloud account. You can sign up for a free account here
- Node.js 18.13.0 or newer
Once you have signed up for the Tigris Cloud account, create a new project called vectorsearchapp.
Next, make a note of the clientId and clientSecret, which you can get from the Application Keys
section of the project.
Bootstrap the application
Tigris provides a tool called create-tigris-app that you can use to quickly bootstrap a new Node.js
application. We will use this tool to create an application with Tigris and OpenAI clients already
configured.
npx create-tigris-app@latest --project vectorsearchapp --example vector-search-openai
Command output
✔ What is the clientId? … client_id_here
✔ What is the clientSecret? … *******************************************************
Creating a new app in /Users/ovaistariq/projects/vectorsearchapp.
Downloading files for example vector-search-openai. This might take a moment.
Initializing project with template: vector-search-openai
Using npm.
Installing dependencies:
- @tigrisdata/vector: latest
- csv-parse: ^5.3.6
- dotenv: ^16.0.3
- express: 4.18.2
- openai: ^3.2.1
- reflect-metadata: ^0.1.13
- zod: ^3.19.1
added 410 packages, and audited 411 packages in 12s
111 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Initialized a git repository.
Success! Created vectorsearchapp at /Users/ovaistariq/projects/vectorsearchapp
Inside that directory, you can run several commands:
npm run dev
Starts the development server.
npm run build
Builds the app for production.
npm start
Runs the built app in production mode.
We suggest that you begin by typing:
cd vectorsearchapp
npm run dev
The create-tigris-app tool will create a new directory called vectorsearchapp. Let's
change into this directory.
cd vectorsearchapp
We will need the OpenAI Organization ID and API key to authenticate with OpenAI. Visit the API Keys page to retrieve the API key and the Org Settings page to retrieve the Organization ID.
Then update the .env file with the OpenAI Organization ID and API key.
OPENAI_ORG=your_org_id_here
OPENAI_API_KEY=your_api_key_here
Now we are all set.
Next, we will demonstrate how simple it is to store the embeddings in Tigris and then use the Tigris client to search for similar documents. We will use a subset of reviews from the Amazon fine-food reviews dataset.
Generating embeddings with OpenAI
Let's run the following command to generate embeddings for the reviews and store the reviews and embeddings in Tigris.
npm run seed
The command performs the following steps:
- Creates a Tigris Search Index
- Reads the reviews from a CSV file AND generates embeddings via OpenAI
- Store reviews and embeddings in Tigris
Now that the embeddings have been generated and stored in Tigris, we can start the application and perform searches.
Running the vector search application
Let's run the following command to start the application in development mode running on http://localhost:3000:
npm run dev
Command Output
> vectorsearchapp@0.1.0 dev
> npx nodemon
[nodemon] 2.0.22
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts
[nodemon] starting `npx ts-node ./src/index.ts`
info - Using Tigris at: api.preview.tigrisdata.cloud:443
🚀 Server ready at: http://localhost:3000 ⭐️
Searching for similar reviews
Let's perform some searches now.
curl \
'http://localhost:3000/search?searchString=dog&k=10' \
-H 'Content-Type: application/json'
This returns the following results:
[
[
{
"content": "Excellent, lo-fat, lo- calorie treat for dogs. I have a German Shorthair Pointer-and these treats are her favorite( the organic peanut butter cookies are for smaller dogs, but she loves them-and for weight-control I like them!)",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B002HQH04O",
"ProfileName": "Judith Mariani",
"Score": 5,
"Summary": "The best",
"Time": 1325462400,
"UserId": "ATGPKFUEPU3UU"
}
},
0.1918412446975708
],
[
{
"content": "We have three dogs and all of them love this food! We bought it specifically for one of our dogs who has food allergies and it works great for him, no more hot spots or tummy problems.<br />I LOVE that it ships right to our door with free shipping.",
"metadata": {
"HelpfulnessDenominator": 4,
"HelpfulnessNumerator": 4,
"ProductId": "B0019CW0HE",
"ProfileName": "Toby's mom",
"Score": 5,
"Summary": "Great food!",
"Time": 1292889600,
"UserId": "A25BGFRHYHEZKK"
}
},
0.19350367784500122
],
[
{
"content": "My 7 year old Cocker Spaniel loves this food and it's good for him -- he needs to lose a few pounds and this is doing the trick. Glad I found it on Amazon. Hard to find elsewhere.",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B003AO5DLO",
"ProfileName": "SH",
"Score": 5,
"Summary": "Happy Dog",
"Time": 1350086400,
"UserId": "A3IQFDK2A3APGE"
}
},
0.1942516565322876
],
[
{
"content": "This is great dog food, my dog has severs allergies and this brand is the only one that we can feed him.",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B0019CW0HE",
"ProfileName": "gare",
"Score": 5,
"Summary": "Great dog food",
"Time": 1334966400,
"UserId": "A7JWCYVPF7KCO"
}
},
0.1949772834777832
],
[
{
"content": "I have a standard poodle and pomeranian who both do wonderful on this food. I have switched them to a different food (due to price) a couple of times and end up going right back to natural balance.",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B0019CW0HE",
"ProfileName": "Lisas201",
"Score": 5,
"Summary": "Great food for my my dog who has a sensitive stomach.",
"Time": 1335398400,
"UserId": "A1OGD72Y2Y26CF"
}
},
0.19964390993118286
],
[
{
"content": "These are healthy and my mixed Llasa, Tibetan Terrier adores them. And they are healthy and his breath doesn't smell and his teeth are in good shape because of such products. A great healthy product.",
"metadata": {
"HelpfulnessDenominator": 1,
"HelpfulnessNumerator": 1,
"ProductId": "B002MV23XM",
"ProfileName": "Shirley Sacks",
"Score": 5,
"Summary": "my dog loves these",
"Time": 1337040000,
"UserId": "A2M6O8YORUA8PW"
}
},
0.20004874467849731
],
[
{
"content": "Great food! I love the idea of one food for all ages & breeds. Ît's a real convenience as well as a really good product. My 3 dogs eat less, have almost no gas, their poop is regular and a perfect consistency, what else can a mom ask for!!",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B0019CW0HE",
"ProfileName": "Lorraine \"Agelechio\"",
"Score": 5,
"Summary": "Great Food",
"Time": 1347408000,
"UserId": "A1EC57S6C2VLTL"
}
},
0.2010520100593567
],
[
{
"content": "This food is great - all ages dogs. I have a 3 year old and a puppy. They are both so soft and hardly ever get sick. The food is good especially when you have Amazon Prime shipping :)",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B0019CW0HE",
"ProfileName": "J. Romagnoli \"*jroma\"",
"Score": 5,
"Summary": "Mmmmm Mmmmm good.",
"Time": 1333929600,
"UserId": "ABG53EBZBKD23"
}
},
0.20133048295974731
],
[
{
"content": "This is a very healthy dog food. Good for their digestion. Also good for small puppies. My dog eats her required amount at every feeding.",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B00171APVA",
"ProfileName": "Carol A. Reed",
"Score": 5,
"Summary": "Healthy Dog Food",
"Time": 1351209600,
"UserId": "A21BT40VZCCYT4"
}
},
0.20262646675109863
],
[
{
"content": "Our pup has experienced allergies in forms of hotspots and itching from other dog foods. The cheap 'you can buy it anywhere' food not only have crazy preservatives in them but can cause health problems for your pets. This food works wonders on reducing allergies and our dog loves the food.<br />This message is RAMSEY FrAnkenSteiN approved.",
"metadata": {
"HelpfulnessDenominator": 0,
"HelpfulnessNumerator": 0,
"ProductId": "B0019CW0HE",
"ProfileName": "FuNky Faja \"SiLkk\"",
"Score": 5,
"Summary": "Great allergy sensitive dog food, dogs love it",
"Time": 1297296000,
"UserId": "A3UII2114114PI"
}
},
0.2029741406440735
]
]
We have searched for "dog" and the results are sorted by the vector distance. The vector distance is a measure of how similar the search string is to the review text. The closer the vector distance is to 0, the more similar the review text is to the search string.
Next Steps
🚀 Signup for Tigris Cloud - and start building your next application with it. We are here to help you get started. Join our Discord community to ask any questions you might have.