# Building with OpenAI 

Gadget has a variety of purpose-built features for building apps that use the exciting innovations in artificial intelligence.

*   vector allows storing and retrieving vectors for implementing [semantic search](https://docs.gadget.dev/guides/glossary#semantic-search) for LLM memory
*   file allows storing generated artifacts like images, audio, or video synthesized by AI models
*   HTTP Routes [streaming support](https://docs.gadget.dev/guides/http-routes#streaming-replies) support for sending slow-to-generate responses from upstream APIs
*   Support for the [`openai`](https://www.npmjs.com/package/openai) and [`langchain`](https://js.langchain.com/docs/) node modules (without any annoying edge-environment drawbacks)

## Working with LLMs 

Large Language Model (LLM) APIs allow developers to build incredible apps that can understand and generate text. To work with LLMs like OpenAI's [GPT-4](https://platform.openai.com/docs/guides/chat) or Anthropic's [Claude](https://www.anthropic.com/product), you can make API calls from your Gadget app to these services.

### Using vectors for LLM state 

Developers often need to pass specific context to a call to an LLM. Content like:

*   the current user's details
*   documentation relevant to the user's questions
*   up-to-date information about current events

are all commonly inserted into an LLM prompt to refine the results. This is by no means a complete list -- just about any text can be sent in. However, since the LLMs have a limited input context size, you can't just pass everything under the sun. Instead, you must choose a relevant subset of your available context to the LLM.

Identifying and retrieving the most relevant context to pass is a common problem with a lot of depth. Most folks turn to a _vector similarity search_ to solve this problem. Vector similarity searches allow you to store a big library of potential context entries and retrieve only the most relevant entries for each prompt. Then, only the relevant entries can be inserted into the prompt, keeping it short enough to be processed by the model. Relevance can be assessed without having to build a big rules system or a deep understanding of the plain-language that users might use to describe their problems.

Gadget supports vector similarity searching, filtering and sorting using the vector field on your app's data models.

#### Vector embeddings of strings 

Vector search requires computing a vector for the content you want to search on. Vector similarity search allows you to ask the database to return records with vectors that are similar to an input vector, so you need a tool to convert any incoming strings (or images, etc) into a vector that you can then store and search with. This conversion-to-a-vector process is called **embedding**. The quality of the embedding system is what allows strings that are semantically similar but different character-wise to have similar vectors.

For example, let's say a user is asking a chatbot the question "What will keep me dry in the rain?". If we have a bunch of stored content that we might want to feed to the LLM, we don't want to just search for the term "dry" or "rain" in the text, we'd ideally search for related terms, like "umbrella", "shelter", or "waterproof". Vector embeddings are the fancy algorithms that group all these different English words together by producing similar vectors for similar text.

```mermaid
flowchart LR
    subgraph docs [Document processing]
    Documents[/Documents\]
    model{{Embedding Model}}
    Documents-->|vector embedding|model
    end
    Gadget[(Gadget Database)]
    model-->|Store vectors|Gadget
```

Gadget recommends using OpenAI's [embeddings API](https://platform.openai.com/docs/guides/embeddings) for computing vectors from strings. You can use the `openai` package to make calls to the embeddings API:

```typescript
const response = await openai.createEmbedding({
  model: "text-embedding-ada-002",
  input: "what will keep me dry in the rain",
});
const vector: number[] = response.data.data[0].embedding;
```

Read more about installing the `openai` npm package .

#### Storing vectors on models 

To create a database of context for your LLM apps, you can store context as a [Model](https://docs.gadget.dev/guides/models) in Gadget. Models allow you to store structured, unstructured, and vector data together in a single record within your app's serverless database.

For example, if we were building a chatbot for a product to answer questions powered by existing documentation, we could store each paragraph of the documentation as a record in a **Documentation** model, and then use this stored documentation to power the prompt generation for an LLM. We can create a model with the following fields:

*   a string `body` field to store the plain text
*   a url `url` field for where the documentation actually came from
*   a vector `embedding` field for storing a 

Then, when creating a new record, we can compute the embedding for the `body` string automatically, and store it in the `embedding` field. In the `create` action of the model, we can add code to the `onSuccess` function to compute the embedding and save it.

```typescript
// import the openai client from a file where we construct it
import { openai } from "../../openai";

export const run: ActionRun = async ({ api, record }) => {
  // get an embedding for the record.body field
  const response = await openai.createEmbedding({
    model: "text-embedding-ada-002",
    input: record.body,
  });
  const vector = response.data.data[0].embedding;

  // store the vector in the db on this record
  await api.internal.description.update(record.id, {
    description: { embedding: vector },
  });
};
```

#### Retrieving similar records 

Once you have records stored in your models with a vector field, you can query these records to retrieve records that are similar to a given vector.

```mermaid
flowchart LR
    subgraph inputp [Input processing]
    model2{{Embedding Model}}
    input([User Input])
    input-->|vector embedding|model2
    end
    Gadget[(Gadget Database)]
    model2--o|Search by vector similarity|Gadget
```

Using the example **Documentation** model mentioned earlier, we can query the stored records to retrieve the body of the documentation snippets most relevant to a given prompt:

```typescript
const relevantRecords = await api.documentation.findMany({
  sort: {
    embedding: {
      cosineSimilarityTo: [
        0.1, 0.15, 0.1, 0.5,
        // ... the remainder of a vector embedding
      ],
    },
  },
  first: 10,
  select: {
    id: true,
    body: true,
  },
});
```

This query will retrieve the top 10 most similar records to the given vector, and return only the `id` and `body` fields of each record. The `embedding` field does not need to be selected in this case as it's only being used to power the sorting of the results, and we don't actually need the stored vectors client side.

Read more about the sorts and filters offered by your models with vector fields in the [API Reference](https://docs.gadget.dev/api/example-app/development/sorting-and-filtering#sorting-by-vector-distance).

When sorting or filtering by vectors, you _need_ to use the same embedding API to compute the vectors stored on models and the vectors from any user input. For OpenAI, this means you must pass the same `model` parameter to all your text embedding calls so that the vectors returned are all in the same vector space.

#### Selecting vector distance calculations as fields 

In addition to sorting by vector distance, you can select vector distance calculations as fields in your queries to retrieve the actual similarity scores or distance values for each record.

For any vector field, you can select two additional fields:

*   `{fieldName}CosineSimilarityTo` - Returns the cosine similarity score between the stored vector and a provided vector
*   `{fieldName}L2DistanceTo` - Returns the L2 (Euclidean) distance between the stored vector and a provided vector

Both fields accept a `vector` argument that specifies the vector to compare against. The returned values are floats representing the calculated distance or similarity.

```typescript
await api.document.findFirst({
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] },
    },
  },
});
```

```tsx
const [result, refresh] = useFindFirst(api.document, {
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] }
    }
  }
});
const { data, error, fetching } = result;
```

```graphql
query FindDocument {
  document {
    id
    body
    embeddingCosineSimilarityTo(vector: [1, 2, 3])
  }
}
```

```json
{}
```

You can select both distance calculations in the same query:

```typescript
await api.document.findMany({
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] },
    },
    embeddingL2DistanceTo: {
      [api.$args]: { vector: [1, 2, 3] },
    },
  },
});
```

```tsx
const [result, refresh] = useFindMany(api.document, {
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] }
    },
    embeddingL2DistanceTo: {
      [api.$args]: { vector: [1, 2, 3] }
    }
  }
});
const { data, error, fetching } = result;
```

```graphql
query FindManyDocuments {
  documents {
    edges {
      node {
        id
        body
        embeddingCosineSimilarityTo(vector: [1, 2, 3])
        embeddingL2DistanceTo(vector: [1, 2, 3])
      }
    }
  }
}
```

```json
{}
```

When selecting vector distance calculations, the input vector must have the same dimensions (length) as the stored vector. If the dimensions do not match, the query will return an error.

If the vector field is `null` for a record, both `{fieldName}CosineSimilarityTo` and `{fieldName}L2DistanceTo` will return `null` for that record.

You can combine selecting distance calculation fields with sorting by vector distance. This is useful when you want both the sorted results and the actual similarity scores:

```typescript
await api.document.findMany({
  sort: {
    embedding: {
      cosineSimilarityTo: [1, 2, 3],
    },
  },
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] },
    },
  },
  first: 10,
});
```

```tsx
const [result, refresh] = useFindMany(api.document, {
  sort: {
    embedding: {
      cosineSimilarityTo: [1, 2, 3]
    }
  },
  select: {
    id: true,
    body: true,
    embeddingCosineSimilarityTo: {
      [api.$args]: { vector: [1, 2, 3] }
    }
  },
  first: 10
});
const { data, error, fetching } = result;
```

```graphql
query FindManyDocuments($sort: [DocumentSort!]) {
  documents(sort: $sort, first: 10) {
    edges {
      node {
        id
        body
        embeddingCosineSimilarityTo(vector: [1, 2, 3])
      }
    }
  }
}
```

```json
{ "sort": { "embedding": { "cosineSimilarityTo": [1, 2, 3] } } }
```

#### Passing context to an LLM 

With your data embedded into vectors within your app's database, you can query it for context to add to a prompt for an LLM. The flow is generally this:

1.  Take the user's input and embed it into a vector
2.  Query the database to find the most similar context using vector similarity against the input vector
3.  Build a bigger, internal string prompt for the LLM by adding the retrieved context to the user's prompt
4.  Send the prompt to the LLM and forward along its response to the user.

```mermaid
flowchart TB
    subgraph inputp [Input processing]
    direction LR
    model2{{Embedding Model}}
    input([User Input])
    input-->|1. Vector embed user input|model2
    end

    model2-->|2. Search by vector similarity|Gadget
    subgraph promptgent [Prompt generation]

    Gadget[(Gadget Database)]
    Gadget-->|Relevant context|prompt
    prompt(3. Generate combined prompt)
    input-->prompt
    end
    prompt-->llm{4. Invoke LLM}-->output(((Output)))
```

For example, if we have documentation for a product stored in a **Documentation** model as mentioned above, we can use the most relevant documentation to build a prompt for an LLM. We can accept the user's input as a param to an [HTTP Route](https://docs.gadget.dev/guides/http-routes) and then embed it into a vector, retrieve the most relevant records, and then send a combined prompt to OpenAI's chat completion API:

```typescript
import { openai } from "../openai";
import { RouteHandler } from "gadget-server";

const route: RouteHandler<{ Querystring: { question: string } }> = async ({
  api,
  request,
  reply,
}) => {
  // get the user's question from a ?question= query param
  const question = request.query.question;

  // get a vector embedding for the question
  const response = await openai.createEmbedding({
    model: "text-embedding-ada-002",
    input: question,
  });
  const questionVector = response.data.data[0].embedding;

  // find the most relevant records from the documentation model
  const relevantRecords = await api.documentation.findMany({
    sort: {
      embedding: {
        cosineSimilarityTo: questionVector,
      },
    },
    first: 5,
    select: {
      id: true,
      body: true,
    },
  });

  // build a big prompt out of a base prompt, the user's question, and the retrieved context
  const completion = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    messages: [
      {
        role: "system",
        content:
          "You are a helpful assistant who answers users questions with some related documentation.",
      },
      {
        role: "user",
        content: `
          Related documentation: ${relevantRecords
            .map((record) => record.body)
            .join("\n")}

          Question: ${question}
        `,
      },
    ],
  });

  await reply.send(completion.data.choices[0].message!);
};

export default route;
```

Then, we can call this route to get a chat completion:

```bash
> curl https://example-app--development.gadget.app/ask?question=what%20will%2keep%20me%20dry%20in%20the%20rain
A raincoat or shelter will keep you dry in the rain.
```

At this point, you've successfully implemented semantic search for LLM prompt augmentation in your app! Well done!

For vector similarity to work, we need to _also_ embed the user's input into a vector, and compare that vector to the vectors in the database. We compare vectors to vectors, so anything that got embedded on the way into the database has to be compared to other embeddings only.

## Working with Image Generators 

Text-to-image APIs like [Stable Diffusion](https://stablediffusionapi.com/)'s or [DALL-E](https://platform.openai.com/docs/api-reference/images) can be used with Gadget to build incredible image generation apps.

### Generating images 

Images can be generated within Gadget apps by making API calls to one of the outstanding image generation models out there, like Stable Diffusion, or OpenAI's DALL-E.

### Using Stable Diffusion 

For generating images with Stable Diffusion, we recommend using `got` for making HTTP requests. Add `got` version 11 to your package.json:

```json
// in package.json
{
  "dependencies": {
    "got": "^11"
  }
}
```

and hit the `Run yarn install` button to install the package. Then, add a `STABLE_DIFFUSION_API_KEY` environment variable to your Environment Variables in your Settings.

Now, you can use `got` to make requests to Stable Diffusion in your action code or HTTP routes:

```typescript
const response = await got("https://stablediffusionapi.com/api/v3/text2img", {
  method: "POST",
  json: {
    key: process.env["STABLE_DIFFUSION_API_KEY"],
    prompt: "a cat",
    samples: 1,
    width: "512",
    height: "512",
  },
}).json();

// will be a URL to an image of a cat! this url will expire after 1 hour
const url = response.output[0];
```

### Using DALL-E 

To use DALL-E, we recommend using the `openai` client from npm. Install the `openai` package using the .

Then, use the `openai` package in your action code or routes:

```typescript
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env["OPENAI_API_KEY"],
});

const response = await openai.images.generate({
  prompt: "a cat",
  n: 1,
  size: "1024x1024",
});

// will be a URL to an image of a cat! this url will expire after 1 hour
const url = response.data[0].url;
```

### Storing images 

Text-to-image APIs typically have a short expiration time for the images they generate. As a result, the URLs returned from these APIs will stop working after a short time. To preserve any images for later use, you need to copy them into your own storage.

Gadget's file field type works well for storing any blob content, including generated images. You can copy files from external URLs in your Gadget app's storage with the `copyURL: "https://some-url"` input format for file.

To generate and store an image, you can call your preferred image generation API, then pass a URL to a generated image to the `copyURL` input of a file field on a model of your choosing.

For example, we can create a new `generate` Action on an example `image` Model, and add code to the `run` function for generating the image with the Stable Diffusion API:

```typescript
import got from "got";

/**
 * Action code for the generate action on the Image model
 */
export const run: ActionRun = async ({ api, record, logger }) => {
  const generateResponse: { status: string; output: string[] } = await got(
    "https://stablediffusionapi.com/api/v3/text2img",
    {
      method: "POST",
      json: {
        key: process.env["STABLE_DIFFUSION_API_KEY"],
        // pass a prompt from a field named `prompt` on the image model
        prompt: record.prompt,
        samples: 1,
        width: "512",
        height: "512",
      },
    }
  ).json();

  if (generateResponse.status != "success") {
    logger.error({ response: generateResponse }, "error generating image");
    throw new Error("generating image failed");
  }

  const url = generateResponse.output[0];
  logger.info({ url }, "generated image");

  // update the image field to store this generated image by passing the url to the `copyURL` input param
  await api.image.update(record.id, { image: { image: { copyURL: url } } });
  logger.info("stored generated image in Gadget");
};
```

Now, you can generate new images with your `api` client by calling the `generate` action, and access the generated images from Gadget's high-performance cloud storage:

```typescript
const record = await api.image.generate({ image: { prompt: "a cool looking cat" } });
// record.image.url will be a URL to an image of a cat! this url will not expire!
console.log(record.image.url);
```

## Installing dependencies 

Gadget apps are built with [Node.js](https://nodejs.org/en/), and you can install any dependencies that communicate with third-party AI APIs like OpenAI's by adding packages to your `package.json`.

### Installing OpenAI 

To install the [`openai`](https://npmjs.com/package/openai) package from npm, add it to your `package.json`:

```json
// in package.json
{
  "dependencies": {
    "openai": "^4.55.7"
  }
}
```

and hit the `Run yarn install` button to install the package.

Then, add an `OPENAI_API_KEY` environment variable in your Environment Variables in your app's Settings section. You can get an OpenAI API Key from the [OpenAI developer portal](https://platform.openai.com/docs/introduction).

Next, construct an instance of the client object in a helper file:

```typescript
import OpenAI from "openai";

export const openai = new OpenAI({ apiKey: process.env["OPENAI_API_KEY"] });
```

Now, you can use the `openai` package in your action code or routes:

```typescript
import { openai } from "../openai";

// ... somewhere in your route
const response = await openai.images.generate({
  prompt: "a cat",
  n: 1,
  size: "1024x1024",
});

// will be a URL to an image of a cat!
const url = response.data[0].url;
```